What it does
Experiment Designer and Readout covers both ends of an A/B test, and it exists because of one specific failure: tests ended too early on an insufficient sample, then read as though the result meant something. In design mode you give it a hypothesis, a baseline conversion rate and a minimum detectable effect, and it computes the required sample size per variant and the estimated duration at your traffic through a standard two-proportion power calculation. In readout mode you give it the observed control and treatment counts, and it runs a two-proportion z-test on the primary metric and every guardrail, applies a Bonferroni correction across all the comparisons, and returns a ship, iterate, hold or continue decision with the statistics behind it. The peeking guard is the part that earns its place. Before returning any ship-or-iterate call, it checks the observed sample against the design requirement — and if the test is under-sampled, the recommendation is forced to "continue the test" rather than a decision the data cannot support. That is a refusal built into the code, not a caveat in the output. It is equally disciplined about missing inputs. Without a baseline rate it uses a clearly labelled placeholder and raises a critical finding saying the plan is illustrative only. Without daily traffic it reports the duration as not estimable rather than inventing a number. Each step is callable alone. `compute_sample_size` answers how many users you need. `estimate_duration` answers how long. `check_guardrails` tests whether anything regressed. `check_peeking` answers whether you are allowed to call it yet. `get_decision` returns the verdict and rationale. Every statistic is computed deterministically. The language model narrates the numbers for a product audience and never recomputes or invents one, with a template fallback when no model is configured. No connectors and no credentials, and no mock data either — it runs offline on the numbers you provide. Built for product managers, growth and data teams.
Example prompts
- Design a test for our new pricing page — how many users and how long?
- Here are the results — can we ship it?
- Did any guardrail metric regress in this test?
