A product requirements document covering the end-to-end design, build, and iteration of an AI-powered experiment planning tool for Product Managers — from problem identification to shipped product.
Product Managers who run A/B experiments spend a disproportionate amount of time on the planning phase — not because the work is intellectually difficult, but because it is structurally inconsistent. There is no standard format, no shared checklist, and no guardrail against common mistakes like underpowered experiments, missing guardrail metrics, or poorly defined success criteria.
This problem was identified firsthand. At Yahoo, experiment planning was done manually and inconsistently across teams. Different PMs used different templates, different metric definitions, and different standards for what constituted a well-designed experiment. The result was avoidable mistakes — experiments that ran too short, experiments that shipped without guardrails, and reviews that couldn't be compared across teams.
Each PM designed experiments in their own format. There was no shared standard for what a complete experiment plan included.
Underpowered experiments, missing guardrail metrics, and vague success criteria appeared repeatedly across teams.
A well-structured experiment plan could take 2–3 hours to write from scratch, pulling PMs away from higher-leverage work.
Senior PMs had internalized best practices around sample size, randomization, and metric selection. Junior PMs didn't — and there was no lightweight way to transfer that knowledge.
The PM Experiment Planner is a focused web application that takes a PM's hypothesis and experiment context as input and generates a complete, structured experiment design plan using the Claude AI API. The output includes a recommended primary metric, guardrail metrics, sample size guidance, experiment design, risk analysis, and a definition of success — in under 60 seconds.
An early version of this concept was scoped as an experiment review generator — a tool that would take results as input and produce a stakeholder writeup. That scope was intentionally narrowed during the design phase after identifying a fundamental problem: Claude cannot reliably interpret metric results without knowing how those metrics are defined and computed at your specific company. The planner avoids this entirely by operating upstream of the data.
The primary user is a Product Manager at a technology company who runs or oversees A/B experiments. They understand what they want to test but need help structuring the plan rigorously — identifying the right metrics, estimating sample size, and anticipating risks before launch.
2–5 years of PM experience. Runs 4–8 experiments per quarter. Comfortable with the concept of A/B testing but does not have a statistics background. Spends more time than they should writing experiment plans from scratch, often using an inconsistent personal template.
Under 2 years of experience. Has not yet built intuition for what makes a well-designed experiment. Most likely to make avoidable mistakes — running experiments that are underpowered, missing guardrail metrics, or shipping without a clear definition of success. This tool acts as a structured guide.
Data scientists or analysts who need statistical rigour beyond directional sample size guidance. Teams with existing experimentation platforms that include built-in planning tooling. The tool is deliberately a starting point, not a replacement for a full experimentation infrastructure.
| Feature | Description | Scope |
|---|---|---|
| Experiment input form | Structured form with 8 fields split across hard required, soft required, and optional tiers | V1 |
| Example templates | 3 pre-filled experiment templates (Engagement, Conversion, Retention) that load with one click | V1 |
| AI plan generation | Claude API call that generates a structured 8-section experiment plan from user inputs | V1 |
| Copy to clipboard | One-click copy of the full generated plan for use in Confluence, Notion, or email | V1 |
| Field validation | Hard blocks for required fields, soft warnings with user confirmation for incomplete optional fields | V1 |
| Dark / light mode | Full dark mode support with navy + indigo theme; light mode available | V1 |
| Experiment review generator | Takes results as input and generates a stakeholder writeup — deprioritised due to hallucination risk on proprietary metrics | V2 |
| User accounts / history | Saving past plans, team sharing, plan versioning | V2 |
| Sample size calculator | Interactive calculator with baseline rate, MDE, and confidence level inputs | V2 |
Input fields are grouped into three tiers based on how much they affect output quality. The tiering is explicitly surfaced in the UI so users understand the consequences of leaving fields blank.
If any hard required field is empty on submit, generation is blocked, the field is highlighted with a red border, and an inline error message appears. No browser alerts or popups are used — all errors are inline. If soft required fields are empty, a warning banner appears above the generate button: the user must actively confirm before proceeding, preventing silent output degradation.
Claude (claude-sonnet-4-20250514) handles one task: generating the experiment plan from user inputs. The API key is stored as an environment variable and never exposed in frontend code. Lovable routes the API call server-side to avoid CORS restrictions and enable streaming.
Claude is instructed to generate exactly 8 sections with strict length constraints per section. This prevents verbose, generic output and keeps the plan tight and actionable.
| Section | Length constraint | Notes |
|---|---|---|
| TL;DR | 3 bullets max | What's tested, key metric, biggest risk |
| Experiment Overview | 2 sentences | What and why |
| Hypothesis | 1 sentence | Structured "We believe that…" format |
| Primary Metric | 3 sentences | Name, rationale, measurement method |
| Guardrail Metrics | 2 sentences each, 2–3 metrics | Experiment-specific, not generic |
| Sample Size & Runtime | 3–4 sentences | Directional guidance + what info is needed |
| Experiment Design | 2–3 sentences + 1 key consideration | Control vs variant, key design risk |
| Risks & Watch-outs | 3 risks, 2 sentences each | Specific to this experiment only |
| What Good Looks Like | 3 sentences total | Positive / negative / inconclusive result |
Two explicit constraints are built into the prompt to prevent the most common AI failure modes in this domain:
No Ship/Kill recommendation. Claude is explicitly instructed not to make a go/no-go recommendation. That judgment depends on company context, team strategy, and business priorities that Claude cannot access.
No hallucinated metric interpretations. Claude recommends what metrics to watch and why — it does not interpret metric results or make claims about what specific numbers mean for a specific company's product.
The app is intentionally built with a minimal technical footprint. No backend, no database, no user authentication. All session data lives in the browser and resets on page refresh. This keeps the build fast, the cost near-zero, and the codebase maintainable by a single non-engineer PM.
Each experiment plan generation costs approximately $0.01–0.02 at Claude Sonnet pricing. A $5 API credit is sufficient for 250–500 generations — more than enough for development, testing, and early public use. A monthly workspace spend limit is set to cap total exposure.
The full product — from problem framing to shipped and iterated app — was completed in approximately one week as a solo project, with no engineering support.
Identified the problem from firsthand experience at Yahoo. Scoped the initial concept as an experiment review generator, then pivoted to an experiment planner after identifying the hallucination risk in result interpretation. Defined the V1 feature set.
Defined the 8 input fields, the three-tier validation model, and the full Claude prompt. Iterated on prompt length constraints and output sections. Ran a simulation of the prompt to validate output quality before building.
Pasted the comprehensive Lovable starter prompt. Reviewed the generated UI against the spec — form fields, template cards, validation logic, and output panel. Wired in the Claude API key via environment variables.
Applied dark navy + indigo theme. Added AI-Powered badge, template card hover states, form card accent border, and subtle radial gradient in the header. Each change was a separate Lovable prompt to avoid regressions.
Fixed three key issues: dark mode text visibility in output panel, excessive left/right whitespace on desktop, and raw JSON rendering during generation. Replaced the loading bar with streaming text. Added latency expectation message.
Deployed via Lovable one-click publish. Wrote portfolio case study and PRD. Set API spend limit to cap monthly cost exposure.
Decision: The original concept was an experiment review generator — a tool that takes results and generates a stakeholder writeup. This was narrowed to an experiment planner during design.
Rationale: Reviewing results requires interpreting proprietary metrics that Claude cannot reliably understand without company-specific context. Planning an experiment requires expertise that is generalisable. The pivot eliminates the hallucination risk entirely while delivering more upstream value.
Decision: Claude is explicitly instructed never to recommend shipping or killing an experiment.
Rationale: This judgment depends on business context, team strategy, and company priorities that Claude cannot access. Including a recommendation would create false confidence. The tool defines what good looks like — the PM makes the call.
Decision: All data lives in the browser and resets on page refresh. No user accounts, no plan history, no saved state.
Rationale: Keeps the V1 build fast and the codebase simple. User accounts and plan history are clear V2 features once there is evidence of repeat usage.
Decision: Form fields split into hard required (blocks generation), soft required (warns but allows generation with confirmation), and optional (labelled but never enforced).
Rationale: A single required/optional binary would either block too aggressively or allow severely degraded output silently. The three-tier model gives users agency while preventing the worst outcomes — a plan generated with no audience, no duration, and no goal is nearly useless.
Decision: Claude recommends guardrail metrics based on experiment context. The user does not need to know their guardrail metrics upfront.
Rationale: At the planning stage, many PMs — especially junior ones — do not know what their guardrail metrics should be. That is precisely what the tool is for. Requiring the user to input guardrails defeats the purpose. Claude surfaces them; the PM validates them against their specific context.
Decision: V1 provides directional sample size guidance in prose. An interactive calculator with baseline rate, MDE, and confidence level inputs is scoped to V2.
Rationale: The directional guidance is sufficient to flag underpowered experiments and identify what information the team needs. A full calculator requires baseline data the user may not have at planning time. Ship the simpler version first and validate whether users actually need the calculator before building it.
The single highest-leverage decision in this project was the prompt structure — specifically the length constraints per section and the two explicit guardrails (no Ship/Kill recommendation, no metric interpretation). These constraints produced dramatically better output than an unconstrained prompt. Writing a good prompt requires the same thinking as writing a good product spec: clear inputs, clear outputs, explicit constraints, and anticipated edge cases.
The pivot from review generator to experiment planner was the most important product decision in this project. It was made by asking "where does the AI have reliable generalizable knowledge?" — and then scoping the product to that zone. AI tools fail when they are asked to interpret context-specific information they cannot access. They succeed when they are asked to apply general expertise to a specific situation.
A 20-second wait for a blank screen followed by a full response dump feels slow and broken. The same 20-second response with text streaming in from second one feels fast and responsive. The total time is identical. Streaming implementation was one of the most impactful UX improvements in the build — and it had nothing to do with the AI output quality.
Lovable's AI builder is powerful but prone to regressions — asking for multiple changes in one prompt often breaks things that were already working. The most effective workflow was one change per prompt, using Lovable's built-in version history to snapshot before each iteration. This discipline made the build faster overall even though it felt slower in individual prompts.