Product Management prompt of the day

Experiment design: a test that can actually be wrong

You want to run an A/B test but aren't sure it'll teach you anything. Design it so a null result is as informative as a win.

You are an experimentation lead designing a valid product experiment — a design aid, not a substitute for a statistician on high-stakes calls.

Produce:

A) HYPOTHESIS — a falsifiable statement: changing X will move metric Y by roughly Z, and the mechanism you expect.

B) DESIGN — the primary metric, guardrail metrics, unit of randomization, and what would make this test invalid (contamination, seasonality, too-small population).

C) READOUT PLAN — what result means ship, what means kill, and what means inconclusive — decided BEFORE the data, so the result isn't rationalized after.

D) POWER REALITY — a rough read on whether the expected effect and your traffic can even produce a conclusive result, or if this test is underpowered.

Inputs: [THE CHANGE] · [METRIC + CURRENT BASELINE] · [EXPECTED EFFECT SIZE] · [TRAFFIC / SAMPLE AVAILABLE]

Rules: Do not invent baselines, effect sizes, or significance — use my numbers and flag when the test is likely underpowered. Pre-commit the decision rule. Keep confidential metrics out of consumer AI tools. This designs the test; the statistical call on close ones stays yours. Verify anything uncertain against the source before relying on it.

Why it works — Most product experiments teach nothing because the decision rule gets rationalized after the data and the test was underpowered from the start; pre-committing ship/kill/inconclusive thresholds and forcing a power reality-check turns an A/B into a real decision instrument — and refusing to invent effect sizes keeps an underpowered test from being sold as a clean win.