A/B Testing: The Statistics Mistakes That Fake Your Wins

Your dashboard says the new checkout lifted conversion by 14 percent, significant at 95 percent. You ship it, and revenue does not move. Nobody lied, but your experiment did, and it will keep lying until you fix the statistics underneath it.

This post is a field guide to the specific mechanisms that manufacture fake wins: peeking, underpowered designs, sample ratio mismatch, multiple comparisons, and the winner's curse. Each one has a concrete fix you can implement this week.

Why so many "wins" evaporate at rollout

An A/B test is a noisy measurement of a small effect. When you run many noisy measurements and only act on the ones that cross a threshold, you systematically select for lucky noise. That is the winner's curse: the tests you ship are, on average, overestimates of the true effect. Teams that rerun their winning variants as holdback tests find the measured lift shrinks, sometimes to zero, more often than they expect.

The curse gets worse when the underlying process is broken. Every mistake below inflates your false positive rate, your effect size estimates, or both. Stack two or three of them and your experimentation program becomes a random number generator with a nice UI.

Mistake 1: Peeking at the p-value

The classic. You launch a test, check the dashboard daily, and stop the moment the p-value dips below 0.05. It feels diligent. It is actually the fastest way to fabricate results.

A fixed-horizon test (the standard t-test or z-test every tool computes) controls the false positive rate at 5 percent only if you look once, at a sample size you chose in advance. Under repeated peeking, the p-value is a random walk that will wander below 0.05 by chance alone. Check daily for a month on an A/A test and your realistic false positive rate can climb into the 20 to 30 percent range. You are not testing anymore. You are waiting for noise to blink.

The fix: pick one of three disciplines

• Fixed horizon, enforced. Compute the sample size up front, do not evaluate significance until you hit it. Simple, robust, requires organizational discipline more than math.
• Group sequential designs. Pre-plan a small number of interim looks (say 3 to 5) with alpha spending functions like O'Brien-Fleming. You can stop early for large effects without inflating error rates.
• Always-valid inference. Methods like the mixture sequential probability ratio test (mSPRT) give confidence sequences that stay valid under continuous monitoring. This is what powers "anytime-valid" results in platforms like Statsig, Eppo, and Optimizely's stats engine. The cost: wider intervals and longer runtimes for the same minimum detectable effect.

If your tool shows a live p-value and your team treats it as a stopping signal, you have the peeking problem regardless of what the vendor brochure says. Ask specifically whether the engine is fixed-horizon or sequential, and align your process to the answer.

Mistake 2: Underpowered tests

Power is the probability your test detects a real effect of a given size. Most teams never compute it, and it shows. A test on 4,000 visitors with a 3 percent baseline conversion rate has almost no chance of reliably detecting a realistic 5 percent relative lift.

Underpowering does two nasty things:

• You miss real wins. A true improvement reads as "not significant" and gets killed.
• Your significant results exaggerate. When power is low, the only effects that clear the significance bar are the ones that got lucky upward. This is Type M (magnitude) error, and it is why underpowered programs report spectacular lifts that never replicate.

Run the math before you launch. Ten lines of Python:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

baseline = 0.03          # current conversion rate
mde_rel = 0.10           # minimum detectable effect, relative
effect = proportion_effectsize(baseline, baseline * (1 + mde_rel))

n_per_arm = NormalIndPower().solve_power(
    effect_size=effect, alpha=0.05, power=0.80, ratio=1.0
)
print(round(n_per_arm))  # roughly 51,000 per arm

Around 51,000 visitors per arm to detect a 10 percent relative lift on a 3 percent baseline, at 80 percent power. If your site gets 2,000 visitors a week, that test runs for a year. The honest conclusions are uncomfortable but useful: test bigger, bolder changes with larger expected effects, test higher-traffic pages, move the metric closer to the change (clicks on the redesigned element rather than final purchase), or accept that some questions are not answerable with your traffic and decide by design judgment instead. We covered how to structure that pipeline of bets in our guide to building a CRO experimentation system.

Mistake 3: Sample ratio mismatch, the silent killer

You configured a 50/50 split. The test ends with 50,900 users in control and 49,100 in treatment. Close enough? No. That is a sample ratio mismatch (SRM), and it usually means your randomization or logging is broken, which invalidates every other number in the readout.

Common causes we see in real implementations:

• Bot filtering that fires after assignment and interacts with the variant (the treatment loads slower, times out more, and drops more sessions).
• Redirect-based tests where the redirect itself loses users from one arm.
• Assignment keyed on a cookie that gets reset by a consent banner or an app update, reassigning returning users mid-test.
• Triggering the experiment exposure event at different points in each variant's code path.

The check is one chi-square test, and it should run automatically on every experiment:

from scipy.stats import chisquare

observed = [50900, 49100]
stat, p = chisquare(observed)   # expected 50/50
if p < 0.001:
    print("SRM detected: investigate before reading any metric")

Use a strict threshold like 0.001 because you will run this on every test and do not want false alarms. But when it fires, stop. Do not "note it and interpret with caution." An SRM test result is not a weaker result, it is a corrupted one. Find the mechanism, fix it, rerun.

Mistake 4: Multiple comparisons and segment fishing

One experiment, one primary metric, alpha 0.05: a 5 percent false positive rate. Now add four secondary metrics, then slice by device, browser, new versus returning, and traffic source. You just ran 40 hypothesis tests. The probability that at least one comes up "significant" by pure chance is close to 90 percent.

This is how "the variant lost overall but won on mobile Safari for returning users" gets shipped. That segment result is almost always noise, discovered by exhaustive search and promoted to insight after the fact.

The fixes are procedural, not mathematical heroics:

• One primary metric, declared before launch. It alone decides ship or kill.
• Guardrail metrics (latency, error rate, unsubscribe rate) that can veto a ship but never rescue a loss.
• Corrections for planned families of metrics. Bonferroni is brutally simple; Benjamini-Hochberg controls false discovery rate with less power loss and is usually the better default for secondary metrics.
• Segments generate hypotheses, never conclusions. If mobile Safari looks interesting, run a new pre-registered test targeting it.

Your analytics setup matters here too. If event definitions drift between tools, you will fish across metric variants without realizing it. Getting clean, trustworthy GA4 data is a prerequisite for trustworthy experiments, not an afterthought.

Mistake 5: Novelty effects and calendar blindness

Two timing failures fake wins on otherwise sound statistics.

Novelty and primacy effects. Returning users notice the change. Some click it because it is new (novelty inflates the lift), some avoid it because it is unfamiliar (primacy deflates it). Either way, week one behavior does not predict steady state. Mitigations: run at least two full weeks, plot the effect by day and by user cohort (first exposure date), and be suspicious of lifts that decay across cohorts.

Calendar blindness. Weekend traffic converts differently from weekday traffic. Payday weeks differ from other weeks. A test that runs Tuesday to Friday samples a biased slice of your audience. Always run whole weeks, and never stop a test mid-week because it crossed a threshold (see Mistake 1, which this compounds).

Mistake 6: Statistical significance without practical significance

A p-value below 0.05 tells you the effect is probably not exactly zero. It does not tell you the effect matters. With enough traffic you can detect a 0.2 percent relative lift with immaculate confidence, and it can still be a bad ship if the variant adds maintenance cost, page weight, or complexity.

Flip it around too: "not significant" does not mean "no effect." It often means "we could not tell." The confidence interval is the honest summary. An interval of [minus 1 percent, plus 9 percent] says the test was inconclusive, not that the idea failed.

Make the decision rule explicit before launch: ship if the point estimate exceeds your cost-of-change threshold and the interval excludes meaningful harm. Write that threshold down. A landing page team that knows which elements actually drive conversion can set these thresholds from expected impact rather than vibes.

Level up: variance reduction with CUPED

Once the basics are clean, the highest-leverage upgrade is variance reduction. CUPED (Controlled-experiment Using Pre-Experiment Data) adjusts each user's metric using their pre-experiment behavior as a covariate:

adjusted_metric = metric - theta * (pre_metric - mean(pre_metric))
theta = cov(metric, pre_metric) / var(pre_metric)

For metrics where past behavior predicts future behavior (revenue per user, sessions, engagement), variance reductions in the 30 to 50 percent range are common in practice, which translates directly into shorter tests or smaller detectable effects. Most modern experimentation platforms support it; if yours does not, it is a few dozen lines in your analysis pipeline.

The pre-launch checklist

Run this before every experiment. It takes ten minutes and kills most fake wins at the source.

1. Write a one-line hypothesis: the change, the audience, the expected mechanism.
2. Declare one primary metric and up to three guardrails.
3. Compute required sample size from baseline rate, minimum detectable effect, alpha 0.05, power 0.80.
4. Confirm the runtime covers at least two full weeks at current traffic. If it exceeds six to eight weeks, redesign the test (bigger change, closer metric, bigger surface).
5. Choose the stopping rule: fixed horizon with no early reads, or a sequential engine with pre-planned looks.
6. Define the ship rule: what point estimate and interval bounds justify shipping, given the cost of the change.
7. Verify assignment: exposure event fires at the same point in both arms, bot filtering happens before assignment, identity is stable across the test.
8. Enable the automatic SRM check with a 0.001 threshold.
9. Log all of it somewhere immutable before launch. Pre-registration is what makes the analysis honest.

Reading results without fooling yourself

When the test ends, work through it in order. First the SRM check: if it fails, discard the readout. Then the primary metric with its confidence interval, against the pre-registered ship rule. Then guardrails, which can veto. Then secondaries and segments, labeled explicitly as hypothesis-generating. Finally, log the result (including losses and inconclusives) in a searchable experiment archive. Programs that only remember their wins reinvent their failures every two quarters.

And periodically run A/A tests, same experience in both arms. Around 5 percent of them should come up significant at alpha 0.05. If you see materially more, your stack has a bug: SRM, dependent sessions treated as independent users, or a stats engine misconfigured for your traffic patterns. An A/A test is the cheapest audit your experimentation program will ever get.

How Innovation T can help

Innovation T builds experimentation systems that survive contact with statistics: platform selection and setup, sequential testing configuration, automated SRM and A/A monitoring, CUPED pipelines, and the analytics foundation underneath it all. We have seen where these programs break, and we engineer the failure modes out before your first test launches.

If your test results feel too good to replicate, or your team argues about what a readout means, talk to us. Explore our digital marketing and engineering services or contact us for an experimentation program audit.