A/B Testing
Rules
- Hash-based assignment: deterministic variant selection using
hash(userId + experimentId) % 100— no randomness drift between sessions - Variant config: store experiments in a config object
{ id, name, variants: [{ id, weight }], status }— never hardcode variants - Split traffic server-side: assign in middleware or API route, set cookie for consistency — client-side splits cause flicker
- Statistical significance: require 95% confidence (p < 0.05) before declaring a winner — use z-test for proportions
- Sample size: calculate minimum sample before starting —
n = (Z^2 * p * (1-p)) / E^2where E is minimum detectable effect - One metric per experiment: define a primary metric upfront (conversion rate, click-through, revenue) — secondary metrics are exploratory only
- Mutual exclusion: users in one experiment should not overlap with conflicting experiments — use experiment layers
- Feature flags: wrap experiment code in feature flag checks — clean up losing variants after experiment ends
- Duration: run for at least 1-2 full business cycles (7-14 days minimum) to account for day-of-week effects
- Logging: track
experiment_id,variant_id,user_id,timestampfor every impression and conversion event
Avoid
- Peeking at results before sample size is reached — inflates false positive rate
- Changing experiment parameters mid-run — invalidates statistical analysis
- Testing too many variants at once — increases required sample size exponentially
- Client-side variant assignment without cookies — causes flickering and inconsistent experiences