Experiment setup checklist for A/B tests

Check off each step of setting up an A/B test, from the hypothesis and metrics through randomization, sample size, duration, the decision rule, and the rollback plan, with your progress saved in this browser. The guide to what A/B testing is covers the background behind these steps.

0 of 24 steps

Hypothesis

0 of 4

  1. Errorhypothesis/is-falsifiable
    Grade hypothesis (opens in a new tab)
  2. Errorhypothesis/names-metric-and-direction
    Grade hypothesis (opens in a new tab)
  3. Warninghypothesis/explains-the-mechanism
    Grade hypothesis (opens in a new tab)
  4. Errorhypothesis/is-based-on-evidence
    Grade hypothesis (opens in a new tab)

Metrics and guardrails

0 of 4

  1. Errormetrics/has-one-primary-metric
  2. Warningmetrics/primary-metric-measures-behaviour
  3. Errorguardrails/are-listed
  4. Errorguardrails/have-launch-thresholds

Variants and audience

0 of 4

  1. Errorvariants/control-is-described
  2. Errorvariants/each-differs-from-control-stated
  3. Errorvariants/multi-change-bundles-are-labelled
  4. Warningaudience/has-exclusion-criteria

Randomization and power

0 of 6

  1. Errorrandomization/states-the-unit
  2. Warningrandomization/unit-matches-interference
  3. Errorrandomization/states-the-split
  4. Errorpower/has-a-sample-size-calculation
    Open sample size calculator (opens in a new tab)
  5. Errorpower/states-the-significance-level
    Open sample size calculator (opens in a new tab)
  6. Errorpower/traffic-makes-the-test-feasible
    Open end date calculator (opens in a new tab)

Duration and analysis

0 of 3

  1. Errorduration/covers-full-business-cycles
    Open end date calculator (opens in a new tab)
  2. Errorduration/has-a-fixed-horizon
    Open end date calculator (opens in a new tab)
  3. Erroranalysis/names-the-statistical-test
    Pick a statistical test (opens in a new tab)

Decision and rollout

0 of 3

  1. Errordecision/has-criteria-before-launch
  2. Warningdecision/plans-for-an-inconclusive-result
  3. Errorrollout/has-a-rollback-plan