What Is A/B Testing?

What Is A/B Testing?

A/B testing, also written AB testing, is a randomized controlled experiment that compares two versions of one thing to find out which performs better. The thing can be a web page, an email, a price, an app feature, or a search ranking rule. Teams run an A/B test to learn whether a change causes a better result before they roll it out to everyone.

Version A is the control, usually what people see today. Version B is the variant, the same thing with one change. The test sends each visitor to one version by chance and compares a single metric between the two groups.

Every A/B test has four parts:

• A change to evaluate, such as a new headline, a shorter form, or a faster checkout.
• A random split, so the two groups differ only by that change.
• One primary metric chosen before launch, such as signups per visitor.
• A statistical comparison that says whether the gap between groups is larger than chance alone would produce.

Take a signup page. Half of the visitors see the current headline, “Start your free trial”, and half see “Try it free for 30 days”. Because chance decided who saw which headline, a higher signup rate in one group can be credited to the headline, and the team keeps the winner.

A/B Testing Meaning

In A/B testing, the letters name the two versions being compared. A is the original and B is the change. When a test adds more versions, the names continue as A/B/C and so on, which is called an A/B/n test. “AB test” and “A/B test” mean the same thing, and people drop the slash when they type.

Different fields use different names for the same method:

• Split test, bucket test, and split-run test are older names for an A/B test, and Wikipedia lists all three as synonyms.
• Online controlled experiment is the formal name that researchers use, for example Ron Kohavi and Stefan Thomke in Harvard Business Review.
• Randomized controlled trial (RCT) is the same design in medicine, where the groups receive a drug or a placebo.
• Control is version A, and the variant, variation, or treatment is version B.

The word “test” can suggest pass or fail, yet an A/B test measures an effect, such as “signups rose by 0.4 percentage points”, together with how sure the team can be of that size.

AB Test Definition

In statistical terms, an AB test is a randomized experiment in which units are assigned by chance to a control or a treatment, and the difference in a pre-specified outcome metric is tested against the null hypothesis that the treatment has no effect. Wikipedia describes it as “a randomized experiment that usually involves two variants (A and B)”, and Optimizely calls it a methodology for comparing two versions of a webpage or app to determine which one performs better.

A complete AB test definition names each of these terms:

• Unit: what gets randomized, such as a user, a session, a device, or an account.
• Treatment: the single change applied to the variant.
• Outcome metric: the number that decides the test.
• Null and alternative hypotheses: “the change has no effect” against “the change moves the metric”.
• Significance level, power, and minimum detectable effect: the error rates and the smallest change worth finding, which together fix the sample size. They are explained under A/B testing statistics in plain words.

Randomization is what makes the design work. Wikipedia notes that simple A/B tests are not valid for observational or quasi-experimental situations, so comparing last month’s visitors with this month’s does not qualify. Seasons, campaigns, and traffic sources change between those months, and only a concurrent random split removes them from the comparison.

A short history of A/B testing

The method is older than the web, and each step below added a piece of the modern test.

• 1747: James Lind compared treatments for scurvy among twelve sailors in six pairs, and the pair given oranges and a lemon recovered best. It is one of the first reported controlled clinical experiments, although it did not use random assignment.
• 1923: Claude Hopkins described testing advertising with coupons in Scientific Advertising, an early form of the split test without statistical significance.
• 1925: Ronald Fisher stated the case for randomization in Statistical Methods for Research Workers, because randomization eliminates bias and permits a valid test of significance.
• 1948: The Medical Research Council’s streptomycin trial for tuberculosis became the first medical RCT, with Austin Bradford Hill credited for conceiving the modern design.
• February 27, 2000: Google ran its first A/B test, which asked whether a results page should show 10, 20, 25, or 30 results. Wired reports that the larger pages loaded slower and scored worse, which showed that tenths of a second matter.
• 2006 to 2009: Google launched Website Optimizer in 2006, and Dan Siroker and Pete Koomen founded Optimizely at the end of 2009 with an editor that non-programmers could use.
• 2007 and 2012: The Obama campaign tested its splash page, and Bing tested an ad headline change that raised revenue by about 12%. Both are covered under A/B testing examples.

How A/B testing works

An A/B test moves through the same five stages every time:

1. Choose one change and one metric. The team writes down what it will change and which number decides the result.
2. Split visitors at random. Each visitor is assigned to A or B, and the assignment holds on every later visit so nobody sees both versions.
3. Collect data to a planned size. The sample size is fixed before launch, and the test runs until it is reached.
4. Compare the groups. A statistical test measures the gap in the metric and how much of it chance could explain.
5. Decide. The team ships the variant, keeps the control, or writes the next hypothesis from what it learned.

Four conditions keep the comparison fair: both versions run at the same time, assignment is random, only one thing differs, and the deciding metric is chosen before launch.

Why teams run A/B tests

Analytics shows what happened, and an A/B test shows what a change caused. Because chance sorts people into groups, every other influence, such as a press mention or a seasonal spike, reaches both groups equally, so a gap between them comes from the change.

Teams value this for several reasons:

• It settles opinion fights. Optimizely describes the shift as moving the conversation from “we think” to “we know”.
• It limits the cost of being wrong. A bad idea reaches a fraction of traffic, and the team sees the damage in the data before it reaches everyone.
• It finds ideas nobody would rank first. At Bing, an ad headline change sat in the backlog for more than six months as a low priority. When an engineer finally tested it, revenue rose about 12%, worth more than $100 million a year in the United States.
• It shows how often ideas fail. Kohavi, Tang, Xu, and colleagues report that about two-thirds of experiments fail to improve the metric they were designed to change, and 80 to 90% fail in well-optimized areas such as search.

A team that ships without testing keeps the two-thirds that fail along with the third that works, and cannot tell them apart.

The A/B testing process in eight steps

The steps below follow a test from idea to decision.

Step 1: Find a problem worth testing

Start from evidence of a problem, such as a funnel step where many visitors leave, a theme in support tickets, or a pattern in session recordings. A test that starts from evidence teaches the team something even when it loses, because the evidence says what to try next.

Step 2: Write a hypothesis

A hypothesis states what will change, for whom, by how much, and why. This template covers the parts, shown with made-up numbers:

• Evidence: “Because 62% of mobile visitors leave the pricing page without scrolling to the plans…”
• Change: “…moving the plan table above the feature list…”
• Effect: “…will raise the share of mobile visitors who start a trial by at least 0.5 percentage points.”

A weak hypothesis such as “a better pricing page will improve results” cannot be proven wrong, since any outcome can be called an improvement. ExpLint flags this with the hypothesis/is-falsifiable and hypothesis/names-metric-and-direction rules, and the hypothesis/is-based-on-evidence rule asks where the idea came from.

Step 3: Choose the metrics

Pick three kinds of metric before launch:

• One primary metric that decides the test, such as trial starts per visitor. With several primary metrics, the chance that one of them wins by luck climbs quickly.
• Guardrail metrics that must not get worse, such as page load time, refunds, or support contacts.
• Diagnostic metrics that explain the result, such as clicks on the new element.

Choose a metric close to the value you want. A 2009 headline test on the Highrise signup page counted clicks on any sign-up button, which the team noted was a step short of completed paying accounts, so a headline that attracts clicks may not attract customers.

Step 4: Pick the unit and the split

The randomization unit is the thing that gets assigned, usually a user. Assigning by session lets one person see both versions, which muddles the comparison. A 50/50 split reaches the needed sample soonest. Count only people who reach the changed experience, since visitors who never see the change add noise and shrink the measured effect.

Step 5: Calculate the sample size and duration

Work out how many visitors each version needs before the test starts, with the ExpLint sample size calculator or by hand. The inputs are the baseline rate, the smallest effect worth detecting, the significance level, and the power, and a worked example appears under A/B testing statistics in plain words. Divide the total sample by daily eligible traffic to get the run time, then round up to whole weeks so that every day of the week appears equally. CXL advises that a test span full business cycles, since behavior on a Sunday differs from behavior on a Tuesday.

Step 6: Build the test and check it

Before real traffic arrives, check these points:

• Both versions render correctly on the browsers and devices you support.
• Every event for the primary and guardrail metrics fires in both versions.
• Assignment is sticky, so a returning visitor sees the same version.

After launch, check the split itself. If you set 50/50 and the groups come out 52/48 on a large sample, something is wrong. Fabijan and colleagues call this sample ratio mismatch, a symptom of data-quality problems that can make a bad change look good and get shipped. An A/A test, which compares two identical versions, is a way to confirm the setup before a real test.

Step 7: Run the test and leave it alone

Do not stop when the dashboard first turns green. Evan Miller simulated stopping at the first significant result and found false positives in 26.1% of experiments where nothing was different, against the 5% the test promises, because every extra look is another chance to be fooled by noise. Do not change the design, the traffic split, or the metrics while the test runs either.

Step 8: Analyze, decide, and write it down

Once the planned sample is in, run the comparison once and report three things: the size of the effect, the confidence interval around it, and what happened to the guardrail metrics. Apply the decision rule you wrote before launch. A flat result is an answer, and it frees the team to put its effort into a better idea.

Record the hypothesis, the setup, the result, and the decision where the next person can find it. A searchable archive keeps a team from running the same losing test twice, which is the job of the ExpLint search tool for the specs a team has already linted.

A/B testing statistics in plain words

Statistics answers one question for an A/B test: is the gap between A and B larger than luck would produce? For depth, the standard reference is Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang, and Ya Xu.

Null hypothesis, alternative hypothesis, and the p-value

Every test starts by assuming nothing happened. The null hypothesis says A and B perform the same, and the alternative hypothesis says they differ.

The p-value measures that contradiction. Evan Miller puts it this way: “Assuming there is no underlying difference between A and B, how often will we see a difference like we do in the data just by chance?” His article calls a result significant when that chance is low, such as 5% or 1%.

The American Statistical Association published a statement on p-values in 2016 to correct common misreadings. Two of its six principles matter most for A/B tests:

• A p-value does not measure the probability that the hypothesis is true. A p-value of 0.03 does not mean B has a 97% chance of being better.
• A p-value does not measure the size of an effect or the importance of a result, so a tiny lift can be significant on a large sample and still be worthless.

Significance level, power, and minimum detectable effect

Three settings define how sensitive a test is:

• Significance level (alpha) is how often the test declares a winner when nothing really differs, called a false positive or Type I error. The usual setting is 5%.
• Power is how often the test detects an effect of the minimum size when it exists. Missing a real effect is a false negative or Type II error. The usual setting is 80%.
• Minimum detectable effect (MDE) is the smallest change worth finding. Choose it from what the business would act on, since a smaller MDE needs a much larger sample.

Evan Miller’s sample size calculator defaults to these values, 80% power and a 5% significance level.

Sample size with a worked example

Miller gives a rule of thumb for the sample each version needs: n = 16 × σ² ÷ δ², where δ is the minimum effect and σ² is the variance of the metric. For a conversion rate, σ² is the baseline rate times one minus the baseline rate. The 16 comes from 80% power and a 5% significance level.

Suppose 5% of visitors sign up today, and the team wants to detect a 10% relative lift, which is 5% rising to 5.5%.

• Variance: 0.05 × 0.95 = 0.0475.
• Minimum effect: 0.055 − 0.05 = 0.005.
• Sample per version: 16 × 0.0475 ÷ 0.005² = 0.76 ÷ 0.000025 = 30,400.

A full two-proportion calculation at the same settings gives about 31,200 per version, so calculators differ by a few percent. That makes about 62,500 visitors across both versions. At 5,000 eligible visitors a day the test needs 12.5 days, which rounds up to two full weeks. To run your own numbers, use the ExpLint sample size calculator.

The table shows how fast the requirement grows as the baseline falls and the lift shrinks. Each figure is visitors per version at 80% power and a 5% significance level.

Three patterns show up in the table:

• Halving the effect you want to detect roughly quadruples the sample.
• A low baseline rate needs far more traffic than a high one.
• A site with little traffic should test bigger changes, since only those are detectable in a reasonable time.

Confidence intervals and practical significance

A confidence interval gives the range of effects that the data supports. A result of “+0.2 to +0.8 percentage points” says more than “significant”, because it shows both the likely size and the uncertainty. Compare the interval with the smallest effect that justifies the cost of shipping and maintaining the change. A result can be statistically significant and still too small to matter, and the interval makes that visible.

Frequentist, Bayesian, and sequential testing

Three analysis styles are common, and each has its own rules:

• Fixed-horizon (frequentist) tests set the sample size in advance and look at the result once. This is the style the p-value and sample size guidance above describes.
• Sequential tests allow looks along the way. Johari, Pekelis, and Walsh show that standard p-values are unreliable when people stop based on continuous monitoring, and they propose always-valid p-values that stay correct at any stopping time.
• Bayesian tests report the probability that B beats A and a range of likely effects. They are popular in testing platforms, and the team still agrees on the stopping rule before launch.

Pick the method before the test starts and use the rules that come with it. Mixing a fixed-horizon plan with daily checking is how false positives enter.

Variance reduction with CUPED

Noisy metrics need larger samples. CUPED uses each user’s behavior before the test to remove predictable noise from the metric. In the paper that introduced it, variance on Bing fell by about 50%, which gave the same power with half the users or half the duration.

Each extra variant or metric adds another chance for a false win, so a test with ten metrics will often show one result that looks significant by luck. Name the primary metric in advance, and apply a correction such as Bonferroni when you read several secondary ones.

Types of A/B tests

Several variations of the design exist for specific jobs.

Another distinction is where the split happens. A client-side test decides the version in the visitor’s browser, which suits marketing pages and needs no engineering release. A server-side test decides the version in your own application, which suits product features, pricing, algorithms, and anything that must load without a flicker.

A/B testing examples

These examples come from published sources. Where a number is an estimate or a spoken remark, the text says so.

Bing: an ad headline

In 2012 an engineer at Bing launched a test of how the search engine displayed ad headlines. The idea had waited more than six months, and program managers had rated it low priority. Revenue rose about 12%, which Kohavi and Thomke put at more than $100 million a year in the United States, without harm to the key user-experience metrics. An alert flagged the result as too good to be true, and further analysis confirmed it. A team cannot rank ideas well without testing them.

The Obama campaign: a splash page

In December 2007 the campaign tested its splash page with four button labels and six media options, three images and three videos, which made 24 combinations. Dan Siroker described the test in a post on the Optimizely blog.

• Traffic: 310,382 visitors, or roughly 13,000 per combination.
• Metric: sign-ups divided by visitors who saw the version.
• Winner: the “Learn More” button with the family photo, at 11.6% against 8.26% for the original page, a 40.6% relative gain.
• Surprise: staff favored a video, and every video lost to every image.

Siroker estimated that the gain, if it held, produced about 2.88 million extra email addresses and, at roughly $21 per address, about $60 million in donations. That dollar figure is his extrapolation from assumed conversion rates, so treat it as an estimate. The measured part, the sign-up rate, is the solid result.

Highrise: a signup headline

In 2009 the team at 37signals tested five headline and subhead pairs on the signup page for Highrise, counting clicks on any green sign-up button over 4,000 page views. The best version used “30-day free trial on all accounts” with a subhead promising that signup takes less than 60 seconds, and it beat the original, “Start a Highrise Account”, by 30%. Jason Fried’s write-up is candid about the limits. The test changed headline and subhead together, and it measured clicks on a sign-up button, a step short of paying accounts.

Netflix: artwork

Netflix tests the image shown for each title, with take rate as the metric, which is plays divided by impressions with very short plays excluded. Its engineering blog describes a title-level design that assigns each member one artwork variant and usually picks a winner within a few days. Faces with expressive emotion and recognizable characters tended to do well, and the best image differed by region. The post gives no percentage lift, so none is quoted here.

Booking.com: testing as a habit

A 2017 paper by Booking.com’s experimentation team reports more than a thousand concurrent experiments across its apps, partner tools, and internal systems, and says every change, from redesigns to bug fixes, ships inside an experiment. Owners record their hypotheses and metrics up front.

Email: subject lines

Email tools make A/B testing routine. Mailchimp’s documentation lists four variables to test, the subject line, the sender name, the content, and the send time, with up to three variations. The tool sends each version to a share of the list, picks a winner by open rate, click rate, or revenue, and sends the winner to everyone else.

AI features: prompts and models

Teams also compare prompts and models in production. GrowthBook’s guidance on A/B testing LLM features treats a prompt comparison and a model comparison as separate experiments, recommends randomizing by user, and suggests running for about two weeks at minimum. The same rules apply as for any test, namely a hypothesis, one primary metric, and a planned sample size.

Google: shades of blue

The story of Google testing 41 shades of blue is famous and thinly sourced. Designer Doug Bowman mentioned it in his 2009 resignation post, and a Google UK executive later told an audience, as The Guardian reported, that a slightly purpler blue earned about $200 million a year in extra ad revenue. Neither source publishes a method or data, so treat the figure as an anecdote.

What the examples teach

• Small changes to words and images often beat large redesigns.
• Staff opinion is a poor predictor, as the Obama videos and the low-ranked Bing idea both showed.
• Metrics closer to value, such as paid accounts, are harder to move and more honest than clicks.
• Most tests lose, and a team that logs its losses learns the most.

Common A/B testing mistakes

Most failed A/B tests fail before the first visitor arrives, because the plan left out a hypothesis, a sample size, or a metric. These mistakes come up most often:

• Stopping when the result first looks significant. Every look at a running test is another chance to see luck as a win. Fix the sample size in advance and read the result once, or use a sequential method built for repeated looks.
• Running with no hypothesis. A test with no stated reason, metric, and direction has no result to learn from, since any outcome can be explained afterward.
• Choosing the metric after seeing the data. With enough metrics, one will look good by chance, so name the primary metric before launch.
• Testing without enough traffic. An underpowered test misses real effects and exaggerates the ones it finds. Calculate the sample size first, and if the traffic is not there, test a bigger change.
• Changing the test while it runs. VWO advises against altering goals, design, or traffic allocation mid-test, since the groups stop being comparable.
• Ignoring the calendar. CXL says a test should span full business cycles, because paydays, weekends, and promotions move behavior.
• Slicing the results afterward. Looking at twenty segments after the fact will turn up a few that seem to differ by chance. Define the segments that matter before launch.
• Skipping the split check. A group ratio that differs from the plan signals a bug in assignment or tracking, and Fabijan and colleagues warn that it can turn a harmful change into an apparent win.
• Watching only one metric. A change that lifts signups while it slows the page or raises refunds is no win. Set guardrail metrics up front.
• Treating one surprising win as fact. When most ideas fail, a share of significant results are false alarms. Kohavi and colleagues advise replicating surprising results before acting on them.

When not to run an A/B test

An A/B test costs time and traffic, and some questions are better answered another way. GrowthBook’s guidance on when not to run a test offers a useful filter. Skip the test when:

• The result cannot change the decision. If the team ships whatever the outcome, the test only delays the release.
• Traffic is too low. If the smallest effect worth acting on cannot be detected within the business horizon, test a bigger change, a higher-volume metric, or use user research.
• Ethics, law, or safety forbid it. A security fix or an accessibility requirement should ship to everyone, and withholding it to measure a conversion gain is wrong.
• Users affect each other. In marketplaces and social networks one person’s version changes another’s outcome, so a plain split gives biased answers and designs such as switchback tests fit better.
• A cheaper method answers the question. Comprehension and feasibility questions suit usability tests, interviews, and prototypes.

A/B testing tools

An experimentation tool assigns visitors to versions, records events, and runs the statistics. The right one depends on who builds the tests and where the change lives. Each platform below links to a guide that shows how to lint its test plans with ExpLint.

Google Optimize is missing from the list because Google ended Optimize and Optimize 360 on September 30, 2023. Google named AB Tasty, Optimizely, and VWO among the tools that integrate with Google Analytics 4, and Firebase A/B Testing remains supported. Email tools such as Mailchimp include their own A/B testing for subject lines and send times.

A/B test checklist before launch

Run through this list before a test goes live.

Where ExpLint fits

ExpLint checks the planning half of this list, from the hypothesis to the sample size. You point your AI assistant at a test plan in your backlog or experiment platform, and ExpLint reports each rule the plan breaks as an error or a warning, with the words that triggered it and the reason behind the call. An error means the test cannot answer its question as written, and a warning means the result will be hard to judge. A project can switch rules on or off or rewrite them in plain English. Installation steps are in the ExpLint documentation, and the guides cover each backlog tool and experiment platform.

A/B testing FAQs

Sources

• Trustworthy Online Controlled Experiments, by Kohavi, Tang, and Xu
• The Surprising Power of Online Experiments, Harvard Business Review
• Online randomized controlled experiments at scale: lessons and extensions to medicine
• How Not To Run an A/B Test, by Evan Miller
• A/B testing sample size calculator, by Evan Miller
• The ASA statement on p-values: context, process, and purpose
• Diagnosing sample ratio mismatch in online controlled experiments
• Always valid inference: bringing sequential analysis to A/B testing
• Improving the sensitivity of online controlled experiments by utilizing pre-experiment data
• A/B testing, Wikipedia
• A/B testing glossary entry, Optimizely
• How Obama raised $60 million by running a simple experiment
• The A/B test: inside the technology that's changing the rules of business, Wired
• Selecting the best artwork for videos, Netflix Technology Blog
• Democratizing online controlled experiments at Booking.com
• Writing decisions: headline tests on the Highrise signup page
• About A/B tests, Mailchimp
• When not to run a test, GrowthBook
• A/B testing guide, CXL
• Optimize sunset, Google Analytics Help