Lint Statsig experiments with explint.com before you launch
Read an experiment from Statsig with its MCP server, check the setup with explint.com before you start it, and fix the parts that would make the result hard to trust.
What you need before you start
- A Statsig project with an experiment drafted, including its hypothesis, primary metric, and allocation.
- A Statsig Console API key for the MCP connection, which you create in Project Settings under Keys and Environments when your admin allows personal keys.
- An assistant that can hold two MCP connections at once, and an A/BLint project your team can tune.
- A few minutes to connect, plus time before launch to act on what the check finds.
Connect Statsig and A/BLint
- Add Statsig's MCP server in your assistant at https://api.statsig.com/v1/mcp following Statsig's MCP setup for your assistant, and give it the Console API key it asks for.
- Add A/BLint in the same assistant at https://explint.com/mcp and finish the browser sign-in, or use an API key from the MCP page in your A/BLint workspace if your assistant asks for a key instead.
- Ask the assistant to list your active and draft experiments, because that list proves the key works and shows the names you can ask about.
- Name the experiment you want checked and your A/BLint project, so the assistant reads the right draft.
How the check runs
- The assistant reads the experiment from Statsig, including its hypothesis, groups, allocation, primary and secondary metrics, and any targeting rules.
- It writes those details into one plain text that keeps your field names, which makes the report easy to match back to the Statsig screens.
- It sends the text to A/BLint with ablint__lint and brings back errors, warnings, and passes.
- It explains each finding in plain words and points at the Statsig setting that would fix it, such as the allocation field or the metric choice.
- You update the experiment in Statsig, then the assistant reads it again and lints it again to confirm the fix.
Copy and paste this prompt
Change the words in square brackets to name your own project, board, or test.
Prompt for your assistant
Use the Statsig MCP tools to read the experiment called [experiment name]. Build one text from its hypothesis, groups and what each group sees, allocation, primary and guardrail metrics, targeting, and planned run time. Lint that text with explint.com against the project [your A/BLint project name]. Show me each error and warning with the Statsig field it came from. Do not start or change the experiment. Suggest fixes in this chat, and re-check after I update Statsig.
What A/BLint looks for
A/BLint scores the text against the rules switched on in your project. The default project uses the rules below, and your team can turn rules off, change how strict they are, or add its own rules in plain English. An error means the test cannot answer its question as written, and a warning means the result will be hard to judge when it comes in.
| Rule | Level | In plain words |
|---|---|---|
| hypothesis/names-metric-and-direction | error | The guess names the number it expects to move and which way it should move, such as a rise in checkout completion. |
| metrics/has-one-primary-metric | error | One main number decides the test, so the team agrees before the test starts which result matters most. |
| guardrails/are-listed | error | The write-up names the numbers that must not get worse, such as speed, errors, refunds, or people leaving. |
| randomization/states-the-unit | error | The plan says who or what gets split into groups, such as a person, a visit, or an account, because every rate depends on that choice. |
| randomization/states-the-split | error | The plan gives the share of traffic each version gets, so the team can judge whether the test can finish in time. |
| variants/each-differs-from-control-stated | error | Each version says exactly what changes, so the team can tell afterwards which change caused the result. |
| decision/has-criteria-before-launch | error | The team writes down before launch what result would ship, what result would stop, and what result would mean running the test again. |
| audience/has-exclusion-criteria | warning | The plan says who is left out, such as staff, bots, or test accounts, and why they would muddy the result if they stayed in. |
An example you can compare against
Before
Checkout button colour
- Hypothesis
- A green button will do better.
- Groups
- Control 50 percent, Test 50 percent
- Primary metric
- Revenue
After, checked again
Shorter checkout button label on phones
- Hypothesis
- If we change the phone checkout button from Place your order securely to Pay now, then the share of checkout starts that end in a paid order will rise, because usability sessions showed people pausing to work out whether tapping would charge them twice.
- Main number
- Share of checkout starts that end in a paid order.
- Guardrails
- Payment errors must stay under 1 percent, and support contacts about double charges must not rise.
- Groups
- Control keeps the current label. Test changes only the label, with the same colour, size, and position.
- Who is split
- Signed in users on phones, split by user, 50/50, staff and test accounts left out.
- Size and time
- About 22,000 checkout starts over two full weeks, enough to spot a 1 point rise.
- Decision rule
- Ship if the main number rises and the guardrails hold.
Still flagged on the rewrite
- errorguardrails/have-launch-thresholdsThis guardrail names a worry without the number that would settle it.
- warningdecision/has-criteria-before-launchShipping is covered, and the flat and falling results have no agreed outcome.
The stronger plan says who gets split and who stays out, and it keeps the change down to one label so the result points at one cause. Statsig makes allocation and metrics easy to set quickly, which is exactly why a slow read of the plan before launch earns its time.
Fix what it finds and check again
Start with the errors, and have the assistant suggest each rewrite in the chat before anything is written back to Statsig, so wording your team already agreed stays yours to approve. Rewrite the experiment plan so a teammate reading it cold could say what will change, what number will judge it, and what would stop a launch, then lint the same experiment plan again and ask which rules now pass that failed before.
Limits and things to check yourself
- Your Statsig admin controls who can make Console API keys, so if the MCP connection fails at sign-in, ask an admin to allow a key for your role before you debug the assistant setup.
- Metrics computed in your warehouse or in a custom Statsig setup need their plain English meaning pasted into the text, because the checker can only judge the words it receives.
- The check covers the written plan. It cannot confirm your SDK calls, your event names, or that exposures are logging correctly once the test starts.
- A passing score says the write-up covers what the rules ask for. It cannot say the idea is worth testing, that the traffic exists to reach a clear result, or that the change is safe to build, and those calls stay with your team.