The rest of the site is written for someone deciding whether to try this. This page is written for whoever they hand it to afterwards. Nothing here is different from what the marketing pages claim, it is just spelled out.
Every page is sorted into a group by three things: what kind of page it is, roughly how much traffic it gets, and how old it is. Traffic is banded on a log scale because click distributions are heavily skewed and a linear band would put a 4-click page and a 400-click page in the same bucket.
The split happens inside each group. That way the difference measured later is the work, rather than the difference between a pricing page and a three-year-old blog post. Groups smaller than four pages are pooled together, because splitting a group of one is a coin flip that adds noise without adding comparability.
Pages are sorted into a fixed order before anything random happens, and each group draws from its own random stream. Adding a page to one group cannot change the assignment of a page in another, and re-running the same seed on the same site produces byte-identical rows.
| Dimension | Bands |
|---|---|
| Page type | Inferred from the URL and template. Unclassified pages form their own group rather than being guessed at. |
| Traffic | 0, 1-9, 10-99, 100-999, 1000+ clicks over the trailing 90 days |
| Age | Under 3 months, 3 to 12, 12 to 36, over 36, unknown |
| Minimum group | 4 pages. Smaller groups are pooled. |
| Control share | Never below 30%, whatever ratio is requested. |
This is the part that makes the final number worth anything. If pages could be moved between groups after the fact, a bad result could always be tidied into a good one, and nobody outside the company would ever know.
So the guarantee is not a promise in a policy document. It is a constraint in the database, and it applies to our own engineers with production access. The same trigger protects the audit log.
-- assignments are append only, enforced in PostgresCREATE TRIGGER assignment_no_update BEFORE UPDATE OR DELETE ON assignment FOR EACH ROW EXECUTE FUNCTION refuse_mutation();-- the control group can never fall below 30%ALTER TABLE experiment ADD CONSTRAINT holdout_floor CHECK (holdout_share >= 0.30);-- and the audit log is protected the same wayCREATE TRIGGER audit_event_no_update BEFORE UPDATE OR DELETE ON audit_event FOR EACH ROW EXECUTE FUNCTION refuse_mutation();
The headline figure is the difference in clicks between the two groups over the measurement window, indexed to the day of the split so that both groups start at the same place. The range around it is a 95% interval.
On the marketing pages we write that range in plain words, somewhere between +6% and +22%. In the report you get the exact bounds, the sample size, and the raw series for both groups as a CSV. It is the same number either way.
A wide interval is not a flaw in the measurement, it is the measurement. Eighty pages is a small sample and the range reflects that honestly. It narrows every week the experiment runs.
| In the report | On the site |
|---|---|
| +14.2% | +14.2% |
| 95% CI [+6.1%, +22.4%] | somewhere between +6% and +22% |
| n = 80 pages, 41 / 39 | eighty pages, split roughly in half |
| Confounders logged: 1 | Google shipped an update on 2 August and it moved both halves |
| Series export: CSV | download both lines |
With roughly 40 pages per arm over 90 days, a reliably detectable effect is around 0.6 standard deviations, which for click distributions usually means a 30 to 50 percent lift. A real 10 percent improvement would come back "cannot distinguish from chance" most of the time.
Defining failure as "the interval includes zero" would therefore mean giving away most second quarters while the product was working. Direction is binary, unarguable, verifiable by the customer, and roughly a coin flip under the null. The interval is still published on every report.
| Outcome | Definition |
|---|---|
| Ahead | Treated mean change above control mean change. Paid, continue. |
| Behind | Treated below control. Next quarter free, speculative rewrites reverted. |
| Void | Too few changes approved, a holdout page edited, the site migrated, or Search Console disconnected. No guarantee, reason stated. |
| Version | 1, dated 5 September 2026. A running experiment keeps the version it started under. |
Sixteen months of history, not two years. That is all Search Console keeps. An experiment you start today is the only clean baseline you will ever have for the pages you are about to change.
Pages are not fully independent. Refreshing one page can help or cannibalise its twin through internal links. Pairing inside a group reduces this but does not remove it, and we do not claim otherwise.
Ninety days is short for slow-moving pages. A page that gets four clicks a month will not produce a usable signal in a quarter, in either group. Those pages are reported as inconclusive rather than folded into the headline.
We cannot see AI answer surfaces directly. Search Console does not break them out. What we can measure is whether the pages being cited across those surfaces share a property, and freshness is the one that keeps showing up.