The rest of the site is written for someone deciding whether to try this. This page is written for whoever they hand it to afterwards. Nothing here is different from what the marketing pages claim, it is just spelled out.
Every page is sorted into a group by three things: what kind of page it is, roughly how much traffic it gets, and how old it is. Traffic is banded on a log scale because click distributions are heavily skewed and a linear band would put a 4-click page and a 400-click page in the same bucket.
The split happens inside each group. That way the difference measured later is the work, rather than the difference between a pricing page and a three-year-old blog post. Groups smaller than four pages are pooled together, because splitting a group of one is a coin flip that adds noise without adding comparability.
Pages are sorted into a fixed order before anything random happens, and each group draws from its own random stream. Adding a page to one group cannot change the assignment of a page in another, and re-running the same seed on the same site produces the same split, byte for byte. The seed and the group definitions are stored with the experiment.
To redraw a published split yourself: take every page in the report and put it in its matching set, named template|traffic band|age band exactly as the report shows. Pool every set with fewer than four pages into one set called pooled|small. Sort each set by page URL. Shuffle it with mulberry32, seeded with the 32-bit FNV-1a hash of the string <seed>:<set name>, using a Fisher-Yates walk from the end. The first round(n × ratio) pages of the shuffled set are the improved group, where ratio is the experiment’s treated ratio, capped at floor(n × 0.7) so the left-alone share of any set never drops below 30%. The rest are left alone. Every input is on the report.
| Dimension | Bands |
|---|---|
| Page type | Inferred from the URL and template. Unclassified pages form their own group rather than being guessed at. |
| Traffic | 0, 1-9, 10-99, 100-999, 1000+ clicks over the trailing 90 days |
| Age | Under 3 months, 3 to 12, 12 to 36, over 36, unknown |
| Minimum group | 4 pages. Smaller groups are pooled. |
| Control share | Never below 30%, whatever ratio is requested. |
Splitting forty pages in half leaves twenty a side, and the effect a run that size can reliably detect is roughly forty-five to seventy percent. A real improvement would come back indistinguishable from an ordinary quiet month almost every time. Refusing those websites was the honest answer to that, but it was not the only one.
So below forty pages the comparison is staggered rather than split. Every page is improved, in four groups drawn by lot: the first as soon as the plan is agreed, then on day 30, day 60, and day 90 once the window has closed. At any moment inside the ninety days some pages have been improved and some have not, so there is a live control throughout, and each page is also its own before-and-after. Nothing is held back permanently; the last group waits three months rather than forever.
The figure is the same per-page difference in differences, computed once per period against a treated set that grows, then pooled across periods in proportion to the information in each. A period with one page on a side is nearly uninformative and is weighted as such. The final period, once every group has gone live, has no control left and drops out on its own. The interval resamples pages rather than periods, because periods share pages and resampling them would understate the uncertainty.
To redraw a staggered plan yourself: take every page in the report, put it in its matching set, named template|traffic band|age band exactly as the report shows, and pool sets smaller than four into pooled|small. Sort each set by page URL. Shuffle it with mulberry32, seeded with the 32-bit FNV-1a hash of <seed>:wedge:<set name>, using a Fisher-Yates walk from the end. Then deal the shuffled pages into groups one at a time, continuing the count across sets in alphabetical order of set name, so the nth page overall, counting from zero, goes into group (n mod 4) + 1.
| Group | Work begins |
|---|---|
| 1 | The day the plan is agreed |
| 2 | Day 30 |
| 3 | Day 60 |
| 4 | Day 90, once the window has closed. The comparison for the whole run. |
| Minimum | 8 pages, so every group has something in it |
This is the part that makes the final number worth anything. If pages could be moved between groups after the fact, a bad result could always be tidied into a good one, and nobody outside the company would ever know.
So the guarantee is not a promise in a policy document. It is a constraint in the database, and it applies to our own engineers with production access. The same trigger protects the audit log.
-- assignments are append only, enforced in PostgresCREATE TRIGGER assignment_is_append_only BEFORE UPDATE OR DELETE ON assignment FOR EACH ROW EXECUTE FUNCTION reject_mutation();-- the control group can never fall below 30%ALTER TABLE experiment ADD CONSTRAINT experiment_holdout_floor CHECK (treated_ratio > 0 AND treated_ratio <= 0.7);-- and the audit log is protected the same wayCREATE TRIGGER audit_event_is_append_only BEFORE UPDATE OR DELETE ON audit_event FOR EACH ROW EXECUTE FUNCTION reject_mutation();
The headline figure is the difference in clicks between the two groups over the measurement window, indexed to the day of the split so that both groups start at the same place. It is taken within each matching set first, improved pages against left-alone pages of the same kind, traffic band and age, and then averaged across sets in proportion to their size. A set missing one side drops out; if no set has both sides, the run is void. Pooling every page together instead lets a strong season that lifts all pages alike read as a win, because the two groups weight busy and quiet sets slightly differently: simulated with no real effect, a season lifting every page 60% made the pooled figure say “ahead” 54.5% of the time instead of half, and within sets 51.6%. The range around the figure is a 95% interval from a bootstrap that resamples pages within each set and arm.
On the marketing pages we write that range in plain words, somewhere between +6% and +22%. In the report you get the exact bounds, the sample size, and the raw series for both groups as a CSV. It is the same number either way.
A wide interval is not a flaw in the measurement, it is the measurement. Eighty pages is a small sample and the range reflects that honestly. It narrows every week the experiment runs.
| In the report | On the site |
|---|---|
| +14.2% | +14.2% |
| 95% CI [+6.1%, +22.4%] | somewhere between +6% and +22% |
| n = 80 pages, 41 / 39 | eighty pages, split roughly in half |
| Confounders logged: 1 | Google shipped an update on 2 August and it moved both halves |
| Series export: CSV | download both lines |
With roughly 40 pages per arm over 90 days, a reliably detectable effect is around 0.6 standard deviations, which for click distributions usually means a 30 to 50 percent lift. A real 10 percent improvement would come back “cannot distinguish from chance” most of the time.
Defining failure as “the interval includes zero” would therefore mean giving away most second quarters while the product was working. Direction is binary, unarguable, verifiable by the customer, and roughly a coin flip under the null. The interval is still published on every report.
| Outcome | Definition |
|---|---|
| Ahead | Treated mean change above control mean change. Paid, continue. |
| Behind | Treated below control, or a run whose direction turned on a single page the concentration cap reversed: settled as behind in the customer’s favour. Next quarter free. The original text of any page, on request. |
| Void | Fewer than half of the proposed improvements approved, a holdout page edited, the site migrated, or Search Console disconnected. No guarantee, reason stated. |
| Version | 2, dated 18 September 2026. A running experiment keeps the version it started under. |
Sixteen months of history, not two years. That is all Search Console keeps. An experiment you start today is the only clean baseline you will ever have for the pages you are about to change.
Pages are not fully independent. Refreshing one page can help or cannibalise its twin through internal links. Pairing inside a group reduces this but does not remove it, and we do not claim otherwise.
Ninety days is short for slow-moving pages. A page that gets four clicks a month will not produce a usable signal in a quarter, in either group. Those pages are reported as inconclusive rather than folded into the headline.
We cannot see AI answer surfaces directly. Search Console does not break them out. What we can measure is whether the pages being cited across those surfaces share a property, and freshness is the one that keeps showing up.