What we expect to get wrong
Issue zero. Before any experiment has finished, a list of the ways the first ones will probably fail, so you can check us against it later.
This letter will mostly be about experiments that came out behind. That is not a marketing angle. It is what the product does: half of a site is rewritten, half is left alone on purpose, and ninety days later the two halves are compared. Some of those comparisons will show the rewritten half ahead. Some will not. Both get a public page with the same layout and the same prominence, and both get written up here.
Since nothing has finished yet, issue zero is a list of predictions. Not about results, about failures. These are the ways we expect the first experiments to go wrong, written down now so that when one of them happens you can hold the letter up against it.
One. Quiet pages will not move. A page that gets three visits a week cannot show a difference in ninety days, whichever half it is in. The measurement leaves those pages out of the headline and reports them as inconclusive. On a site where most pages are quiet, that will leave a headline built on a dozen pages, and the range around it will be wide. We would rather publish a wide range than a narrow one we made up.
Two. A rewrite will make a page worse. Not often, and not on purpose, but a section that read well to us will lose something the old version had, and the number will say so. When it does, the change comes back off, the page goes back to what it was, and the report lists it. If it happens to the same kind of page twice, the picker stops choosing that kind.
Three. Something outside the site will move both halves at once. A Google update, a season, a competitor going quiet. This is the case the control group exists for, and it is also the case where a reader looking only at the rewritten line will see a jump and credit us for it. The report shows both lines for that reason, and the event gets logged on the day it happens.
Four. A site owner will edit a page we were leaving alone. Usually to fix a typo, sometimes because they forgot. The crawl will see the change, the page will come out of both halves from that day, and the report will say which page and when. If it happens to enough pages the experiment is void rather than lost, and the guarantee does not apply. We will say that plainly on the setup screen before anyone agrees to anything, and we expect it to happen anyway.
Five. The interval will include zero more often than anyone would like. Ninety days on forty pages is a small experiment. A result that says the rewritten half is ahead by nine percent, somewhere between minus two and plus twenty, is an honest result and an unsatisfying one. We will publish it as it is. The guarantee turns on which side of zero the best estimate landed, not on the range clearing it, and the report will say which of the two it is looking at.
Six. Someone will ask for the version without the control group. It would be cheaper, faster and easier to sell, and it is the one thing we will not build. The moment there is a version without a holdout, everything on the site is a claim again.
When one of these happens, it goes in the next letter with the site's published report beside it, if the owner chose to publish. If you want to be told when that is, the free check on the site has no email field, on purpose. Run it on your own site instead, and come back to this page.