Experimentation: test whether an SEO change moved your traffic

Experimentation answers the question every SEO change raises: did it work, or did the whole site move? You define a treatment group (pages you changed), a control group (similar pages left alone) and a launch date. AutoPSEO compares each group after launch with its own 28 days before and tests whether the gap between them beats everyday noise. Because the control shares seasonality, Google updates and demand swings, this isolates your change far better than a plain before-and-after.

Create an experiment

Press New and fill in:

• Name and Hypothesis: what you changed and what you expect, for example "FAQ schema on product guides lifts CTR by 10%".
• Start date: the day the change went live. End date is optional; without one, the experiment runs to the latest day with data.
• Metric: Clicks, Impressions, CTR or Position. All four are reported; this one decides the winner.
• Treatment group and Control group: each a List of URLs (up to 500 absolute URLs) or a URL pattern (regex), such as ^https://www.example.com/guides/.

Two shortcuts help with groups. Upload a CSV with URL groups reads a file with url and group columns; the first group in the file becomes the control and the second the treatment. Auto-suggest groups loads your top pages of the last 28 days and proposes groups by first folder (any folder with at least three pages), ready to use as patterns.

Choose a control group that resembles the treatment: same template, similar traffic, same kind of search intent. Assign pages to the groups before you look at results, not after.

How the result is calculated

1. Windows. The pre-period is the 28 days before the start date; the post-period runs from the start date to the end date or the newest day available.
2. Indexing. Each group's daily values after launch are divided by that group's own average day before launch, so 1.0 means "as before". This puts a small treatment group and a large control group on the same scale.
3. Uplift. The relative uplift is the treatment's average index minus the control's: +0.12 means the treatment did 12 percentage points better relative to its own baseline.
4. Test. Welch's t-test compares the two groups' daily indexed values without assuming equal variance, giving a p-value, a 95% confidence interval and Cohen's d as an effect size (negligible below 0.2, small below 0.5, medium below 0.8, large from 0.8).
5. Winner. The result is significant when p is below 0.05; the Winner is then the treatment or the control (for position, lower is better). Otherwise it is Inconclusive.
6. Cumulative difference. Day by day since launch, AutoPSEO sums the treatment's actual value minus what it would have been had it followed the control (its pre-launch average × the control's index): the extra clicks, or whatever metric you chose, the change produced.

Statistical Test Results also shows the Minimum detectable effect, about 2.8 standard errors of the difference: the smallest uplift this experiment could reliably detect at 80% power. When a result is not significant, compare your expected effect with it before concluding the change did nothing.

Read the results

Open an experiment from the list to see Relative uplift, each group's change against its pre-launch period, the p-value and the Winner. The charts show both groups' daily index with the launch marked, and the cumulative difference since launch. Tables break the result down by group, by metric with confidence intervals, by page (the 300 pages with the most post-launch volume) and by day. Mark an experiment completed when you are done; it stays in the list with its results.

When to trust it

The page warns you when:

• fewer than 14 days have passed since launch;
• fewer than 14 pre-launch days are available;
• with clicks or impressions as the metric, either group averaged fewer than 5 a day before launch, so day-to-day noise dominates;
• a group had no traffic at all before launch, so its change cannot be measured.

Even a significant result is observational, not a randomised trial: another change shipped to one group, or a ranking update that hit the two templates differently, can masquerade as your effect. Keep an annotation of the launch and of anything else that happened during the test.

Testing without upsetting Google

This report compares groups of real pages, so you do not need to serve variants to Google. If you do run an A/B test with variant URLs, Google asks you not to show Googlebot different content than users, to point variants at the original with rel="canonical", to use temporary (302) redirects, and to end the test once you have a result (A/B testing best practices for Search).

How AutoPSEO helps

The content engine's refresh and optimisation tasks change one page at a time (keeping articles fresh). To judge whether those changes pay off, put the refreshed articles in a treatment group, similar untouched articles in the control, and use the date the first refresh went live.