Change Point Detection: date the shifts in your traffic

Change Point Detection finds the dates on which your daily clicks or impressions settled at a new level and stayed there, as opposed to single odd days. For each change it gives the average before and after, the size of the shift measured against day-to-day noise and a confidence score. Use it to date the start of a decline or a gain precisely before you go looking for the cause.

How shifts are found

1. Over the whole window, the report adds up each day's distance from the window's average, a cumulative sum or CUSUM. Where the level changes, that running total bends, and the day where it is furthest from zero is the most likely split.
2. Each side of a split must hold at least 7 days.
3. The shift is measured as the gap between the two averages divided by the pooled standard deviation of the two sides, in other words in units of normal day-to-day noise. A split is kept only if that number reaches the Sensitivity you set.
4. Confidence comes from a permutation test: the days of the segment are shuffled 200 times, and the confidence is the share of shuffles whose CUSUM range comes out smaller than the real one.
5. Each kept split is searched again on both sides (binary segmentation), up to 8 change points.

Sensitivity runs from 0.5 to 3 standard deviations, 1.5 by default. Lower values find smaller shifts. The date of a change point is the first day of the new level. NIST's handbook describes CUSUM as a method suited to small, sustained shifts in a mean, which is the case it handles here.

Reading the results

• Cards: Change points detected (with the number of segments), Largest shift as a percentage (with the series' day-to-day noise), Latest change point with its direction and confidence, and the change from the first segment's average to the last.
• Series with change points draws the daily values, a 7-day smoothed line and each segment's average, with a dashed line and the percentage change at every change point.
• CUSUM statistic shows the running total itself: a change of slope there is a change of level in the series.
• The table lists Date, Direction (Up or Down), Before average, After average, Change, Shift (SD units), Confidence, Days before and Days after.

Change the metric or the sensitivity after a run and the detection is redone in the browser. The run always covers at least 90 days; a shorter range is extended backwards and a note says which dates were used.

What it assumes

• Traffic moves in flat steps. The model is a series of levels plus noise. A steady climb has no single break, so it may be cut into several steps or produce none.
• The weekday pattern stays in. Weekly ups and downs count as noise, which makes small shifts harder to clear. On a site with a strong weekly rhythm, look at Time-Series Decomposition alongside it.
• Recent changes need time. With 7 days required on each side, a shift that began in the last six days cannot be found yet.
• Confidence is a guide, not a filter. It does not decide whether a point is listed, the shift size does. Shuffling also treats each day as independent, while traffic on consecutive days is related, so read the confidence as relative rather than as an exact probability.
• Biggest break first. Binary segmentation finds the strongest split before the smaller ones, so two shifts a week apart can be reported as one.

What to do with a change point

Line the date up with what happened around it: a deployment, a migration, a template change, an outage or a Google ranking update. Then narrow it down by running the report again with a filter on a folder, a country or a device to see which part of the site moved. When you know the reason, record it as an annotation.

For single unusual days rather than lasting moves, use Anomaly Detection. To be told about drops as they happen, set up alerts.