Signal or noise: the third run

What the third run is for

The second run gives you one post-change observation. The third gives you a second one, with nothing changed in between — which lets you see how much the number moves on its own.

If run three sits close to run two and both are clearly different from the baseline, you have a result worth acting on. If run three swings back toward the baseline, what you saw in run two was substantially variance, and the honest conclusion is that the change did not demonstrably work.

This is the run that gets skipped, because by run two the team already has the story it wants. Taking it is the single practice that most separates an AEO programme that compounds from one that accumulates unverified folklore.

Recording the outcome honestly

Write down which of three things happened: the change worked, it did not, or the result was inconclusive. Inconclusive is a legitimate and common outcome, and recording it as such is what stops it from being remembered as a success six months later.

Record the effect size too, not just the direction. A change that moved one prompt from absent to mentioned is a different fact from one that moved you from mentioned to recommended across four prompts, and treating them as the same is how a tactic gets over-generalised.

What to do with each outcome

If it worked: do the same class of change on the next-highest-value gap. You have evidence about your specific category, which is worth more than any generic best practice.

If it did not: revisit the diagnosis rather than the tactic. The usual cause is a correct fix applied to the wrong failure mode.

If it was inconclusive: either the effect is real but smaller than your variance, or there is no effect. Both point to the same next step — pick a higher-leverage change rather than repeating this one harder.

Lab: Run 3 of 3

Needs a free SolCrys account (no credit card). Spends run 3 of 3.

1. Change nothing since run two.
2. Trigger the run.
3. Compare all three: baseline, post-change, and confirmation.
4. Write one paragraph stating worked, did not work, or inconclusive — and the effect size.
5. Keep that paragraph. It is the first entry in your own evidence base.

If the honest answer is inconclusive, write inconclusive. That record is worth more than a flattering one.

Start Free — free plan, no credit card.

Check yourself

• Did run three confirm run two, or pull back toward the baseline?
• What effect size did you actually observe?
• Which of the three next steps does your outcome point to?

Go deeper

This lesson is the spine. These guides are the depth.

Continue

• Previous: Your second run
• Next: Turning this into an operating cadence