04 · Results
Did it work?
14 days after each release we compare its goal step and the goal itself with the baseline. A change that didn't help is still a result: it tells us what not to try again.
Demo data: connect Mixpanel for real events · Text written by templates, not Claude
Goal since 18 Sept1.8% → 3.7%
Shipped4
Improved1
No change · unclear2 · 0
Measuring1
- ● Measuring
Larger answer buttons on mobile
Released 23 Oct · result on 13 Nov · day 14 of 14Measure Before After Change People Finish the first test (goal step) 43.6% 41% too early 1,702 → 251 Trial users ask for paid access (goal) 3.7% 2% too early 1,702 → 251 What we learnedToo early to say. Interim numbers are shown for the days measured so far.NextCheck again on 13 Nov 2026. - ✓ Improved
Show the paywall link on results
Released 2 Oct · measured 19 Nov over 14 daysMeasure Before After Change People Send a paid-access request (goal step) 10.2% 23.3% +13.1 pts 313 → 163 Trial users ask for paid access (goal) 2% 3.7% +1.7 pts 1,835 → 937 90% interval 7.1 to 19.4 pts · P(improved) 100%
What we learnedImproved: more people now send a paid-access request (10% before, 23% after). "Show the paywall link on results" worked, and we are confident the change caused it.NextKeep it. - ○ No change
Hide the timer in practice mode
Released 25 Sept · measured 19 Nov over 14 daysMeasure Before After Change People Finish the first test (goal step) 43% 43.8% +0.8 pts 1,811 → 962 Trial users ask for paid access (goal) 2% 3.4% +1.4 pts 1,811 → 962 Come back on day 2 (next step, guardrail) 47.8% 49.4% +1.6 pts 90% interval -2.5 to 4.0 pts · P(improved) 66% · OVERLAPPING RELEASE: goal credit is split
What we learnedNo change: about the same share of people finish the first test (43% before, 44% after). "Hide the timer in practice mode" was not the reason people stop at "Finish the first test". We won't suggest it again. Another change went live around the same time, so part of this may come from it.NextKeep it or undo it; either is fine. - ○ No change
Shorter signup form
Released 18 Sept · measured 19 Nov over 14 daysMeasure Before After Change People Start the first test (goal step) 43.9% 42.5% −1.5 pts 4,205 → 2,167 Trial users ask for paid access (goal) 1.8% 2% +0.2 pts 1,847 → 920 Finish the first test (next step, guardrail) 43.1% 42.6% −0.5 pts 90% interval -3.6 to 0.7 pts · P(improved) 13% · OVERLAPPING RELEASE: goal credit is split
What we learnedNo change: about the same share of people start the first test (44% before, 43% after). "Shorter signup form" was not the reason people stop at "Start the first test". We won't suggest it again. Another change went live around the same time, so part of this may come from it.NextKeep it or undo it; either is fine.