What we noticed
From Nshir's synthetic data, last 28 days. Each insight names the goal step it affects. Last run 19 Nov 2026, 17:21 UTC.
Demo data: connect Mixpanel for real events · Text written by templates, not Claude
From Nshir's synthetic data, last 28 days. Each insight names the goal step it affects. Last run 19 Nov 2026, 17:21 UTC.
Demo data: connect Mixpanel for real events · Text written by templates, not Claude
1,096 of the 1,922 people who started did not finish (57%). 61% of them left before question 3. People on phones quit this early more often than people on desktop (38% vs 25%).
Why it matters: People who quit here never reach the steps that lead to "More trial users ask for paid access", so fixing the start of "Finish the first test" lifts every step after it.
Possible reasons (not confirmed yet)
Measured numbers
Start the first test → Finish the first test: 826 of 1922 (43%) continue, 57% drop · 1096 of 1922 starters (57%) left without finishing; 667 (34.7% of starters) left before question 3. Mobile 37.9%, desktop 24.8%.
How sure we are. Counted from reliable events with plenty of people, over one period.
S2 · funnel_drop · sample 1922 · 2026-09-29 to 2026-10-27
-- funnel_timelines.sql
-- params: ["nshir","2026-09-29T00:00:00.000Z","2026-10-27T00:00:00.000Z",["internal_qa_1","internal_qa_2"],"site_visited",["site_visited","test_started","test_finished","session_started","locked_feature_viewed","paid_access_requested"],"2026-10-27T00:00:00.000Z"]
-- Per-user timelines for the funnel. Cohort: users whose first entry event falls in [$2, $3).
-- $1 product_id, $2 cohort start, $3 cohort end, $4 excluded distinct_ids,
-- $5 entry event, $6 funnel event names, $7 horizon end (events after it are ignored)
with cohort as (
select distinct_id,
min(time) as first_time,
(array_agg(props->>'platform' order by time))[1] as platform,
(array_agg(props->>'browser' order by time))[1] as browser
from events
where product_id = $1 and name = $5 and distinct_id <> all($4::text[])
group by distinct_id
having min(time) >= $2::timestamptz and min(time) < $3::timestamptz
)
select c.distinct_id, c.platform, c.browser, e.name, e.time
from cohort c
join events e on e.product_id = $1 and e.distinct_id = c.distinct_id
where e.name = any($6::text[]) and e.time >= c.first_time and e.time < $7::timestamptz
order by c.distinct_id, e.time
S3 · progress_dropoff · sample 1922 · 2026-09-29 to 2026-10-27
-- progress_dropoff.sql -- params: ["nshir","2026-09-29T00:00:00.000Z","2026-10-27T00:00:00.000Z",["internal_qa_1","internal_qa_2"],"test_started","answer_selected","question","test_finished"] -- Where inside a step users stop. Starters of $5 in [$2, $3); last value of the progress -- property $7 on event $6 after starting; whether they reached $8. -- $1 product_id, $4 excluded distinct_ids with starters as ( select distinct_id, min(time) as t0, (array_agg(props->>'platform' order by time))[1] as platform from events where product_id = $1 and name = $5 and distinct_id <> all($4::text[]) group by distinct_id having min(time) >= $2::timestamptz and min(time) < $3::timestamptz ), progress as ( select s.distinct_id, coalesce(max((e.props->>$7)::int), 0) as last_step from starters s left join events e on e.product_id = $1 and e.distinct_id = s.distinct_id and e.name = $6 and e.time >= s.t0 group by s.distinct_id ), finished as ( select distinct s.distinct_id from starters s join events e on e.product_id = $1 and e.distinct_id = s.distinct_id and e.name = $8 and e.time >= s.t0 ) select s.platform, p.last_step, (f.distinct_id is not null) as finished, count(*)::int as users from starters s join progress p using (distinct_id) left join finished f using (distinct_id) group by 1, 2, 3 order by 1, 2
Run run_9beac44cdca34f40 · context v1 · prompt S3/v2 · model offline-template
Before 22 Oct this happened in 0.4% of attempts. Since then it has happened 35 times out of 172 attempts, to 35 people.
Why it matters: People who hit this error lose their progress on "Finish the first test", and most of them do not try again.
Possible reasons (not confirmed yet)
Measured numbers
exam_submit_failed 0.4% → 20.3% of attempts since 2026-10-22 (35 of 172) vs the 28-day baseline
How sure we are. A clear jump compared with the 28 days before; the cause is not confirmed yet.
S4 · error_spike · sample 172 · 2026-10-22 to 2026-10-27
-- error_rate_daily.sql
-- params: ["nshir","2026-09-01T00:00:00.000Z","2026-10-27T00:00:00.000Z",["internal_qa_1","internal_qa_2"],"exam_submit_failed","exam_submitted"]
-- Daily failure rate of an action: failures $5 vs successes $6, from $2 to $3.
-- $1 product_id, $4 excluded distinct_ids
select date_trunc('day', time)::date::text as day,
count(*) filter (where name = $5)::int as failures,
count(*) filter (where name = $6)::int as successes,
count(distinct distinct_id) filter (where name = $5)::int as failed_users
from events
where product_id = $1 and name in ($5, $6) and time >= $2::timestamptz and time < $3::timestamptz
and distinct_id <> all($4::text[])
group by 1
order by 1
Run run_9beac44cdca34f40 · context v1 · prompt S3/v2 · model offline-template
24% of iPhone users on Safari who reached "Start the first test" went on to "Finish the first test" (144 of 614), compared with 52% of everyone else.
Why it matters: IPhone users on Safari are 32% of people at this step. At everyone else's rate, about 176 more of them would reach "Finish the first test".
Possible reasons (not confirmed yet)
Measured numbers
Start the first test → Finish the first test: platform=ios, browser=safari 23.5% (144 of 614) vs 52.1% for everyone else
How sure we are. Enough people in this group; checked over one period.
S6 · segment_gap · sample 614 · 2026-09-29 to 2026-10-27
-- segment_step_gap.sql
-- params: ["nshir","2026-09-29T00:00:00.000Z","2026-10-27T00:00:00.000Z",["internal_qa_1","internal_qa_2"],"test_started","test_finished",0]
-- Conversion from $5 to $6 per platform and browser. Users whose first $5 is in [$2, $3);
-- converted if $6 happens after it, at least $7 calendar days later, before $3.
-- $1 product_id, $4 excluded distinct_ids
with f as (
select distinct_id, min(time) as t,
(array_agg(props->>'platform' order by time))[1] as platform,
(array_agg(props->>'browser' order by time))[1] as browser
from events
where product_id = $1 and name = $5 and distinct_id <> all($4::text[])
group by distinct_id
having min(time) >= $2::timestamptz and min(time) < $3::timestamptz
),
c as (
select f.distinct_id
from f
join events e on e.product_id = $1 and e.distinct_id = f.distinct_id and e.name = $6
and e.time >= f.t and e.time < $3::timestamptz
and date_trunc('day', e.time) >= date_trunc('day', f.t) + make_interval(days => $7::int)
group by f.distinct_id
)
select coalesce(f.platform, 'unknown') as platform, coalesce(f.browser, 'unknown') as browser,
count(*)::int as n, count(c.distinct_id)::int as converted
from f left join c using (distinct_id)
group by 1, 2
order by 1, 2
Run run_9beac44cdca34f40 · context v1 · prompt S3/v2 · model offline-template
63% of Safari users on desktop who reached "Start the first test" went on to "Finish the first test" (78 of 124), compared with 42% of everyone else.
Why it matters: If we learn what helps them, it may work for everyone.
Possible reasons (not confirmed yet)
Measured numbers
Start the first test → Finish the first test: platform=desktop, browser=safari 62.9% (78 of 124) vs 41.6% for everyone else
How sure we are. Enough people in this group; checked over one period.
S7 · segment_gap · sample 124 · 2026-09-29 to 2026-10-27
-- segment_step_gap.sql
-- params: ["nshir","2026-09-29T00:00:00.000Z","2026-10-27T00:00:00.000Z",["internal_qa_1","internal_qa_2"],"test_started","test_finished",0]
-- Conversion from $5 to $6 per platform and browser. Users whose first $5 is in [$2, $3);
-- converted if $6 happens after it, at least $7 calendar days later, before $3.
-- $1 product_id, $4 excluded distinct_ids
with f as (
select distinct_id, min(time) as t,
(array_agg(props->>'platform' order by time))[1] as platform,
(array_agg(props->>'browser' order by time))[1] as browser
from events
where product_id = $1 and name = $5 and distinct_id <> all($4::text[])
group by distinct_id
having min(time) >= $2::timestamptz and min(time) < $3::timestamptz
),
c as (
select f.distinct_id
from f
join events e on e.product_id = $1 and e.distinct_id = f.distinct_id and e.name = $6
and e.time >= f.t and e.time < $3::timestamptz
and date_trunc('day', e.time) >= date_trunc('day', f.t) + make_interval(days => $7::int)
group by f.distinct_id
)
select coalesce(f.platform, 'unknown') as platform, coalesce(f.browser, 'unknown') as browser,
count(*)::int as n, count(c.distinct_id)::int as converted
from f left join c using (distinct_id)
group by 1, 2
order by 1, 2
Run run_9beac44cdca34f40 · context v1 · prompt S3/v2 · model offline-template
In the last 7 days, 40 of 422 people had this recorded twice, less than a second apart.
Why it matters: Every number built on it for "Finish the first test" is off until this is fixed, so other decisions on this step rest on wrong data.
Possible reasons (not confirmed yet)
Measured numbers
answer_selected fired twice within 1 s for 40 of 422 users (9.5%) in 7 days; 244 duplicate events
How sure we are. Counted directly from the raw events.
S11 · double_fire · sample 422 · 2026-10-20 to 2026-10-27
-- double_fire.sql
-- params: ["nshir","2026-09-29T00:00:00.000Z","2026-10-27T00:00:00.000Z",["internal_qa_1","internal_qa_2"]]
-- Same user, same event within 1 second, in the 7 days before $3.
-- $1 product_id, $4 excluded distinct_ids ($2 unused, kept for a uniform signature)
with e as (
select distinct_id, name, time,
lag(time) over (partition by distinct_id, name order by time) as prev
from events
where product_id = $1 and time >= $3::timestamptz - interval '7 days' and time < $3::timestamptz
and distinct_id <> all($4::text[]) and $2::timestamptz is not null
)
select name,
count(*)::int as events,
count(*) filter (where prev is not null and time - prev < interval '1 second')::int as dup_events,
count(distinct distinct_id)::int as users,
count(distinct distinct_id) filter (where prev is not null and time - prev < interval '1 second')::int as dup_users
from e
group by name
order by name
Run run_9beac44cdca34f40 · context v1 · prompt S3/v2 · model offline-template