Engine settings
How the engine is learning
Calibration adjusts the numbers, lessons adjust the reasoning, prompt versions adjust the instructions. Each change is logged and can be rolled back.
SHIPPED THAT IMPROVED33%
BRIER SCORE0.44
MEDIAN LIFT ERROR3.5 pts
EFFORT ERROR1.0 d
APPROVAL RATE—
INSIGHT → SHIPPED—
3 measured results. Multipliers stay at 1.0 until 5 results, shrink toward 1.0, stay within 0.5–1.5 and move at most 20% per update.
Calibration
| SCOPE | MULTIPLIER | VALUE | RESULTS | UPDATED |
|---|---|---|---|---|
| category: ux | kI | 0.870 | 3 | 19 Nov 2026, 17:21 UTC |
| category: ux | kC | 0.926 | 3 | 19 Nov 2026, 17:21 UTC |
| category: ux | kE | 1.375 | 3 | 19 Nov 2026, 17:21 UTC |
Change log
| category: ux | kE | 1.400 → 1.375 | 19 Nov 2026, 17:21 UTC | |
| category: ux | kC | 0.973 → 0.926 | 19 Nov 2026, 17:21 UTC | |
| category: ux | kI | 1.087 → 0.870 | 19 Nov 2026, 17:21 UTC | |
| category: ux | kE | 1.167 → 1.400 | 19 Nov 2026, 17:20 UTC | |
| category: ux | kC | 1.019 → 0.973 | 19 Nov 2026, 17:20 UTC | |
| category: ux | kI | 1.200 → 1.087 | 19 Nov 2026, 17:20 UTC | |
| category: ux | kE | 1.000 → 1.167 | 19 Nov 2026, 17:20 UTC | |
| category: ux | kC | 1.000 → 1.019 | 19 Nov 2026, 17:20 UTC | |
| category: ux | kI | 1.000 → 1.200 | 19 Nov 2026, 17:20 UTC |
Factor weights
Live: impact ×0.6, severity ×0.4. A proposal appears after 30 measured results.
Detector precision
| DETECTOR | INSIGHTS | LED TO AN IMPROVED RESULT |
|---|---|---|
| funnel_drop | 2 | 0 |
| progress_dropoff | 1 | 0 |
| error_spike | 1 | 0 |
| segment_gap | 6 | 0 |
| double_fire | 1 | 0 |
| untracked_event | 1 | 0 |
| missing_property | 1 | 0 |
Lessons
| LESSON | SCOPE | WEIGHT | STATUS | |
|---|---|---|---|---|
| On "Start the first test", "Shorter signup form" did not move conversion (-1.5 pts, n=6372). | step: s2 | 1.00 | active | |
| On "Finish the first test", "Hide the timer in practice mode" did not move conversion (0.8 pts, n=2773). | step: s3 | 1.00 | active | |
| On "Send a paid-access request", "Show the paywall link on results" moved conversion (13.1 pts, n=476). | step: s6 | 1.00 | active |
Prompt versions
| STAGE | VERSION | STATUS | EVAL | MODEL | |
|---|---|---|---|---|---|
| FUNNEL | v1 | live | — | claude-haiku-4-5 | |
| PROFILE | v1 | live | — | claude-haiku-4-5 | |
| S10 | v1 | retired | — | claude-opus-5-5 | |
| S10 | v2 | live | — | claude-opus-5-5 | |
| S11 | v1 | live | — | claude-opus-5-5 | |
| S3 | v1 | retired | — | claude-opus-5-5 | |
| S3 | v2 | live | — | claude-opus-5-5 | |
| S4 | v1 | live | — | claude-opus-5-5 | |
| S5 | v1 | retired | — | claude-opus-5-5 | |
| S5 | v2 | live | — | claude-opus-5-5 | |
| S6 | v1 | live | — | claude-opus-5-5 | |
| S7 | v1 | retired | — | claude-haiku-4-5 | |
| S7 | v2 | live | — | claude-haiku-4-5 | |
| S8 | v1 | retired | — | claude-haiku-4-5 | |
| S8 | v2 | live | — | claude-haiku-4-5 | |
| S9 | v1 | live | — | — | |
| preamble | v1 | retired | — | — | |
| preamble | v2 | live | — | — |
A candidate needs an eval pass (pnpm eval <stage>) and one shadow run before approval. Live prompts are never edited in place.
Model calls
| STAGE | CALLS | FAILED | AVG LATENCY | COST |
|---|---|---|---|---|
| S10 | 3 | 0 | 324 ms | $0.000 |
| S11 | 3 | 0 | 322 ms | $0.000 |
| S3 | 1 | 0 | 393 ms | $0.000 |
| S4 | 1 | 0 | 378 ms | $0.000 |
| S5 | 12 | 0 | 694 ms | $0.000 |
| S6 | 12 | 0 | 344 ms | $0.000 |
| S7 | 12 | 0 | 322 ms | $0.000 |