Second Avenue
A consignment retailer guarantees each consignor a target price and pays the difference when a contract misses. About 15% miss, costing roughly $3.6M a year, and nobody notices until the contract expires. I designed a flag to catch them early, pre-registered how I'd judge it, and the backtest said my first bet failed. The fix cut false alarms from 80% to 36%.
Test the riskiest assumption first
After discovery, sizing and a PRD, the riskiest assumption was clear: that a simple pacing flag catches underperforming contracts weeks earlier without burying Ops in false alarms. That can be tested with a throwaway prototype before asking engineering for a production backlog, so I did that first.
I committed the go-live criteria and the rule for choosing settings before running anything. Then I backtested 40 threshold and day-floor settings on a year of contracts.
The verdict: the bet doesn't hold as specified
| Go-live criterion | Target | Pilot default | Today's practice | Met |
|---|---|---|---|---|
| Median warning before expiry | 30+ days | 75 days | 0 days | Yes |
| Recall at the halfway point | 70%+ | 68% | 0% | Narrowly no |
| False-positive rate | 40% or less | 80% | 6% | No |
| New flags per week | 25 or fewer | 23 | 2 | Yes |
The flag catches far more, far earlier: 80% of underperformers at some point, a median of 75 days before expiry, against 33% caught by today's final-week review. But about 18 of its 23 weekly flags are false alarms. At 10 minutes each, that's three hours of Ops time a week spent mostly on noise. None of the 40 settings met the false-positive guardrail.
Why it failed: early pacing is mostly timing noise
| Days into the contract | 15 to 30 | 30 to 45 | 45 to 60 | 60 to 75 | 75 to 90 |
|---|---|---|---|---|---|
| Share of flags that were right | 19% | 26% | 38% | 56% | 90% |
Being behind early mostly means the big items haven't sold yet, not that they won't. With 15% of contracts underperforming, a first-month flag is barely better than picking at random. Requiring the flag to persist didn't help: holding it for 7 days cut false alarms only from 80% to 77% while lead time fell from 75 to 43 days. The problem wasn't one segment either; every channel and contract size sat at 77 to 83%.
The fix: count what's still on the shelf
Measuring progress only in dollars already collected ignores everything still listed. I pre-registered a second experiment, tuning on older contracts and testing once on the most recent six months, and compared three designs.
| On the test contracts | Bar | A: linear pacing | B: projects unsold inventory | C: B plus watch-and-act |
|---|---|---|---|---|
| Median lead time | 30+ days | 56 | 30 | 30 |
| Recall at halfway | 70%+ | 59% | 38% | 44% |
| False-positive rate | 40% or less | 66% | 37% | 36% |
| New flags per week | 25 or fewer | 18 | 9.5 | 8.5 |
Counting unsold inventory solved the false-alarm problem and halved the weekly workload. What still fails is recall at halfway, and the sweep shows why: in this book, reaching 70% by day 45 means accepting about 70% false alarms. That's a trade-off for the business to choose, so the PRD now presents it as one instead of promising both.
The question layer, evaluated with cost
Ops and Finance can ask about flagged contracts in plain language. The model gets six governed tools and no free-form SQL. I scored it on 30 questions (rewordings, in-scope, out-of-scope and ambiguous ones), three runs per model, and a question counts only if it passes every run.
| Model | Passed every run | Tokens per run | Cost per run |
|---|---|---|---|
| Claude Sonnet | 30 of 30 | 402,227 | $1.08 |
| Claude Haiku | 29 of 30 | 327,758 | $0.05 |
Haiku missed one question about which flagged contracts can still go to auction, passing it in 1 of 3 runs. At about a twentieth of the cost, the call depends on how costly that miss is, which is exactly the conversation I'd want to have with the Ops lead.
Backlog and roadmap
Only after the prototype did I write the backlog: six epics with functional stories traced to the revised PRD, laid out as Now, Next and Later, and synced to GitHub Issues. The parts engineering owns, like landing real source systems, reconciliation and access control, were deliberately left out of the prototype.
The most useful thing a prototype did here was say no before engineering spent a quarter on it.