Support Desk
Agents that resolve support tickets end to end, refunds included. The model can only propose an action; plain code decides whether it happens.
For AI PM: safety metrics are release gates. Any wrong action, false promise or leaked card number above zero blocks a deploy.
For product: auto-resolution is a target band (50 to 65%), not a number to maximize, because pushing it higher means escalating less.
For technical PM: idempotent executor, defense in depth (the executor re-checks limits itself), bounded revision loop, per-step traces.
For ops: an approval queue for refunds over the limit, an audit log, and CSV export. The limit starts at $50 and rises after a clean week.
For programs: a launch plan that starts in shadow mode, comparing every decision with a human's before anything is sent.
| Eval tickets passing | 19 / 19 |
| Wrong or unauthorized actions | 0 |
| Promises that didn't happen | 0 |
| Must-escalate reaching a person | 100% |
| Resolved without a person | 63% |
Offline runs use a stand-in model with injected faults. They test the safety net, not live-model quality.