Support Desk

The problem

Most support tickets are routine: where's my order, refund a damaged item, cancel, reset a password. Each still waits hours for someone to look up the order, check the policy and click a button. Chatbots don't fix this. They can talk about refunds but can't safely do them, and when one guesses at policy or says "I've refunded you" without doing it, that creates a new ticket and real financial risk.

What I built

Six steps, three of which are plain code. The boxed steps are deterministic, so they can't be talked out of a decision.

1. Triage (AI): intent, order number, sentiment, confidence, escalation signals.
2. Knowledge (retrieval): finds the help-center articles that govern the answer.
3. Resolver (AI): drafts the reply and proposes at most one action from an allow-list of three.
4. Policy Guard (code): allowed, needs a person, or denied.
5. QA Critic (code): blocks replies that over-promise, misstate amounts, leak data or aren't grounded.
6. Executor (tool): acts once, re-checks hard limits itself, and logs everything.

The browser demo runs an 11-ticket inbox through the same logic. It has an approval queue, an audit log, an evals panel and a refund-limit slider. You can also write your own ticket and slip in a rogue $999 refund to watch the checks catch it.

Key decisions

1. The AI can only propose

Refunds happen only on the ticket owner's delivered orders, within 30 days, and never above the refundable balance. A confused or manipulated model can at most ask.

2. Spend above a limit needs a person

Refunds over $100 go to an approval queue. The limit is policy, not model judgment, and the launch plan starts it at $50 and raises it only after a clean week.

3. Denials feed back, so customers still get an answer

When the Policy Guard says no ("it has already shipped"), the reason goes back to the Resolver, which writes a kind, policy-grounded explanation. It gets two revisions; after that, a person takes over.

4. Safety-critical messages are templates

"A specialist will reply within 24 hours" is fixed text, never generated, so it can't promise something the system won't do.

What broke, and what I designed against

• The Executor trusted the Policy Guard too much. A unit test showed it would refund an undelivered order if anything ever slipped past policy. Fix: the Executor re-checks its own limits. Defense in depth, not a single gate.
• Duplicate tickets could double-refund. Idempotency keys plus a balance check mean the second request for the same broken item is denied and explained.
• A rogue amount shouldn't need luck to catch. Evals inject a $999 refund on a $45 order. The guard denies it, the Resolver proposes the correct $45, and only that executes.

How it's measured

Safety metrics are release gates: any value above zero blocks a deploy.

Auto-resolution is a band, not a number to maximize. Pushing it higher usually means escalating less, which trades safety for a vanity number.

What I'd do next

1. Run the evals on a live model, with 150+ labeled real tickets across intents.
2. Launch in shadow mode: run on real tickets, send nothing, and compare every decision with the human agent's.
3. Add embedding-based retrieval with its own retrieval eval, and multi-turn replies.

What I learned: agent design is mostly deciding who is allowed to decide. Putting the decision in code also made the system testable. Every rule became a unit test, and every "what if the model does something weird" became an injected fault.