Agents
Teaching an AI Triage Agent When Not to Post
I let my support triage agent run itself on a schedule, and the engineering of "make it automatic" turned out to be the easy half. The hard half was restraint: measuring exactly where the agent's confidence couldn't be trusted, then building a gate around that blind spot so it posts only when it genuinely has something to say.
The situation
I have an AI agent that does the first-pass investigation on incoming support tickets. It reads a new ticket, looks up the customer's actual account state, identity, and eligibility through strictly read-only access to our internal services, checks the production logs, and writes a clearly AI-signed internal note so the engineer who picks the ticket up starts halfway through the work instead of cold. I wrote up how I built it separately. For months I ran it by hand, ticket by ticket, from a dashboard.
That worked, and it also missed the point. A person still had to remember the queue existed, open it, and kick the agent off one ticket at a time. The queue doesn't wait for anyone to remember. Support tickets land on senior engineers rotating through a support duty, pulling them off project work for detective work that mostly repeats, and the whole value of a first pass is that it's already done when the human arrives. So the next step was obvious: let it run unattended.
The step after that was the one that mattered, and it wasn't obvious at all.
The constraint
Three rules I refused to bend:
- Strictly read-only, still. It investigates and explains. It never changes an account. The day it proposes and runs fixes is a separate, human-gated project, not a feature creep into this one.
- Internal-only, enforced in code. Every note it posts is an internal comment the customer never sees. There is a tripwire test that breaks the build if a future change ever drops the internal-only marker. That property is too important to be a convention I hope holds.
- It only speaks when it has something worth saying. A confident, wrong triage on a real customer ticket is worse than silence, because it sends an engineer down the wrong path with borrowed conviction. Posting on every ticket was an explicit non-goal.
I also gave it a hard spending ceiling. Its model usage runs on its own isolated, capped budget, and each run is limited to a small number of tickets. A runaway can't quietly spend, and its cost is cleanly attributable.
What I did
The unusual part, and the part I'd defend hardest: I built the measurement before the autonomy. Most projects like this ship first and evaluate later, if ever. This one got an evaluation harness up front, a stratified set of real, labeled tickets, including the boring "no action needed" and "not ours" cases, that the agent is scored against automatically after every change.
The first honest run put its reliable accuracy in the low seventies. A baseline, not a victory lap. But the accuracy number wasn't the interesting finding. The calibration was. I didn't just measure whether the agent was right. I measured whether it was right when it said it was confident, and the answer had a shape: its confidence was worst calibrated exactly where it mattered most, on the cases where the right move was restraint. No action needed, route it elsewhere, not enough here to say. On those, it would confidently manufacture a cause when the honest answer was "I don't know yet."
So the gate is built around that specific blind spot. On a schedule, an unattended job pulls new tickets and runs each through the full read-only investigation. It drafts a note. Then the gate decides: if the agent is highly confident and has a concrete, actionable finding, the note posts as an internal comment. Everything else gets held, recorded, surfaced to me, and left untouched for a human. Posted, held, or skipped, every ticket lands in a durable audit log I review on a dashboard with a stop switch.
Autonomy arrived in increments, and I let evidence size each step. It deployed paused. Then dry-run, doing everything except post. The plan called for a long dry-run soak, but the queue is quiet, and a soak at that volume would have taken weeks to show me anything a handful of real, gated runs wouldn't. So I switched it on and let the gate, not the waiting, protect the tickets. Trust granted in steps, but steps sized to the data rather than ritual.
Two smaller decisions earned their keep. I paced the polling to the work that actually exists: my first instinct was to check every few minutes, which just pays to re-read the same quiet queue, so it runs roughly hourly with a look-back window wide enough that nothing slips through. And a separate model-as-judge pass checks whether each claim in a drafted note is actually supported by the data the agent retrieved, rather than plausibly invented.
What's happened so far
It runs live and unattended, posting only what clears the gate. A month in, a few things have already moved:
- The notification path got fixed. It now pings me on its own chat integration when it posts, holds, or fails, instead of leaning on a personal credential.
- The audit trail and the held-for-a-human queue moved from a page only I looked at to the team's shared dashboard. What the agent did, and didn't do, is visible to the people it works alongside, which is a quiet forcing function on quality.
- I built a background job that mines resolved tickets into worked examples for the agent's playbook, and then I deliberately switched its schedule off. A learning loop should run when a human is actually reading what it produces, not as unattended ceremony. Same restraint that governs posting.
Honest flags. The savings are still a model, not a number: I know the compute cost (cents to low dollars per ticket, hard-capped, self-recorded), but I haven't yet measured ticket volume through the rotation or the real per-ticket investigation time a human spends. And the calibration blind spot is mitigated, not cured. The gate keeps overconfident guesses off real tickets, but I skipped the long soak by choice, so the safety net right now is the gate plus my own reading of every early live post. Small numbers, watched closely, not trusted blind.
What I'd do differently
- Measure calibration, not just accuracy, from day one, on every agent. Accuracy told me the agent was mediocre. Calibration told me exactly where it was dangerous. Only one of those findings could be turned into a gate.
- Capture the baseline before launch. The two numbers that would turn "the savings are real" into a dollar figure (ticket volume, human minutes per first pass) were both measurable before I automated anything. I have to collect them now, after the fact, in a system that's already changing the behavior it would measure.
- Start on service credentials, not personal ones. The notification path launched on my own credential and had to be swapped later. Anything unattended should be born with its own identity.
- Pace to volume from the start. The every-few-minutes polling instinct came from what the system could do, not what the queue warranted. Cadence is a design decision, and the right input is the work, not the capability.
Take what helps
It is easy to make a model produce an answer for every input. The valuable, harder thing is building one that recognizes the cases where the right output is "hold this for a human," and then measuring whether it's actually right about that. If you're putting an agent anywhere near customer tickets, build the evaluation set first, check the calibration specifically on the cases where the honest answer is nothing, and gate on that. An agent that knows the edge of what it knows is worth far more than one that always has an answer.
Take what helps, skip what doesn't.