Playbook

Letting an AI Agent Ship Bug Fixes, and the Gates That Made It Safe

My triage agent could tell an engineer what was broken and how to fix it. Then someone still had to write the fix, survive review, merge, deploy, and verify. This is what happened when I let a second agent carry eligible bugs the rest of the way (57 fixes landed in its first twelve business days, at a median of about $7 of compute per bug), and the human gates I refused to remove, including the one I removed anyway.

The situation

Before any agent existed, our bug pipeline started with a meeting. Every week, two product people, the QA manager, a QA engineer, the release manager, and an engineering director sat on a call and worked through bugs one at a time: watch the video, squint at the screenshots, figure out what the bug even was, agree on whether it was a bug at all, then haggle over the priority. We had priority guidelines, but product had learned that anything P3 or lower went into a dark hole to die (often true), so priorities got inflated whenever someone was emotionally attached to a thing getting fixed instead of never getting fixed. Then we assigned an owner, a team lead, who moved the bug to their own board (sometimes), then handed it to an engineer, and only then did anyone start writing the fix.

That meeting was the first thing I set out to solve. I wrote an agent to do the first pass instead, starting from the meeting's own first question ("is it ours": does this bug even belong to our pod), then how urgent, likely root cause, a fix plan. Real leverage, but it stops at a suggestion. In practice the diagnosis got paid for twice: once when triage figured out the cause, and again when the engineer who picked up the ticket re-derived it before touching code. Then the downstream tail: review send-backs, rebases, a deploy that takes most of an hour to reach anywhere visible, a QA pass. None of those hops is hard. Strung together, a one-line fix for a well-understood bug still cost a day of a real person's attention, most of it waiting and context-switching, not thinking.

The constraint

A machine writing code and running its own tests is one thing. A machine deciding its work belongs in the shared mainline is another. So the design started from what the agent is not allowed to do:

  • Not every bug qualifies. Only bugs already confirmed valid, owned by my team, contained to a single service, away from our highest-risk flow, and rated low or medium risk. The eligibility gate exists so the agent is never guessing on hard cases.
  • Merge required a human. When I wrote that rule down I used the word "permanently." Remember that word; it comes back.
  • Send-backs had to be normal, cheap events. Reviewers and QA bounce things. The system needed a bounded rework loop (try again, twice at most, then escalate to a person) rather than ignoring feedback or looping forever.

What I did

Once a bug clears eligibility, the flow runs as watched hops: the fix agent works in an isolated branch, writes the change plus a regression test, reviews its own diff against our house standards, and opens a merge request, then watches the GitLab pipelines (linter, unit tests, security scans) until they go green. From there an AI reviewer and a human both weigh in. Only a genuine send-back (an unresolved critical/high finding, or a QA failure) triggers a rework cycle, capped at two before a person takes over. After merge, a deploy watcher tracks the change until it's live, and two independent verification agents (one exercising the API behavior, one driving the actual UI) check the result against the bug's acceptance criteria. Even when both come back clean, the ticket doesn't close on its own: it waits for a human QA person, who reviews the fix and tests it by hand when they judge that's needed. I don't know how long that manual check will stick around (the same question hangs over the human approval on merge requests), but today it has the last word. Every hop lands in a durable ledger, and a separate step distills the ledger into lesson packs the next run reads. A mistake made once becomes a standing note before it's repeated.

Triaged bug in from the triage agent Fix branch · code · tests Review AI + human reviewers Rework bounded · two cycles max Merge gated · automatic human valve: repo approvals Deploy watch until it’s live on dev Dual verification API agent + UI agent Ticket resolved closed automatically Ledger outcome written down Lessons distilled for the next run human valve: QA verdict human valve: lesson approval fed into next run
The loop, hop by hop, and the three places a human deliberately stays in it.

Three decisions mattered more than the pipeline:

  • Shadow mode before live dispatch, every time. New pieces watched real tickets and classified what they would have done before being allowed to act. That caught a real near-miss: an early classifier read a severity-sounding word inside a reviewer's otherwise-fine sentence, misjudged the ticket as sent-back, and would have dispatched a live rework agent on a bug that didn't need one. The classifier was reworked so a stray word in a comment's prose can't flip a ticket to sent-back on its own. The bigger lesson: arm a system for hard-to-reverse actions as late as possible. Pre-arming "just in case" is what turns a shallow misclassification into a real unwanted action.
  • Two verifiers, because they fail differently. The API check and the UI check have each caught things the other missed. When the UI verifier has been wrong, it's been wrong the same way every time: passing an easier, adjacent scenario instead of the one reported. That divergence is now my clearest signal for where verification needs to get stricter.
  • And the rule that reversed: the merge. Within a day of writing "a human presses the button, permanently," I changed it. What actually mattered was that a human decides, not that a human clicks. The gate got stricter, not looser: nothing merges without every required approval and every human comment addressed, and it fails closed on any ambiguity. But once those conditions are genuinely met, making a person come back just to click was pure wait time with no judgment in it. Both versions of me were right: the first to refuse until the gate earned trust, the second to hand it over once it had.

What's happened so far

From the loop's own ledger, first twelve business days live:

  • 57 bugs fixed and landed. 46 already verified by a human tester, the rest awaiting a verdict. Every human-QA verdict rendered so far has been a pass.
  • About three-quarters went through with zero rework. Every rework that did trigger resolved on the first retry. None has hit the escalation cap.
  • Total model usage at API list rates: a bit over $300, median ~$7 per bug. And because the loop runs under a flat-fee subscription we already pay, incremental cash cost was actually zero.
  • The value side stays a labeled estimate: at a conservative half-day of engineer attention per small bug, roughly five and a half developer-weeks reclaimed in under three, without consuming any feature capacity. The loop delivered more story points of bug fixes in its third week than the pod's recent average delivers in a full two-week window. Context comparison, not like-for-like, and I flag it that way everywhere I show it.
  • The learning loop has first evidence: across the four problem areas old enough to measure, every bug arriving after that area's first hard-won lesson has gone through clean. Six straight first-pass fixes. Small sample, labeled as such.
From the loop's own ledger, two weeks in. Anonymized screenshots pending.

Three human links remain, all deliberate. A tester renders the final pass/fail on every fix. Our two-approval repos structurally require a human teammate's sign-off, so the loop can't approve its own work. And a distilled lesson only enters the packs after I approve it. The mechanical loop is closed. The judgment loop isn't, on purpose.

What I'd do differently

  • Keep shadow mode, on everything. The one item on this list that isn't a change: it's what I already did, and it's the single practice that caught the worst near-miss before it mattered. I'd never let a component take live actions without a watching-only phase first.
  • Name the one-way doors early. Stale state let the loop pull a ticket back out from under a human tester who already had it, more than once, through different holes. I found out the human way, when my dedicated QA person pinged me: "tickets keep moving from done back into merged. Two other testers pinged me to ask what's going on. Can you take a look?" Sealing holes one at a time didn't work. Declaring a rule did: a ticket that reaches QA, or that a human closes, is out of the loop's reach, period. I should have defined that boundary before going live, not after the third incident.
  • Don't write "permanently." The merge reversal taught me to state the invariant ("a human decides") rather than the mechanism ("a human clicks"). Mechanisms age. Invariants hold.
  • Expect collisions with the human workflow. An auto-filed follow-up ticket on QA failures looked obviously right, until a tester sent the same bug back by hand and one problem was tracked twice. It's now off behind a switch. Every automation that writes to a shared system needs a "what does the human already do here?" pass first.

The gap I haven't closed: the UI verifier's false-pass pattern. Until it fails closed on a scenario it can't reproduce (instead of substituting an easier one), human QA stays the final gate, and that's exactly the state I want it in.

Take what helps

The interesting design work was never the model call that writes the fix. It was deciding which surrounding decisions could be handed over safely, and building the harness (shadow mode, bounded retries, gates that fail closed) that lets you find out you're wrong before it matters.

If you take one thing: automate the mechanism only after you can state the invariant it must preserve. A closed loop with human valves beats an open loop and beats a blind one.

Your codebase and risk tolerance will vary. Take what helps, skip what doesn't.

Get new playbooks by email

One email when a new playbook or guide goes up, nothing else. Unsubscribe anytime.