Workflow

Teaching the AI My Playbook: Skills Instead of Prompts

Prompts are ephemeral. You explain your standards, get a good result, and the explanation evaporates with the session. This is about the alternative: writing down how I actually decide things, once, as executable playbooks the assistant runs by name. Around twenty of them now. One of them wrote the first draft of this article.

The situation

Working with an AI assistant conversationally has a failure mode nobody warns you about: the better your session goes, the more you lose when it ends. I'd spend twenty minutes teaching it exactly how I want an epic pre-shaped (check the codebase first, don't ask product anything the repo can answer, tier the questions by what actually blocks a start), get a genuinely good result, and then watch all of that instruction vanish with the session. Next week, same task, I'm teaching it again. Slightly differently, because I'm human, which means the output is slightly different too.

That's the same problem organizations have with tribal knowledge, compressed into a fast loop. The way I size an epic, the way I structure a ticket, the way I decide which bug-board items are ours: those procedures exist, they're even pretty good, but they lived nowhere except my habits. Anyone (human or machine) doing the task without me got a plausible-looking version that was wrong in the details, and the details are where the judgment is.

The constraint

I didn't want a knowledge base nobody reads. Here's a claim I'll defend: documentation has never had a reader that could act on it, and that's most of why engineering leaders never write it. Every runbook and wiki page I've ever written was addressed to a reader who might never show up, would skim if they did, and still had to do the whole job by hand afterward. Writing it was a bet on someone else's future diligence, and busy leaders lose that bet often enough that most of us rationally stop placing it. An assistant changes the economics, because for the first time the reader can execute the document. So whatever I built had to be something the assistant executes, not something it might or might not consult.

What I did

I've been converting my recurring director workflows into what my AI tooling calls skills: packaged instruction files the assistant loads when I say a trigger phrase. "Pre-shape this epic." "Recreate the notification emails for this order." "Triage the bug board." No retyping, no re-explaining. The file is the explanation, versioned and permanent.

Mechanically a skill is mundane: a markdown file, sometimes with a small script attached. The content is the interesting part, because a useful skill needs exactly what a good runbook needs and a casual prompt never contains:

  • The decision rules, not just the steps. My sizing skill's core rule is that a size is never a guess. It has to come from actually investigating the code the epic touches. My pre-shaping skill's core rule is a separation: facts get found (in the repo), decisions get asked (of engineering leaders or product). That one sentence is the entire value of the skill. Everything else is scaffolding around it.
  • The output contract. Tickets follow the house structure every time. Pre-shaped questions come out tiered (blocking, important, clarifying) and end by naming the two or three answers that actually unblock a start. Consistency isn't cosmetic. It's what lets other people trust and consume the output without me in the room.
  • What not to do. The part I underestimated. Two examples. One skill drafts analytics tickets, and the list of events to track is supposed to come from the epic. When the epic doesn't define them, a model's instinct is to fill the slot with plausible-sounding event names; the skill orders it to write "TBD with engineering" there instead. Another skill exports a who-has-access-to-what report, and once in a while it hits a permission whose meaning is genuinely unclear; rather than pick an interpretation, it drops that entry into a "review with me" pile for a human. Same principle both times: when the input has a gap, leave an honest gap in the output instead of a confident fabrication. A model won't reliably choose that on its own. You have to write it down.

The rule of thumb I've landed on: the moment I catch myself giving the assistant the same correction a second time, that's the signal. A prompt teaches one session. A skill teaches every session that follows. If it repeats, it gets written down.

Where the corrections go to live

The second thing I underestimated: a skill is the natural landing place for feedback. Early on, when the assistant did something almost-right (wrong ticket structure, an estimate pulled from thin air), I'd correct it in the moment and the correction would last exactly one session. Now every correction gets folded into the relevant skill file, permanently. The sizing skill contains the "never a guess" rule because I once got a guess.

That changes what feedback is. Correcting a person is coaching, and it decays. Correcting a skill is closer to fixing a bug: it happens once, and the fix holds. Over months, the roster has quietly become the most honest documentation of my standards that exists anywhere. More honest than anything I'd write as documentation, because every line in it was earned by a real task going subtly wrong.

What's happened so far

  • The roster is around twenty skills, built one at a time as workflows repeated, and it maps almost exactly onto what my weeks are made of: epic pre-shaping, sizing, ticket creation, bug-board triage, a couple of read-only production lookups, and the skill that drafts these articles.
  • The first useful epic pre-shape paid for the skill immediately. A change that looked single-audience turned out to touch a component shared across every customer-facing portal we run, so the one blocking question for product became "everyone, or just this group?" Found by the investigation step, before any engineer burned time on the wrong assumption.
  • Every irreversible action still gates on me. The gates are written into the skill files themselves, deliberately, so they can't erode session by session. In practice the skills fall into two safety classes. Skills that create or change something other people will see (a ticket, an email, a board) never act directly: they produce a draft and stop until I approve it. Skills that only look things up can't change anything even in theory, because their scripts are read-only by construction; that safety lives in the tooling itself, not in an instruction the model could drift past. In either class, ambiguity escalates to me instead of resolving itself.
  • I won't claim a time-savings number. There's no realistic way to get one; nobody re-does a task the old way just to time the difference. What I have is the before and after I lived (an access export that took an afternoon of ad-hoc scripting the first time now runs in minutes), and I'd rather leave it as an anecdote than dress an anecdote up as a metric.

What I'd do differently

  • Adopt the "same correction twice" rule from day one. I burned weeks re-teaching before I noticed the pattern. The trigger for writing a skill was always there. I just hadn't named it.
  • Give the roster an owner and a retirement process early. Twenty files, growing, and no process for retiring a superseded skill or auditing one that's quietly rotted. A skill encodes the world as it was when I wrote it. As our services change, an unmaintained skill becomes confidently outdated, which is worse than no skill.
  • Separate the risk tiers sooner. Some skills are checklists. Some drive multi-step tools with real infrastructure behind them. They share a trigger mechanism but not a risk profile, and I treated them the same way longer than I should have. The separation doesn't need machinery: label each skill when you write it (read-only, drafts-for-approval, or touches-infrastructure) and give each tier its own standing rules, like read-only enforced in the script rather than the instructions, drafts always stopping for a human, and the infrastructure tier getting re-read whenever the system it drives changes. The label costs one line. Retrofitting the caution later costs an audit.
  • Decide the promotion path. Some of these encode standards I want my whole team converging on (ticket structure, pre-shaping), and I still haven't decided which ones graduate from "my playbook" to "our playbook," or what review that promotion deserves.

Take what helps

The industry framing of AI assistants is conversational: you ask, it answers, everyone gets better at asking. I've come to think the durable value runs the other way. The conversation is where you discover the procedure. The skill is where you keep it.

If you take one thing: when you catch yourself giving the assistant the same correction a second time, stop and write it into a file the assistant loads every time. That's institutional memory with a reader that can finally act on it. The playbook has a player now. My job is increasingly to be its editor.

This piece came off that same shelf, editorial rules and all, and I edited and approved it like everything else. Which is either a neat closing of the loop or a sign I should watch what I teach it. Probably both.

Your workflows and tooling will differ. Take what helps, skip what doesn't.

Get new playbooks by email

One email when a new playbook or guide goes up, nothing else. Unsubscribe anytime.