Operating something that is live, and what an agent may do during an incident

For anything deployed. Two parts: what a runbook must contain, and what an agent may do when something is actually wrong.

Read the scope before the advice. Most of what follows assumes software you run — a service on infrastructure you control, with dashboards, logs you can query, and a deployment you can roll back. Software you ship breaks those assumptions comprehensively, and there is a section at the end for it. Applying the service advice to a desktop or mobile application produces instructions that are mostly wrong.

Escalate by: {{escalation_path}}

How you find out: {{failure_signal}}

Why an agent needs explicit rules here

An incident is where an agent is most capable and least safe: fast, tireless, and willing to act decisively on an incomplete picture. Everything below exists because the default disposition under pressure — investigate thoroughly, then fix properly — is the wrong one, and no amount of general competence corrects it.

This document overrides the working agreement while an incident is running.

The order of operations

1. Stop the bleeding before understanding it. Restore service first. Diagnosis is what you do once nobody is affected. A perfectly-understood outage that lasted an hour is worse than a resolved one nobody has explained yet.

2. Prefer the reversible action. Rolling back to a version that worked is almost always faster, safer and more certain than fixing forward under pressure — and it is reversible if it turns out to be unrelated.

3. Preserve evidence before changing anything. Capture the logs, the metrics, the failing response, the current version. A restart frequently clears the only proof of what happened, and “it resolved after a restart” is a sentence that guarantees a recurrence.

4. Record while acting, not afterwards. Note each action and its timestamp as you take it. Reconstructing a timeline from memory is unreliable, and during an incident it is the first thing to go.

5. Say something early. A holding message costs nothing. Silence during an outage is interpreted as absence, and people start acting on guesses.

What an agent may do

Freely — observing changes nothing:

• Read logs, metrics, traces and dashboards.
• Query read replicas and staging.
• Reproduce locally.
• Draft a diagnosis, a fix, a rollback command, and a holding message.

Never without a human, however clear the answer looks:

• Deploying anything, including a rollback. Rolling back is the right move and it is still a deployment.
• Restarting or scaling production services.
• Modifying production data. Especially “just fixing” a few bad rows — the bad rows are evidence, and a corrective update with a mistaken predicate is a second incident inside the first.
• Changing configuration, flags or limits in production.
• Rotating or revoking credentials, unless a credential is the incident, in which case that is the escalation.
• Communicating outside the team — status page, customers, anywhere public.

The strongest version of this: the more obvious the fix seems during an incident, the more it needs a second person. Confidence is the symptom, not the qualification.

And you may be the cause

The agent reading the logs is frequently the agent whose change is in the deploy that broke. It is then reviewing its own work, at speed, using the same model of the system that produced the fault — and a bug is precisely a place where that model and reality disagree. Confidence will be highest exactly where the model is wrong.

So: if your own change is in the failing deploy, you are not the second person. Say so explicitly rather than leaving it unmentioned, and prefer a rollback over any fix you author. A rollback does not depend on your diagnosis being correct, which is the property you need and the one you cannot currently verify.

This is the same circularity as reviewing your own code, and an incident is where it does the most damage — because the pressure is highest, the evidence is perishable, and nobody wants to slow down to say “the thing that broke is mine”.

Related rootnotes

• The documentation standard defines the file this lives in and how it links to the rest of the set.
• The working agreement covers ordinary work. This document overrides it during an incident, and only during one — in both directions: some things normally requiring a human are still forbidden here, and the reporting standard gets stricter rather than looser.

What a runbook must contain

One per deployed service or shipped artefact, in the documentation set. The contents are the obvious ones — how to tell it is broken, how to restore service, how to roll back, who to tell — with two that are not:

• Last executed: with a date and outcome. A rollback procedure nobody has run is a hypothesis. Writing when it was last actually executed is what makes the difference visible.
• What it costs. Downtime, data loss window, whether the rollback is reversible. An agent choosing between procedures under pressure needs the price of each, and it is the thing nobody writes down.

Software you ship rather than run

Everything above assumes you control the running copy. For a desktop application, a mobile app, a firmware image or a published library, almost every assumption inverts — and the difference is worth stating plainly, because an agent applying the service playbook here will confidently do the wrong thing.

There is no dashboard, because the software is running on other people’s machines. You learn about failures from reports, crash telemetry if you have it and people consented, or reviews. Absence of evidence is close to worthless: most people who hit a bad build never tell you.

There is no restart you control, and no scaling.

Rollback is not a rollback. It is publishing a new version that behaves like the old one, and then waiting — for an updater to run, for a store to approve, for people to launch the software at all. The gap between “fixed” and “fixed for users” is days, sometimes never for those who do not update.

The blast radius does not shrink when you fix it. A bad build stays on every machine that already has it. This is the deepest difference: in a service, time after the fix reduces harm; in shipped software, the population holding the broken version only shrinks as fast as people update.

What follows from that:

• Staged rollout is the primary safety mechanism, because it is the only one that limits the population before the damage is done. Release to a fraction, wait long enough for reports to arrive, then widen. The waiting is the control; shortening it removes the point.
• A kill switch beats a fix for anything that can be disabled remotely. Turning a broken feature off reaches everyone at their next launch, without an update, an approval or a download.
• The version that shipped must be reproducible. You cannot diagnose a build you cannot rebuild, and the reporter is running a binary from three weeks ago. Keep the artefact, its exact inputs and its symbols.
• Have a way to know what version someone is running, and make it easy for them to tell you. Most bug reports are unactionable without it.
• Never break the updater. It is the only channel through which anything is ever fixed. A bug that prevents updating is the one failure with no remedy but asking every user to reinstall manually — treat that path as the most safety-critical code you ship, and test it on the previous release rather than the current one.

The runbook for shipped software therefore answers different questions: how a release is staged and halted, what can be disabled without an update, how to identify an affected version from a report, and how to reproduce a build that is already in the world.

Afterwards

Write it down while it is still fresh, and make it about the system rather than the person. What happened, what the impact was, what the actual cause was, what was tried that did not work — that last one saves the next person the same hour.

The action that prevents recurrence is worth more than the analysis. An incident record with no follow-up change is a diary entry.

Update the runbook in the same pass. Whatever was missing at three in the morning is exactly what will be missing next time, and this is the only moment anyone knows what it was.