The runbook that read perfectly
The usual disclosure: AI agents did the hands-on work below — including writing the defective runbook, and then, separately, fact-checking it to pieces. Context: the hypervisor from the box-3 series exists to host coding agents in VMs. The software stack going into those VMs came with a runbook that had been assembled by an LLM from documentation. Before committing a disk wipe to it, we commissioned an independent fact-check — and then a second agent, on another machine, ran its own pass without reading the first one's findings. They agreed: five for five.
The stack was real. The instructions were not.
Worth stating first because it changes the risk profile: the runbook was not hallucinated. The agent framework it described is a real, live project; the install URL answers; the headline commands exist. The architecture it proposed was sound and worth building. Every defect lived one level down — in verbs, keys, flags, and paths. That's what made it dangerous.
The five, in ascending order of nastiness:
1. The build command dies at minute one. cd <inference-engine> && make — except that
project's Makefile is now a tombstone: a five-line error saying the build moved to CMake
years of commits ago. Under the deploy script's set -euo pipefail, this kills the provisioning
run immediately, leaving a half-built container. (The binary also moved paths, so the service
unit pointed at nothing.) A human who had actually run this stack would have hit it in the
first five minutes and fixed it before writing anything down.
2. The last instruction in the manual is a flag that does not exist. Zero hits for it in the entire CLI source tree. It's the final command — the one you reach after hours of setup, when you're least inclined to doubt the document.
3. The credential is read by nothing, and warned about by nothing. The runbook says to put the signing key in a config file. The loader maps exactly nine keys out of that config section into the environment; the signing key is not one of them — the source's own docstring says so. What the operator sees: valid YAML, a gateway that starts, then a non-retryable death. Nothing looks wrong. This is the worst of the five, because every other failure at least produces an error near its cause.
4. The URL scheme silently degrades. The config takes a relay URL; two different consumers
parse it with different rules. One maps ws:// correctly; the other speaks only HTTP and dies —
except the transport layer catches the exception and falls back to slow polling, logged at INFO.
A typo becomes a working system that is quietly not the system you configured.
5. The recommended model can't do the job, twice over. The docs require a 64,000-token context for agentic tool use. The runbook's recommended model tops out at 8,192 — and the project's own compatibility table marks it as not tool-calling at all. Bonus, from the second fact-check pass: the popular local-model server defaults to a 2,048-token context, so even a model that can do 64k silently won't, unless you rebuild with the larger context baked in. Two ceilings, both invisible, both far below the floor the software itself states.
The defect that wasn't an error
The sixth finding is the one worth the post. The runbook never states that the hypervisor platform is a bare-metal OS that repartitions the disk. All its references to the guest OS read as if the existing server install would survive. An operator following it in good faith would discover the wipe at the worst possible moment.
Why did every source omit it? Because every source assumed it. Any human writing about that platform takes bare-metal for granted; it's too obvious to state. So no source said it, so the LLM's synthesis couldn't contain it.
Assumed context does not survive summarisation. The dangerous omissions in an AI-assembled document are not the facts the sources got wrong — they're the facts the sources all considered too obvious to write down.
(Same category of error, spotted in the same review: the runbook framed a choice as "platform A vs the hypervisor underneath it" — comparing a thing to a component of itself. A synthesis engine happily builds a decision matrix over a category error, because the sources each made sense alone.)
The failure signature is that everything reads right
Put the five defects side by side and they share one property: plausible, structurally correct, and never executed. The commands look like the real commands. The config keys look like real keys. The flags look like the project's flag style. Nothing pattern-matches as wrong, because the document is a paraphrase of documentation — and documentation is exactly what the model learned the shape of.
A transcript of a working install has a different texture: it contains the workaround for the tombstone Makefile, the actual flag, the env var the loader really reads — because the person who wrote it was stopped by each of these and had to get past them. The errors a runbook survives are the evidence it was run.
Two practical tests we now apply to any AI-assembled runbook before trusting it:
- Execute before you extend. Run step one on a throwaway target before planning around step ten. Every one of our five defects was findable in minutes by execution, and none by reading.
- The step you can't write down concretely is the step you haven't thought through. The build's one real platform gotcha (the tool needing a newer OS than any machine we had) was caught not in planning but at the moment someone had to write the command and the machine it runs on in the same sentence of a work order. Writing the instruction down is itself a check — a vague step in your own runbook is where your understanding stops.
And a closing caveat in the other direction, because this isn't an argument against AI-assembled documents — we generate plenty and this blog is written with one in the loop. It's an argument about what they are: a well-read draft by someone who has never touched the machine. Treat it as that — a strong hypothesis, cheap to verify per-claim — and it's genuinely useful. Treat it as a transcript of experience and it will hand you a half-provisioned container at midnight with nothing red anywhere.
Related: the blocker that was never there — the same disease in human-written handoffs. The version number nobody ran and the flag that doesn't exist are the same defect: a sentence with the texture of a fact.