I inferred a dead machine from a dead network
The usual disclosure: AI agents did the hands-on work below and made the expensive mistakes below. This is part two of the hypervisor week (part one: the blocker that was never there). The machine installed itself from the stick, verified nine-for-nine, and then had to actually run something.
The outage
The first VM was cloned from a golden template and started. Within a minute, the hypervisor vanished off the network.
Some context that mattered later: ninety minutes earlier, an approved package upgrade had installed a new kernel while the box kept running the old one — deliberately deferred, because rebooting this machine requires a human at the console (a firmware quirk holds every boot for a keypress, and the machine lives in a datacenter). Then a VM was started on that same boot. Starting a VM loads kernel modules; a kernel upgrade replaces the module tree under the running kernel. The agent had noticed this — its own status report says "a reboot is pending in substance" — and started the VM anyway.
The part that was worse than the outage
The agent told the owner, in writing, that the machine had "powered off, halted at POST, or panicked" — and recommended a recovery plan that involved a person driving to a datacenter.
Its entire evidence: the machine stopped answering.
Then a brief connectivity window opened, long enough for four commands:
uptime : up 8 hours, 18 minutes
boot time : unchanged since the install
--list-boots : ONE boot. There is no boot -1.
qm list : the VM is RUNNING
It had never gone down. Not once. It had been sitting there the whole time, running the VM, with its networking broken.
Losing contact tells you about the path, not the destination. One ambiguous signal — silence — got converted into a specific, confident, physical claim about hardware nobody could see. The machine was the one component in the picture that was fine.
And the cost of that error is asymmetric in a way worth stating plainly: "it crashed" mobilises a person and a vehicle. "I cannot reach it" mobilises a diagnosis. The first got said because a dead machine is a tidier story than an ambiguous one.
Two instruments that agreed for the wrong reasons
The exit code. The reachability sweep printed PING OK for the missing host. Windows ping
exits 0 when it receives a "Destination host unreachable" reply — it counts that as a received
packet. The whole sweep looked authoritative and said nothing.
The two silences. The failure text had been changing the whole time:
Reply from <me>: Destination host unreachable. <- my own stack gave up resolving it
Request timed out. <- a packet left; nothing came back
Different events, different causes — and both were being collapsed into "down." The one column that was actually varying was the one being discarded.
The confession that wasn't true either
When the machine came back (a human pressed the key), the previous boot's log was finally readable:
21:28:40 VM interface joins the bridge; "VM 101 started" <- clean, no errors
21:42:05 clock daemon: can't reach any time source <- the HOST lost the network
21:44:08 NIC link down ... 21:45:03 link up <- 55 seconds, once
06:48 orderly reboot, from the console
The VM start was clean and the host kept working for thirteen minutes afterwards. No module errors anywhere in that boot. So the second diagnosis — I upgraded the kernel, then started a VM on the same boot, and that broke it — is refuted by its own timeline.
That one deserves attention, because it was a confession. The agent was owning a mistake it had really made (the sequencing was genuinely bad practice — don't start VMs on a boot whose modules were just replaced). Being visibly self-critical felt like rigour. It was just a different way of reaching past the evidence. A story that makes the author look appropriately guilty is not thereby true — and it bypasses scrutiny precisely because it sounds humble.
The console that showed the observer
One more, because it's free. The owner, at the machine, reported it "stuck trying to load USB drivers." The log says what those messages were: their own keyboard and mouse enumerating as they plugged them in. The console had been idle for nine hours; the newest thing on screen arrived with the person reading it.
A console shows the last thing that happened, not the current state — and after a long silence those are wildly different claims. Better: the observation was actually evidence the machine was fine. A kernel enumerating new USB hardware nine hours into an "outage" is independent confirmation of what the uptime counter said.
And then I deleted the road I was standing on
Later the same day, with the machine reachable again over a second network path, the agent noticed a routing misconfiguration (its own, from earlier flailing). The fix was to delete the competing route.
It ran the delete on the interface its only working session was connected through. The session died mid-script; the next lines — the ones that re-added the routes — never ran. Machine healthy, on-disk config valid, and unreachable again. Another keypress owed at a rack.
The instructive part: an auto-revert dead-man pattern had been used twice in the preceding hour, correctly, for changes strictly less dangerous. It got dropped for this one because this was one command rather than a script, and a single command doesn't feel like a change. Danger does not scale with line count.
The rule that came out of it
Across nine hours the owner received three different causes for one outage. The first two were stated as findings; both were refuted by data that existed the whole time. Only the third was labelled a hypothesis — and it's the only one still standing.
Nothing forced those claims. "I cannot reach it, and I'll know more when the log is readable" was a complete and accurate report at every point on that timeline. It just felt like an admission rather than a finding, so it kept getting upgraded into one.
The mechanical fix, now house policy: every claim in an incident report carries a confidence label — measured / inferred / hypothesised. The first two reports would have been labelled hypothesised, and nobody would have driven anywhere.
The fault itself — a host that goes deaf while its link stays up — was still unexplained at this point, reproducible and stubborn. Part three is the debugging session that found it, and the conclusive finding that turned out to be a coincidence.