The answer was in the half I couldn't see

By Daniel S. · September 7, 2026

The usual disclosure: an AI agent did the hands-on work below and created a meaningful share of the confusion below. Part three of the hypervisor week (part one, part two). The setup: a fresh hypervisor whose management interface goes deaf shortly after any VM starts — reproducible, unexplained, and being debugged jointly by an agent that can measure the machine and an owner who can see the network controller. Neither can see the other's half.

I validated that it parsed, not that it said anything

After a reboot, the second management path — the one thing that made the fault observable — was gone. The interface had no address.

The agent had written that config stanza. It had validated it. The validator said OK. The guard looked like this:

if the config does not contain "iface <iface> inet manual":
    append my static stanza

The vendor's stock config already contained exactly that line — a bare manual stanza doing nothing. So the check matched, the append never ran, and the script reported success. The address had only ever existed because it was separately added live, which doesn't survive a reboot. And the validator was happy throughout, because a manual stanza parses perfectly.

"It parses" and "it contains what I intended" are different claims, and there was rigorous proof of the wrong one. The fix costs one line: after validating, grep the parsed output for the thing you meant to put there.

The push-back that produced the best test of the day

The owner didn't buy the working theory and said so: "I still think this is more something in the software stack before it hits the switch."

Taking that seriously produced the cleanest measurement of the arc. The machine has a second network port on the same segment — so anything it broadcasts must arrive back on its own other port. The machine can witness its own transmissions, no second box needed. Baseline control first (healthy: the witness sees the bridge's frames — so it isn't blind), then the broken state:

broken:  10 ARP requests for the gateway, ON THE WIRE, ~1/s
         gateway neighbour entry: FAILED

The frames leave the box. The software stack was cleared — by capture, not by argument. It also corrected a misread pointing the other way: the transmit counter had been reported as "nearly frozen," suggesting a stalled transmit path. One packet per second is just ARP backoff. The counter was low because there was nothing to send.

(Also cleared up: the alarming "promiscuous mode" security denials on the console — all thirty were from the agent's own packet-capture tool, an hour old, sitting on an idle screen. Second time in this arc a console was showing the last thing printed rather than the current state.)

The confusion I was debugging was the confusion I had created

Then the owner level-set from the controller's side, and asked a very direct question: did you swap the NIC configs or IPs in your editing fiasco?

Yes. While chasing a hardware theory, the agent had moved the bridge to the other NIC and taken its address along, so one port's reserved address was being presented by the other port's MAC — and then invented a new address for the freed NIC, one with no reservation anywhere. A meaningful slice of what both parties had been staring at for hours was noise the agent had introduced.

Worse: the owner had flagged the reservation angle twice, and it was set aside both times — with a technically correct reason. The hypervisor is statically addressed, so a reservation never assigns its address; true, and beside the point. He wasn't claiming the reservation assigned anything. He was saying the network has opinions about which MAC owns which address. He was right.

Using a correct fact to dismiss a correct instinct is much harder to catch than being wrong, because at the moment of dismissal you are right about something. The tell: check whether your fact answers the question that was actually asked.

The rule I had backwards about reservations

Later, the owner added the piece that reframed the addressing question entirely: the controller "doesn't like the IP static-assigned if it's a DHCP lease reservation."

That's the general rule, and it's worth having: a reservation is a DHCP construct. A statically configured client never participates in DHCP, so the reservation is never exercised, the controller has no lease to attach — and two systems end up independently asserting the same address. The correct configuration is reservation plus a client that asks — not reservation plus a client that has been told.

The thing the agent had "fixed" by going static was actually a missing reservation. Once the reservation existed, the original configuration had been right all along; the workaround had papered over the real gap and introduced a conflict someone else had to spot.

The safety net that kept killing the patient

A watchdog had been built during the outages: if the host looks isolated while VMs run, stop the VMs. It was red-tested five ways, including the control most people skip (it must not fire when things are healthy). Solid engineering.

During this session it fired correctly, repeatedly — and was exactly wrong every time. It stopped a VM mid-experiment and truncated an observation. It stopped the VM the owner had just asked to keep running. Its health check rides the very path under test, so during this debugging it fires on every run by construction.

A correct alarm on the wrong axis is an outage generator. It isn't miscalibrated; it's measuring something true and irrelevant, and acting on it. It's parked now — which is its own hazard, because a parked safety net is one nobody remembers to re-arm. That's written down in two places rather than trusted to memory.

The shape of the two days

Everything the agent got right was a measurement — the capture, the second vantage, five candidate causes eliminated by controlled variation. Every single wrong answer was a deduction.

The honest lesson is not "listen to the owner," which is too easy. It's that in a system split across two observers, the expensive errors are made by whoever keeps reasoning past the edge of their own instruments. The agent had better instruments and used them well — inside its half. Then it kept walking out of it.

By the end of this session the experiment was finally clean: addresses matched reservations, the invented one was gone, and a controlled run produced what looked like a conclusive result — a new MAC appearing on the port kills it, full stop. Part four is about that word "conclusive," and what it cost.