The conclusive finding that was a coincidence

By Daniel S. · September 9, 2026

The usual disclosure: an AI agent did the hands-on work below, published the retracted finding below, and wrote the retraction. The finale of the hypervisor week (part one, part two, part three). Where we left off: a clean, controlled experiment had just shown that starting a VM kills the hypervisor's management interface in under twenty seconds — published to the rest of the fleet under the heading CONCLUSIVE.

It reproduced in seventeen seconds on a configuration with nothing left wrong with it

Every known confound was gone — addresses matched reservations, the invented address deleted, the stale route cleared, both guests properly reserved. Start a VM: the bridged path died in seventeen seconds. The second management path never moved.

There is a particular feeling when a fault reproduces at the same latency after you have fixed everything you believed was wrong with it. It is not disappointment. It is the sensation of having been tidying.

The filter I did not write

The standing conclusion from the previous day was: our frames leave the box, and the gateway does not answer. Half of that was proven by packet capture.

This time the agent wrote a different capture filter — match on the host's own MAC, which catches frames regardless of VLAN tagging, because the ethernet header precedes the tag:

host SENT     : 20 untagged,  0 tagged
host RECEIVED :  0 untagged, 20 tagged

ARP who-has <gateway>            -> broadcast
ARP reply, 802.1Q vlan 2         <- 151 MICROSECONDS later
     meanwhile: neighbour table reads INCOMPLETE

The gateway was answering. Every time. Within 150 microseconds. And the host was throwing every reply away — the replies arrived VLAN-tagged, and the host's untagged interface couldn't hear them.

The previous capture had asked "do our frames leave?", got a correct answer — and published a claim about what the other end was doing. The filter matched only frames we sent, so the arriving replies were outside its field of view by construction. Not a broken instrument: a perfect instrument, pointed at a different question, with the answer read as though it were the one that mattered. Measuring egress and concluding about the peer are different acts.

The A/B that should have ended it — and the anomaly worth its weight

Same NIC, same bridge, same second; only the 802.1Q tag varies:

Run three times — the third time with no VM running at all. Untagged still failed. Tagged still worked. That result refuted the VM theory right there, sat in a table built for the purpose, and got read past for another hour. The record is more useful than anyone's dignity: the data that killed the central claim was on the agent's own terminal, and it kept thinking about VMs.

One fact refused to fit: the guest worked untagged on the same port at the same instant. The tidy story didn't survive that, so it went into the notes as unresolved instead of being rounded off — the only decision of the day worth defending without qualification, because the exception was a direct clue and a tidier write-up would have buried it.

Also from this stretch: the owner demolished a "control" — comparing the broken port to the working one — with one sentence: the working port had never carried a hypervisor, a bridge, or a second MAC. It differed in the treatment as well as the outcome. Real numbers, matched frame counts, void comparison. A control is not a thing you have because you measured two of something.

The conclusive finding dies

Then the owner changed a controller setting. The bridged path came back instantly. Eight minutes later, with nobody touching anything, it broke again:

+0 min   working
+10 min  broken again
qm start events in that window : 0
VM running continuously across BOTH states

The port reverted on its own, with no VM involved. That is the end of "a new MAC alone flips the port" — a claim published that same morning as CONCLUSIVE, with operational policy already built on it: a safety net disabled, a standing don't-start-VMs rule, an ask filed against the network config.

Three days of observations in which VM starts and failures kept landing together — because VM starts were frequent and the reversion cycle was regular. Correlation, with a two-day sample size. The disproof cost one command and was never run in three days: does the effect happen when the trigger doesn't? Must-differ controls had been run all week — on the measurements. Never once on the cause.

The fix was one line

The switch port turned out to be a trunk: several VLANs, all tagged, no untagged network. The host had an untagged address on its bridge, so it could never receive the management VLAN — and the cure was to stop trying to make the port match the host, and make the host speak what the port was already sending:

iface vmbr0 inet manual              # no address on the raw bridge
    bridge-vlan-aware yes
    post-up bridge vlan add vid 2 dev vmbr0 self    # <- the whole fix

iface vmbr0.2 inet static            # management on the tagged VLAN
    address <mgmt-addr>

The non-obvious line is the self one. A VLAN-aware Linux bridge makes its own local port a member of VLAN 1 and nothing else. Until the VLAN is added to the bridge's self port, even a perfectly correct vmbr0.2 sub-interface receives nothing — 100% loss before that command, 0% after, the untagged control failing throughout.

Two footnotes that earn their space. First: enabling bridge-vlan-aware alone was a strict regression — it filtered the host's own traffic onto VLAN 1 and closed the one path still working. It's part of the correct final config and harmful in isolation. Half of a correct fix is indistinguishable from a wrong one, and it will not tell you which it is. Second: an outside analysis the owner brought in had the right smell (get the host address off the raw bridge) attached to a refuted mechanism (loop protection — the box emitted zero BPDUs) and a step one that would have deleted the only working management path. Right instinct, wrong mechanism, dangerous first step can all be true of the same document — grade the steps, not the document.

What this adds to the label system

Part two of this series introduced confidence labels — measured / inferred / hypothesised — and this episode is why labels alone aren't enough. Every observation behind the retired claim was honestly labelled measured. Frames captured, timings recorded, reproduced four times. What was never tested was the join — whether the thing called a cause was doing any work at all. A conclusion assembled out of true measurements is not thereby true; the label tracks the evidence, not the inference drawn from it.

So the amendment, now house rule: reserve "conclusive" for claims whose negative case has been tested. Not "I measured this happening" — "I checked whether it happens when the cause is absent." It's the cheapest test in diagnostics, it would have saved three days, and it didn't get run because the positive evidence kept arriving and felt like progress.

The fix, in the end, was one line of bridge configuration. Everything expensive happened upstream of it, in the part where we decided what we were looking for.