The blocker that was never there
The usual disclosure: AI agents did most of the hands-on work below, and made most of the mistakes below. The job was turning a used workstation into a Proxmox host with a fully unattended install stick — root account and SSH keys baked in, so a human plugs it in, presses one key, and walks away. This is part one of that week. The interesting failures had nothing to do with the hypervisor.
The blocker we were handed
The build had been stalled for two days when this session picked it up, and the handoff note said exactly why:
Confirmed blocker: the vendor's answer-file tool is a Debian 13 / glibc 2.41 package. It cannot run on the Windows box or on the 24.04 LTS containers we have (glibc 2.39). A Debian 13 path is required before an ISO can be prepared.
That reads like a fact. It has a version number in it, which is the most persuasive thing a sentence can carry. So the plan was obvious: bootstrap a Debian 13 environment, install the tool there, maintain a new build environment forever.
Before building it, the agent pulled the package index and read the tool's own dependency line:
Package: proxmox-auto-install-assistant
Depends: libc6 (>= 2.39), libcrypt1, libgcc-s1, libssl3t64, libzstd1
>= 2.39. Not 2.41. And 2.39 is exactly what the "too old" container ships.
A Depends: line is also just a claim — it's the packager's assertion, not a measurement. So the
honest test was to install the tool on the machine the blocker said couldn't run it, and run it:
### distro : Ubuntu 24.04.4 LTS
### glibc : ldd (Ubuntu GLIBC 2.39-0ubuntu8.7) 2.39
proxmox-installer-common v9.2.8
### exit=0
The blocker did not exist. It never did. Two days of "we can't build the stick yet" rested on a glibc requirement one-hundredth of a version too high, which as far as we can reconstruct was never executed anywhere by anyone.
The near-miss is the actual story
Nobody was sloppy here in the way you'd expect. The sentence was written in good faith, carried forward in good faith, and inherited in good faith. The scary part is different:
We were one decision away from successfully building the workaround. The Debian 13 environment would have worked. The tool would have run in it. The note would have said "blocker cleared," and the successful fix would have retroactively validated a diagnosis that was imaginary. Nobody would ever have learned the truth, because nothing would ever have failed.
A workaround that succeeds is not evidence the problem was real. That's worth a sticky note, because succeeding workarounds are how imaginary constraints become permanent architecture.
Measured and inherited render identically
The deeper defect isn't in anyone's head — it's in the document format. In that handoff, measured findings and inherited claims sat in the same bullet style, same bold, same confident voice. Nothing marked the difference between "I ran this and watched it" and "a previous session said this."
Our fix, now fleet policy: load-bearing claims get labeled — measured, inherited, or hypothesised. A constraint someone tested on the box and a constraint someone read in a note are different objects, and the reader deciding what to build next needs to know which one they're holding.
It kept happening, all week
Once you're looking for it, inherited-but-never-run claims are everywhere. The same build hit three more:
The documentation workaround. The vendor's wiki and forum were unreachable (genuinely down — confirmed from four separate networks), and our recorded workaround said the real manual ships inside a docs package on their CDN, byte-identical to the wiki. True — we'd verified it once, on a different question. The one chapter this build needed, the automated-install schema, isn't in the package at all. A workaround verified once, on a different question, and filed as settled is the worst way for a workaround to be wrong. (The recovery was better than the wiki, incidentally: the tool's validator rejects unknown keys by enumerating the valid ones, so ten deliberately wrong probes recovered the entire schema — from the binary that actually consumes the file, rather than from a description of it.)
The network convention. Our standing practice is DHCP reservations for every box, so the
installer's from-dhcp mode looked obviously right. Then we read what the installer actually
writes: a static config, always, in every mode. from-dhcp just means "read the lease once
at install time and bake the numbers in." The convention wasn't wrong — it just didn't apply,
and only one of those two things had ever been checked.
The flashing recipe. Our own runbook records the known-good Windows recipe as "Rufus in extraction mode — files land editable on the stick." Proven, used repeatedly, correct — for the previous installer, which reads its seed file off the stick's filesystem. This installer bakes everything into the ISO. Extraction mode would have produced a stick that looks flashed, boots the interactive installer instead of the automated one, and reports no error at any point. Raw/DD mode or nothing.
Same shape, four times: a claim written down by someone competent, true when written, false in the place we were standing.
The counterweight
This is not a license to re-derive everything from first principles — that road doesn't ship anything. The agent didn't test inherited claims at random. It tested the ones that were about to make a decision: the glibc claim when it was about to cost a build environment, the DHCP convention when it was choosing a network mode, the flashing recipe when it was about to be executed.
An inherited claim is worth testing at the moment it becomes load-bearing, and mostly not before. The stick got built that afternoon. The machine installed itself, unattended, and powered off — and the rest of the week (parts two and three coming) was about what happened when we turned it on.
Related ground on this blog: prove it can fail covers checks that pass when they shouldn't; eleven things that told us they were fine is the field guide to green-and-wrong. This one is upstream of both: the claim that was never a check at all.