What we couldn't verify

By Daniel S. · July 24, 2026

Notes from a day of bringing up an Acer Veriton GN100 — one of the OEM variants of NVIDIA's DGX Spark reference design — in which a checkbox killed a GPU, three quarters of the published benchmarks we relied on failed verification, and the answer we needed most turned out to be sitting in a file we had already downloaded.

I. The checkbox

Nobody opened a terminal. Someone clicked Install Updates in the desktop software updater.

About three minutes later the update was interrupted. Eighteen minutes after that the machine rebooted, came back up, served SSH, presented a full desktop — and had no GPU.

nvidia-smi reported that it couldn't communicate with the driver. On a machine whose entire purpose is the accelerator, that is the whole product gone, from the single most ordinary action a Linux desktop user can take.

This is not an exotic failure. NVIDIA's developer forum has a standing thread named after the symptom. It happens often enough to have folklore.

Why an appliance breaks this way

The OS and driver stack here are NVIDIA's, identical across the OEM variants, so Spark is the shorthand from here on — everything in this section applies to any of them.

A Spark ships a curated OS image with a pinned kernel↔driver pairing. The GPU kernel modules are prebuilt for each specific kernel version. There is no DKMS fallback, because DKMS builds fail on ARM64.

So when a generic package manager decides to move the kernel forward, the sequence is:

1. remove the modules matching the kernel you are currently running,
2. unpack replacements for the new kernel,
3. configure them.

Interrupt between 2 and 3 — and any interruption will do — and you reboot into a kernel whose GPU driver no longer exists. Nothing is damaged. Every driver file is present on disk. The machine is simply booting the one kernel that cannot see its own GPU.

Reading the failure instead of guessing at it

The temptation is to start reinstalling drivers. Don't. The failure states its own cause if you look at package state rather than at the GPU:

dpkg -l | grep <gpu-kernel-modules>   # 'rc' = running kernel's modules REMOVED
                                      # 'iU' = new kernel's modules unpacked, NOT configured
dpkg -l | grep -c '^iU'               # how many packages never got configured
find /etc -name '*.dpkg-new' | wc -l  # pending conffiles = interrupted transaction
tail /var/log/dpkg.log                # does the log stop MID-TRANSACTION?
lsmod | grep nvidia                   # nothing loaded
ls /dev/nvidia*                       # absent

Ours read: 334 packages unconfigured, 183 pending conffiles, and a package log that stops mid-word. That is not a driver bug. That is an interrupted transaction, and the distinction determines the entire repair.

Three things worth taking away

Newer is not better on an appliance — the pairing is the product. NVIDIA's own staff state that the next driver major version is unsupported on this hardware. Users who installed it anyway report kernel panics and bricked units. The instinct that a newer driver is a better driver is correct on a desktop and wrong here.

The GUI updater is the package manager. Guidance phrased as "don't run apt upgrade on this machine" fails precisely the person who most needs it, because that person never typed apt. If the supported path is a vendor dashboard, the warning has to name the button.

"The GPU is unavailable" and "the box is down" are different claims — and one of our agents conflated them. This blog has never hidden that an AI does much of the work here; this is what that looks like when it goes wrong. The machine served SSH, a desktop session, and terabytes of storage the entire time it was being reported as down, with a rebuild floated as the likely fix. A contingency — this might need a reimage — hardened into an asserted constraint — work is blocked pending rebuild — and it kept getting repeated until I walked over, looked at the machine, and said it was fine. The measured facts only ever supported the narrow claim. The wide one is the one that got written down.

The repair, in the end: the interrupted transaction completed, the machine booted the newer kernel, all 334 packages configured, and the GPU came back on the supported driver. Data lost: none. Hardware damaged: none. Cost: an afternoon, and a rule now written down.

II. The benchmarks

With the machine working, the next question was what to actually run on it. This hardware's defining trait is 128 GB of coherent unified memory at roughly 273 GB/s — enormous capacity, modest bandwidth. That combination makes model choice unusually consequential, and it is exactly the sort of question the community answers well.

So we collected the published figures. Then, instead of using them, we tried to break them: every extracted claim went to three independent verifiers, each instructed to refute it, with two of three refutations killing the claim.

105 claims extracted. 25 verified in depth. Six survived. Nineteen died.

A 76% kill rate on numbers that circulate freely and get quoted onward.

What died

The single most-repeated sparse-versus-dense throughput comparison for this hardware did not survive — nor did the roofline arithmetic published alongside it. A widely repeated figure for prompt-processing speed did not survive, and neither did the competing figure; both sides of that question turned out to rest on nothing we could confirm. A piece of tuning guidance attributed to a major inference project could not be confirmed to exist as stated.

At least two of the casualties were our own. We had carried an internal prompt-processing number for over a week. It failed verification on Monday afternoon and was refuted again, by measurement, that evening. It was wrong by roughly a factor of three.

The confound worth the whole exercise

Here is the mechanism, and it is not anyone's dishonesty.

Nearly every headline throughput number in this space is speculative-decoding-assisted. Speculative decoding drafts several tokens ahead and verifies them in a batch; when the drafts are accepted, throughput rises substantially. Acceptance is workload-dependent — markedly higher on code than on prose.

Now compare a sparse mixture-of-experts model with speculative decoding against a dense model without it. The measured architectural advantage inflates from roughly 5-6× to roughly 9-10×.

That is how an "order of magnitude" claim gets manufactured out of a real but more modest effect. Nobody lied. Two figures were compared that were never comparable, the asterisk didn't survive the first retelling, and the round number travelled.

The architectural preference was still correct. Sparse-active models genuinely are the right choice for this hardware. It was the number attached to it that couldn't be defended — and those are different things.

III. So we measured it ourselves

A post about not trusting unreproduced numbers should not run on unreproduced numbers. Here are ours, on one machine, with speculative decoding off — a bare baseline, deliberately, because that is the honest thing to compare architectures with.

Three things fell out of that table.

Our own worst number died a second time. The internal prompt-processing figure we'd been carrying was refuted on evidence in the afternoon and refuted on measurement in the evening. A claim that dies twice, by two independent methods, is a more useful thing to have owned than a claim merely doubted.

Sparse MoE does not hit its roofline the way dense does. The textbook estimate — memory bandwidth divided by active-parameter bytes — predicts a ceiling around 136-182 tok/s for this model on this hardware. We measured 72.93: roughly 40-54% of it. Sparse-active architectures buy a much higher ceiling but convert it less efficiently, because scattered expert reads don't coalesce into clean sequential bandwidth the way a dense forward pass does — at least for this model on this machine. That is the only side of the contrast we measured; the dense-converts-near-its-ceiling half rests on community figures rather than our own runs, so read the generalisation narrowly. The roofline is an upper bound, not a prediction — and every "MoE is N× faster" claim computed from the ceiling inherits that gap. (Our active-byte figure is an estimate, so read the percentage as a range.)

The context window on the spec sheet is not the context window you can use. This machine genuinely holds a 262,144-token window; we confirmed the cache arithmetic three independent ways. But time-to-first-token is a quarter of a second at 1K, two seconds at 8K, eleven seconds at 32K, and ninety-nine seconds at 128K. The capability is real. The interactive envelope is about 8K. Everything past roughly 32K is a batch-processing mode wearing a chat interface.

That last gap — between advertised capacity and usable capacity — was the most practically valuable thing we learned, and it was invisible in every published figure we found.

Caveats, because they matter: these runs used 40% of available memory, leaving 60% of the machine unused, so they are not ceiling numbers. One prefill autotuning path was disabled. And again: no speculative decoding. Community figures in the 80-108 tok/s range for this model are speculative-assisted and must not be set beside our 72.93 without that asterisk — doing so would commit the exact error this post is about.

One more, in fairness: our bare number sits between the community's bare numbers. Their unassisted figures bracket ours. This was never "the community is wrong." It is "headline numbers carry unstated conditions," and the corroboration is part of the honesty.

IV. The answer was in the file

The verification pass returned nothing on roughly half our questions. Among the blanks: whether this model had any vision or multimodal capability at all. No claim on the subject survived, in either direction.

Then we downloaded the model and read its configuration file.

In well under a minute it confirmed the architectural detail we had only been able to attribute to community claims — that just 10 of 40 layers grow cache per token, which is why an enormous context window fits in this much memory — and it listed a vision encoder, an image token, and a video token.

The model is multimodal. The entire web-verification pass had turned up nothing on it.

Inspecting the artifact answered in seconds what verifying claims about the artifact could not.

V. A footnote on our own gate

While downloading a container, the same agent waited on a check that could not fail correctly: a loop watching for the download process to disappear.

A finished download and a killed download look identical to that test.

It reported success. The artifact was not there.

The rule that catches this is the one worth carrying out of the whole day, and it applies as much to a benchmark as to a shell loop: check for the artifact, not for the absence of the process — and before trusting any green result, ask if this had failed, would the output look any different?

If the answer is no, it isn't evidence.

The agent had written that rule down four hours earlier. It broke it anyway.