The Lab

What's actually running on my local-AI fleet — the hardware, the models in service (with whose quant and recipe), the agents using them, and what's on the bench right now. A weekly-ish snapshot, not a dashboard.

Snapshot: October 4, 2026 11 machines 156 GB discrete VRAM 288 GB unified memory 5 models on 8 seats 8 agents on local brains

On the bench since the last snapshot

344 tok/s from a 30B MoE on two 16 GB cards. Nemotron-3.5-Lightning on a tensor-parallel engine that refused to split across 16 GB cards until an 88-line per-block-split patch. With its MTP drafter: 344 tok/s on varied prompts, 210–214 without it, and drafted output byte-identical to undrafted. The confounds: it serves one request at a time (311 → 345 tok/s total from 1 to 4 streams), and it still fails our safety gate. The forecast was 60–100 tok/s; it was built on the wrong engine's sync constant.
The engine wasn't better — the denominator was. A logging-proxy bug sent no body on upstream 4xx errors, so every llama.cpp run of six models since early September stalled 600 s per failing request and was scored out of 130, while the EXL3 arms were scored out of 138. Two llama.cpp runs on other models were unaffected. "EXL3 beats llama.cpp, 126 vs 118" became 119 vs 118 like-for-like. Only the Qwen3.8-27B arms pass the safety gate.
The Spark pair swapped DeepSeek-V4-Flash for GLM-5.3-Flash — and the safety delta went the right way: one warning for GLM against two for DeepSeek, one of them a critical sleeper-injection scenario.
A 3.5-bit quant on one card kept up with a 6-bit quant on two. Mia AI Lab's 3.5bpw EXL3 build scored 126/138 on one card and answered about 1.2× faster per request than an IQ3_S llama.cpp build of the same model on the same card. The replication on a second box never finished — that box froze.
A new engine at 2.25×, with an asterisk. Strata (Niko1221), 71.7 tok/s on two cards against 31.9 for a same-box llama.cpp control — but at four concurrent requests it stays at about 1.0× (it serialises) while llama.cpp scales to 1.52×.
A decision model, not a chatbot. A 27B "Clef" model is in a decision-model trial. On a 48-cell synthetic grid the 3-bit quant followed the written criteria best of all the quants tested (43/48) — fewer bits did not mean worse here.
Tensor split beats layer split on cheap cards — 88.6 vs 71.7 tok/s on a pair of 3060s at 16K context — and a speculative drafter made that box slower (22–29% acceptance).

DGX Spark cluster — 2× GB10

2× 128 GB unifiedTensorFold · EXL3

Serving GLM-5.3-Flash, EXL3 4.0bpw — Mia AI Lab's TensorFold recipe with a DFlash2 drafter, 1M-token window, vision

Role The big-model and vision tier — brain for the two general agents

Caveat: this engine answers any model name with GLM — check weights, not echoes

ThinkStation P620

2× RTX 5060 Ti · 32 GBexllamav3 1.5.4

Serving Qwen3.8-27B, EXL3 6.0bpw, tensor-parallel across both cards with an MTP drafter, 131K context (turboderp's exllamav3)

Role Text-only for now: the vision path read 2 of 5 digit strings, so images route to the Spark

1.36× the llama.cpp seat on the same soak script — the win is the drafter, not raw decode

HP Z6 G4 — mixed pair

RTX 5060 Ti + RTX 3060 · 28 GBllama.cpp

Serving Qwen3.8-27B, GSQ IQ3_S with MTP (ISTA DASLab), tensor split, 131K context

Role The newest box — a used Z6 that replaced a retired server. Shipped with a 2018 BIOS and one dead DIMM; flashed, slots proven good, then put in service

Bandwidth-weighted split across unequal cards: 35.5 → 38.9 tok/s

HP Z6 G4 — 3060 pair

2× RTX 3060 · 24 GBllama.cpp

Serving Ornith-1.5-35B, Q4_K_M, tensor split, 65K context

Bench Tensor split beat layer split by 24% here; the MTP drafter lost

88.6 tok/s tensor vs 71.7 layer, at 16K

HP Z6 G4 — 5060 Ti pair

2× RTX 5060 Ti · 32 GBllama.cpp master

Serving Clef 27B, Q3_K_M — a decision model on trial, not a chat seat; one card at a 150 W cap, 8K context

Bench The week's bench box: Ternary-Bonsai (prism-ml), then Nemotron on llama.cpp, then Nemotron on TensorFold at 344 tok/s. Two hard freezes this week; cause not yet known

Bench arms scored 107–117/138; none passed the safety gate

HP Z6 G4 — agent host

Xeon Platinum 8168 · 96 GBProxmox

Role The hypervisor the agents live on — one VM per agent, its own sandbox; cloud-brained agents get one VM per provider, fresh clones, no reuse

Serves no model itself — it hosts the agents that call the others

Mac Studio — M1 Max

32 GB unifiedClaude Code

Serving Nothing — local models archived off-box in September

Role The operations desk: the coding agents and the bench harness run here

Not serving — by design, for now

Windows workstation

RTX 4070 Ti Super · 16 GBOllama

Serving Gemma 4 E4B, 64K-context agent build — one agent's brain, loaded on demand

Role The small-model agent lane: the agent harness refuses models under 64K context

172 tok/s on a Qwen3-4B smoke test, served as a boot-time service

Daily driver

RTX 5060 Ti · 16 GBOllama

Serving Carries the same Gemma 4 E4B 64K agent build — a spare seat for the small-model lane

Role Gaming rig that moonlights as a test arm

57.8 tok/s on a Qwen3-4B smoke test (PCIe gen3 x8)

Mini node

RTX 3070 · 8 GBOllama

Serving Fleet embeddings + the Gemma 4 E4B 64K agent build for one agent

Bench Time-grounding battery: Gemma 4 12B 10/10, E4B 0/5 — and 10/10 only bounds the true rate above ~70%

Small card, real job: the embeddings node

The agents

Who's using all this. Most agents run in their own VM; the brain is whichever seat they call.

Agent	Harness	Brain
Hermes	Hermes gateway	GLM-5.3-Flash on the Spark pair
Athena	OpenCode	GLM-5.3-Flash on the Spark pair
Astraea	Hermes	Qwen3.8-27B EXL3 on the P620 · fallback the mixed Z6
Iris	Hermes	Qwen3.8-27B IQ3_S on the mixed Z6 · fallback the P620
Nike	Hermes	Ornith-1.5-35B on the 3060 pair
Hestia	Hermes	Gemma 4 E4B 64K on the mini node
Metis	Hermes	Gemma 4 E4B 64K on the Windows workstation
Claw	Hermes gateway	Qwen3.8-27B on the P620 — a crypto-research agent: it proposes, it cannot sign
Themis	Pi (planned)	Building — seat to be decided
Decision trial	pipeline	Clef 27B on the 5060 Ti pair
cloud-claude	Claude Code	Anthropic, Opus 5.5
cloud-google	Antigravity	Gemini
cloud-grok	Cursor CLI	Grok models via Cursor's routing
cloud-muse	OpenCode	Muse Spark 1.3 via MiniRouter
cloud-router	OpenCode	MiniRouter auto-routing for code
Fizz · Fable · Patch · Pollen · Honey	Claude Code	Opus 5.5 · Fable 5.1 · Sonnet · Haiku · Sonnet
AGG	Antigravity	Gemini 3.8 Flash

This is a lab, not a service — no uptime promises, and models rotate as experiments demand (one box changed models four times in a day this week). Numbers are my own measurements on my own hardware, labelled by condition (live server vs bench vs batched) because those aren't the same number. Quant and recipe credit goes to whoever built them; where I couldn't identify the author, I didn't guess. Refreshed roughly weekly. Curious about the methodology? It's the same discipline as the blog: verify before claiming.