Dense vs. MoE models: why a "35B" model can run like a 3B

By Daniel S. · July 4, 2026

A model labeled "35B parameters" usually means a 35-gigabyte-class memory bill and speed to match — except when it doesn't. Some 35B-class models generate text as fast as a model a tenth their size, because of how they're built internally. That's the difference between a dense model and a mixture-of-experts (MoE) model, and understanding it changes how you shop for local AI hardware.

TL;DR

• A dense model uses every one of its parameters for every single word it generates. A mixture-of-experts (MoE) model splits its parameters into specialized sub-networks ("experts") and only activates a handful of them per word.
• Memory needs follow the model's total parameter count — every expert has to be loaded, because the router doesn't know in advance which ones it'll need.
• Generation speed follows the active parameter count instead — the parameters actually doing work for that word, which can be a small fraction of the total.
• Real example: Qwen3.6-35B-A3B has 35B total parameters but only 3B active per token — it needs the VRAM of a 35B model but generates at roughly the speed of a 3B one. ⏱️ verified 2026-07-05
• What this means for hobbyist hardware: a MoE model can be the best move if you have spare VRAM capacity but limited memory bandwidth — you get big-model knowledge at small-model speed, as long as you can fit it at all.

What's the actual difference between dense and MoE?

A dense model has one feed-forward network per layer, and every token passes through all of it — no shortcuts, no routing, every parameter does work on every word. That's the architecture behind most of the models people first learn on (early Llama and Mistral releases, most 7B–70B "vanilla" models).

A mixture-of-experts model replaces that single feed-forward network with many parallel copies ("experts") plus a small router network that picks which few experts handle each token (Hugging Face: Mixture of Experts Explained, ⏱️ verified 2026-07-05). The rest of the model — attention layers, embeddings — stays largely the same between the two designs; MoE specifically multiplies out the feed-forward block.

The oldest well-known example is Mixtral 8x7B: eight experts per layer, but the router only activates two per token. Its total parameter count is about 47B (not the naive 56B, because attention and embedding layers are shared across experts, not duplicated), while its active parameter count per token is about 13B — meaning it needs the memory of a 47B model but computes like a 13B one (Hugging Face MoE explainer, ⏱️ verified 2026-07-05).

Why does memory need the total, but speed need only the active count?

Because of when each number gets used. The honest short version: the router doesn't decide which experts to use until it has already read the token — so every expert has to already be sitting in memory, ready to be picked, before that decision happens. Nothing can be skipped in advance.

Generation speed works differently, because for a single user chatting with the model, speed is a memory-bandwidth problem, not a compute problem: the GPU has to read a token's worth of active weights out of VRAM before it can produce that token (the same principle covered in VRAM: the one number that decides which AI models you can run). A dense model reads its entire weight set every token. A MoE model only reads the weights of the few experts the router actually picked — the rest sit in memory, untouched, for that token. Fewer bytes read per token means fewer milliseconds per token, which is exactly why MoE speed tracks the active count instead of the total.

A real current example: Qwen3.6-35B-A3B

Qwen's Qwen3.6-35B-A3B, released April 2026, is a clean modern illustration of the pattern. Per its official model card (Hugging Face: Qwen/Qwen3.6-35B-A3B, ⏱️ verified 2026-07-05):

At the standard Q4_K_M quantization, the full weight set is a 21.2 GB file (lmstudio-community GGUF listing, ⏱️ verified 2026-07-05) — squarely a 24 GB-card model by the capacity rule, same as any dense 32–35B model. But because only 3B of its 35B parameters activate per token, real-world reports from users running it on a single RTX 3090 describe generation speeds in the tens of tokens per second — in the range you'd expect from a dense model roughly a tenth the size, not from a dense 35B model, which would be markedly slower on the same card (Hugging Face community discussion, community-reported, not an official benchmark — treat as directional, ⏱️ verified 2026-07-05).

That's the whole trick in one model: pay the VRAM bill of a 35B model, get most of the speed of a 3B one.

What does this mean for hobbyist hardware choices?

The practical takeaway: VRAM capacity and memory bandwidth stop being the same shopping decision once MoE is on the table. For a dense model, a card with less VRAM and more bandwidth trades cleanly against a card with more VRAM and less bandwidth — you're always balancing the same equation (bandwidth ÷ size). MoE breaks that trade in your favor: it lets a card with generous VRAM but modest bandwidth (the profile of a lot of "value" AI cards and unified-memory Macs) run genuinely fast, because it never has to read most of what it's holding.

That reframes what to look for:

• If you have VRAM headroom but a bandwidth-limited card, a large MoE model is often your single best move — it uses the capacity you already paid for and sidesteps the bandwidth you didn't.
• The capacity rule never relaxes. A MoE model that doesn't fit its total parameter count still fails to load, exactly like a dense model — the speed win only exists after you've cleared the memory bar.
• A model's name is not enough information. "35B" alone tells you the memory bill. You need the active-parameter number (often written as -A3B, -A22B, etc. in the model name) to know what speed to expect.

When this won't help

• This doesn't make small-VRAM cards suddenly viable for big models. The capacity requirement is unchanged — an 8 GB card still can't hold a 35B MoE model any better than a 35B dense one.
• Quality isn't identical to a dense model of the active size. A 3B active MoE model isn't simply "as good as a 3B dense model" — it draws on more total learned knowledge, but isn't automatically equivalent to a 35B dense model either. Judge each model on its own benchmarks, not just its parameter math.
• Not every "big-sounding" model is MoE. Plenty of current 30B–70B releases are still fully dense, especially older or research-focused models. Check the model card's active-parameter figure before assuming a speed win — if there's no separate active-parameter number listed, treat it as dense.
• Batching and multi-user serving change the math. The active-only speed advantage is clearest for one person chatting at a time; heavy concurrent serving shifts the bottleneck toward compute, where MoE's advantage shrinks.

FAQ

Is a MoE model always faster than a dense model of the same total size? Yes, for single-user generation, because it reads far fewer bytes per token. A 35B MoE model with 3B active will noticeably outrun a 35B dense model on the same hardware — but it still needs the same VRAM to load.

Does a MoE model need less VRAM than a dense model of the same total size? No. Memory needs track total parameters regardless of architecture. A 35B MoE model and a 35B dense model need roughly the same VRAM; only their generation speed differs.

How do I know if a model is MoE or dense? Check the model card for an "active parameters" figure, or a name suffix like -A3B (3B active) or -A22B (22B active) — increasingly common naming in 2026 model releases. No such figure usually means dense.

What was the first well-known MoE model? Mistral's Mixtral 8x7B (2023) popularized the pattern for openly available models — roughly 47B total parameters, about 13B active per token. Qwen3.6-35B-A3B is a more current example of the same idea, with more experts and a narrower active fraction.

Last updated 2026-07-05. Time-sensitive: specific model names, parameter counts, and quantized file sizes shift as new releases ship — check the model's own card on Hugging Face for current numbers before assuming a specific model still matches the figures above.

Sources (verified 2026-07-05): MoE architecture and Mixtral example — Hugging Face: Mixture of Experts Explained; Qwen3.6-35B-A3B architecture — official model card; quantized file size — lmstudio-community GGUF repository; real-world RTX 3090 speed reports — Hugging Face community discussion.