Serving a 307 GiB Model on Hardware You Own
A 307.49 GiB compressed checkpoint served through vLLM across four DGX Spark nodes and across three H200s — plus the engine comparison that shows when a right-sized 35B MoE is the better answer.
The problem
Two separate questions get collapsed into one. The first is whether a very large model can be served privately at all on hardware an organization already owns: the Kimi-K3 Neuron IQ1_S GGUF is 330.2 GB / 307.49 GiB, which means tensor parallelism across physical nodes, a preflight memory contract, and — as it turned out — four real bugs across vLLM and the GGUF plugin standing between the checkpoint and a first token. The second question is whether the largest model is the right one. On a single DGX Spark, a ~21.4 GB NVFP4 mixture-of-experts checkpoint (Ornith-1.5-35B-A3B, ~3B active parameters per token) serves its full native 262144 context at 74.8–74.9 decode tok/s on one node with no cluster at all. The measurements below cover both paths so the tradeoff is visible instead of asserted, and they include the single highest-impact finding: on the H200 TP3 path, the default compile mode was wrong, and fixing it moved decode from 6.632 to 34.875 tok/s before any speculative decoding was involved.
How it was measured
Size the checkpoint against the node budget before touching a serve script. At 307.49 GiB across four GB10 nodes, steady state is ~78.5 GiB per rank against a 90 GiB preflight free-memory requirement. TP4 is the only viable configuration on four Sparks — and it is the one parallelism value that is not natively block-aligned, which is why the patch set exists.
Fix upstream before benchmarking, and ship the fix as a source overlay rather than a fork. Two small patches on top of a pinned vLLM commit (75231eff2f3873e2bce7cc9558bb5227ea70b808) and the installed GGUF plugin (d94067060884ea87766f12010c3a8b9c2d6715cc), applied with apply_patches.py, with a preflight architecture probe and a verification script. Four bugs had to be found and fixed to get a correct first token.
Move the same overlay onto denser hardware and audit the defaults. On 3× H200 (sm_90) the eager path measures 6.632 tok/s; switching to PIECEWISE CUDA graphs measures 34.875 tok/s target-only. Same weights, same patches, same node count.
Add the released DSpark speculative draft and measure per workload rather than per model. Prose at draft depth N=2 measures 42.464 tok/s; coding at N=3 measures 52.454 tok/s. Each extra speculative token costs ~5.95 ms against a 28.67 ms M=1 decode latency, which is what makes depth a workload decision.
Separate single-stream latency from aggregate throughput. On the same H200 TP3 node set, batch 2 with DSpark measures 51.4 tok/s, batch 4 measures 76.9 tok/s, and batch 8 target-only measures 88.5 tok/s. Capacity planning uses the batched figure; user-perceived latency uses the single-sequence figure.
Ask whether the large model is needed at all. Against the same NVFP4 checkpoint, at TP=1, at full 262144 context, vLLM and SGLang were run head to head on one Spark with uncached prefill (unique-nonce prompts so prefix caching cannot inflate the number) and the same evaluation harnesses: decode 74.8 vs 74.9, sixcat-eval overall 85.0 vs 87.1, hermes-agentic-bench 14/20 vs 16/20, cold start ~3 min vs ~60 min.
Benchmarks
Every value below is a published measurement. Dashes are configurations that were not measured — not zeros, and not estimates.
| Model / checkpoint | Hardware | Serving config | Decode tok/s | Prefill tok/s (uncached) | Quality (sixcat overall) |
|---|---|---|---|---|---|
| Kimi-K3 Neuron IQ1_S GGUF — 330.2 GB / 307.49 GiB | 4× NVIDIA GB10 (DGX Spark), sm_121 | vLLM TP4, --enforce-eager, single sequence | 5.076 (wall-clock) | — | — |
| Kimi-K3 Neuron IQ1_S GGUF — 307.49 GiB | 3× H200 (sm_90) | vLLM TP3, eager | 6.632 | — | — |
| Kimi-K3 Neuron IQ1_S GGUF — 307.49 GiB | 3× H200 (sm_90) | vLLM TP3, PIECEWISE graphs, target-only | 34.875 | — | — |
| Kimi-K3 Neuron IQ1_S GGUF — 307.49 GiB | 3× H200 (sm_90) | vLLM TP3 + DSpark draft N=2 (prose) | 42.464 | — | — |
| Kimi-K3 Neuron IQ1_S GGUF — 307.49 GiB | 3× H200 (sm_90) | vLLM TP3 + DSpark draft N=3 (coding) | 52.454 | — | — |
| Kimi-K3 Neuron IQ1_S GGUF — 307.49 GiB | 3× H200 (sm_90) | vLLM TP3 + DSpark, batch 2 | 51.4 (aggregate) | — | — |
| Kimi-K3 Neuron IQ1_S GGUF — 307.49 GiB | 3× H200 (sm_90) | vLLM TP3 + DSpark, batch 4 | 76.9 (aggregate) | — | — |
| Kimi-K3 Neuron IQ1_S GGUF — 307.49 GiB | 3× H200 (sm_90) | vLLM TP3, target-only, batch 8 | 88.5 (aggregate) | — | — |
| Ornith-1.5-35B-A3B-NVFP4 — ~21.4 GB | 1× DGX Spark (GB10, ~128 GB unified) | vLLM TP=1, 262144 context | 74.8 (74.5 overall t/s @200 tok out) | 4083 @43k · 2278 @~180k | 85.0 |
| Ornith-1.5-35B-A3B-NVFP4 — ~21.4 GB | 1× DGX Spark (GB10, ~128 GB unified) | SGLang TP=1, 262144 context | 74.9 (71.6 overall t/s @200 tok out) | 4839 @43k · 2513 @~180k | 87.1 |
What the numbers cost
Fitting a 307.49 GiB model across four Sparks proves feasibility, not throughput. At 5.076 tok/s single-sequence under --enforce-eager it is an architecture result — the model runs entirely inside your boundary — and it is roughly an order of magnitude below the 52.454 tok/s the same weights reach on 3× H200 with graphs and speculative decoding. If interactive latency is the requirement, that gap is the procurement conversation.
Speculative decoding changes the output. Speculative results will not match non-speculative output token for token, and that is expected behavior, not a defect — but any evaluation, diff, or regression test that assumes determinism has to be rewritten before the speed is usable.
The engine comparison has no single winner. SGLang scored higher on both harnesses (85.0 → 87.1 sixcat, 14/20 → 16/20 hermes-agentic-bench) and led uncached prefill at both 43k and ~180k, while vLLM cold-started in ~3 min against ~60 min and warm-started in ~3 min against ~6 min. Long-context and quality-sensitive work points to SGLang; fast restarts and serve-config iteration point to vLLM. Note also that hermes-agentic-bench is a 20-task battery, so a two-task difference is a narrow margin.
Both agentic scores were produced by the simulated contract layer against an OpenAI-compatible endpoint, and the harness itself has known disagreement between its simulated and native layers — which is why both layers exist. Treat 14/20 vs 16/20 as directional.
What this does not show
- No prefill was measured for any Kimi-K3 configuration; those cells are em-dashes.
- Kimi-K3 has no quality evaluation on any topology. Whether IQ1_S compression holds task quality is untested here.
- The Ornith engine comparison ran once per engine on one date with no repeat-run variance, so those gaps carry no error bars.
- Batch figures are aggregate throughput, not single-stream latency. The two are labelled separately above.
- No security, air-gap, or network-boundary validation evidence is presented on this page.
Run it yourself
Kimi-K3 TP4: exact commit hashes pinned (vLLM 75231eff2f3873e2bce7cc9558bb5227ea70b808, vllm_gguf_plugin d94067060884ea87766f12010c3a8b9c2d6715cc validated against pip 0.0.4, gguf==0.19.0), with published install steps, apply_patches.py, download commands, configuration template, launch script, and a verification script; hardware-specific to 4× GB10. Kimi-K3 TP3: same pinned bases, build scripts in APPLY.md and scripts/build_from_source.sh, FLASH_ATTN_MLA / TRITON_MLA backends, preflight_arch.py architecture probe; validated on sm_90 only — B200, GB200, RTX PRO 6000, RTX 5090, A100 and DGX Spark are documented but untested for this recipe and require running the preflight probe. Ornith: measured 2026-08-19 against the ornith-ai/Ornith-1.5-35B-A3B-NVFP4 checkpoint on vLLM 0.1.dev1+g75231eff2.d20260809 and SGLang 0.5.18.dev760+ge5a3e4d30, PyTorch 2.13.0+cu130, FlashInfer 0.6.17, CUDA 13.0, Python 3.12, aarch64; full serve scripts (logs/serve.sh, logs/serve-sglang.sh) and probes (logs/quick_tps_bench.py, logs/prefill_uncached.py) published, all prefill measured with unique-nonce prompts.