Two Qwen3.8 Models on One DGX Spark
A dense 27B and a 180B mixture of experts on one Spark: what one token costs on each, and why the speed of the big one depends on the text.
In the last post I worked out speed limits for the Sparks on paper. Since then Qwen3.8 came out, so I measured two of its models, each on one Spark (TP=1).
The benchmark is the same script as in the 4090 post: one user, prompts of about 200 tokens, every answer fixed at 150 tokens. The prompts are random filler text, so the runs are comparable.
The dense one: Qwen3.8-27B
I ran the FP8 version on the official vLLM release, without patches. A dense model reads all of its weights for every token. Here that is about 30.6 GB, and at 273 GB/s that takes 112 ms.
The measured time is 118 ms per token, or 8.4 tokens per second. That is 95% of the limit, the same result as on the 4090.
The big one: Qwen3.8-Flash-Next
This is NVIDIA's NVFP4 version of the model. It has 180B parameters in total:
| Part | Parameters |
|---|---|
| Mixture of experts | 125B, 6B active per token |
| Lookup table | 51B |
| Draft model | 4B |
On disk it is 124 GiB. The Spark shows 121 GiB of memory.
It fits because the lookup table, 47.7 GiB, stays on the SSD. The server maps the file into memory and reads only the rows it needs. Everything else takes 81.6 GiB, and 8.68 GiB of that is KV cache, enough for 517,734 tokens. About 20 GiB of the machine stays free. To run it I used a community recipe for a single Spark, which patches vLLM. It takes 12 minutes from start to the first request.
With the random prompts it produces 18 tokens per second, 55 ms per token. That is about twice as fast as the 27B, with more than six times the parameters.
The active weights are about 6 GB, which is 23 ms of reading at 273 GB/s. One step of the model takes about 75 to 80 ms, so here the bandwidth rule does not set the speed. On every step the lookup table goes through the CPU, and that costs more than the read.
Real prompts
The draft model guesses the next three tokens. The main model checks them in one step and keeps the ones that are right. Random filler cannot be guessed, so on the benchmark above the draft almost never helps.
So I also ran six real prompts, three times each, at temperature 0.7. These are the medians:
| Prompt | Tokens per second | Tokens per step |
|---|---|---|
| Write JSON | 43.9 | 3.42 |
| Write code | 41.7 | 3.46 |
| Rewrite a paragraph | 37.9 | 2.97 |
| Explain a concept | 37.1 | 3.02 |
| Write a story | 28.8 | 2.13 |
| Translate to Turkish | 24.2 | 1.96 |
A step takes about 75 to 80 ms for every prompt. What changes is how many tokens one step produces. JSON and code are predictable, so most of the guesses are right. A story or a Turkish translation is harder to guess. The rewrite and translation answers were short, 97 and 47 tokens, so those two rows are less precise.
On this model the same Spark gives 18 to 44 tokens per second, depending on the text.