Same Weights, Different Answers

I was benchmarking Qwen3.6-27B-FP8 with vLLM on H100 and B200, and the newer GPU had the worse tail latency. The server logs explained part of it: the two machines were not running the same computation. On H100, vLLM ran the FP8 matrix multiplications with DeepGEMM. On B200, it had switched DeepGEMM off and fallen back to CUTLASS.

That switch was deliberate. Issue #37804 reported that Qwen3.5-35B-A3B-FP8 lost about twelve points on GSM8K on B200 when DeepGEMM was active, and PR #38083 responded by disabling DeepGEMM’s FP8 GEMMs for Qwen3.5-architecture models on Blackwell, which includes Qwen3.6-27B.

So the same weights, served with a different kernel, gave measurably different answers. I wanted to see that for myself, and to understand what “lost twelve points” actually means. The answer turned out to be more interesting than “the kernel is less accurate”.

Reproducing the Drop

I ran Qwen3.5-35B-A3B-FP8, the checkpoint from the issue, on one B200 with vLLM 0.28.0 under three server configurations that differ only in how they use DeepGEMM:

On Blackwell, DeepGEMM uses E8M0 scales, which can only represent powers of two. Each block of 128 FP8 values is scaled so its largest value ideally reaches 448. Rounding the scale up to a power of two can nearly double it, shrinking the values by up to half before they are rounded to FP8. Normal FP8 values retain their relative precision, but smaller values suffer. Below 1/64, FP8 loses precision, and sufficiently small values round to zero. Doubling the scale pushes more values into this low-precision range, potentially increasing quantization error.

Here is how I set up each server:

• Default is vLLM 0.28.0 as released, with vLLM’s fix (PR #38083). The fix covers only the linear layers: they run on CUTLASS, but the inputs to the MoE experts still use E8M0 scales.
• DeepGEMM undoes the fix, so the linear layers run on DeepGEMM again, as they did before the fix. The release has no flag for this, so I patched the installed vLLM. The fix adds qwen3_5_text and qwen3_5_moe_text to _DEEPGEMM_BLACKWELL_EXCLUDED_MODEL_TYPES in vllm/utils/deep_gemm.py, the list of model types that skip DeepGEMM on Blackwell. I emptied that list.
• Without DeepGEMM sets VLLM_USE_DEEP_GEMM=0, which turns DeepGEMM off completely, so no layer uses E8M0 scales.

I ran three GSM8K checks against each server, three times each, on all 1,319 test questions, 5-shot, with greedy decoding:

• vLLM’s GSM8K script, the check used in the issue and in vLLM’s CI gate for this checkpoint.
• lm-eval’s GSM8K task, the standard implementation, from lm-evaluation-harness 0.4.12, the version vLLM’s CI uses.
• A probe that sends the same prompts as vLLM’s script but saves every answer, so two servers can be compared question by question. It runs once with the script’s 256-token output limit and once with 1,024 tokens.

The table shows the mean of three runs, with the lowest and highest run in parentheses:

vLLM’s script reproduces the issue: DeepGEMM costs 8.7 points against the default server. That is less than the issue’s twelve points, which came from a different setup (two GPUs with data and expert parallelism), but it leads to the same conclusion: DeepGEMM lowers the score far beyond the run-to-run spread.

lm-eval sees no drop beyond the noise. All three servers land within 0.7 points of each other, and DeepGEMM is not even the lowest.

Two Harnesses, Two Verdicts

Same checkpoint, same three servers, same 1,319 questions. One harness says the kernel costs nine points, the other says it costs none. Which one is right?

The two harnesses differ in several ways (stop strings, answer extraction, which five examples they show), but the one I think matters most is how the prompt ends.

vLLM’s script sends one plain string to /v1/completions, and the model continues after the last Answer::

Question: Natalia sold clips to 48 of her friends in April, ...
Answer: Natalia sold 48/2 = <<48/2=24>>24 clips in May. ...
#### 72

(four more examples)

Question: Janet’s ducks lay 16 eggs per day. ...
Answer:

lm-eval sends chat messages to /v1/chat/completions with thinking turned off. The chat template renders them with Qwen’s special tokens and ends the prompt with an already-closed thinking block:

<|im_start|>user
Question: Janet’s ducks lay 16 eggs per day. ...
Answer:<|im_end|>
<|im_start|>assistant
<think>

</think>

Qwen3.5 thinks by default (model card). Under the raw frame there is no closed thinking block, so the model writes its own <think> in nearly every response. Then it makes a choice: close the block at once and answer, or start reasoning. lm-eval’s prompt has already closed the block, so the model never makes that choice.

Where the Points Went

The probe makes it possible to sort every response by what happened to it. Averaged over three runs, per 1,319 questions:

The accuracy of finished answers is the column that changed my reading of the issue. Answers that finished were 96–97% correct on every server. DeepGEMM barely changes the accuracy of finished answers. Instead, it makes the model start reasoning in 120 to 150 more responses per run, and reasoning is where most answers fail to finish.

They fail in two ways. Some run into the output limit. Others stop early: Question is one of vLLM’s stop strings, and when the reasoning itself writes that word, the server ends the response mid-thought. Here is one DeepGEMM response in full:

<think>
The user wants me to solve a word problem about a merchant choosing between two investment plans.

**1. Analyze the input:**
- **Option 1:** Jewelry worth $5,000.
- **Option 2:** Electronic gadgets worth $8,000.
- **Projected Growth (Jewelry):** 2.5% increase.
- **Projected Growth (Electronics):** 1.2% increase.
- **Goal:** Maximize profit.
- **

The model’s next word was Question, so the server stopped there and, as it does with stop strings, left the word out of the text. The scorer took the last number in what remained, the 2 in “1.2%”, as the answer. The correct answer is 125. Sometimes the last number happens to be right, so the probe score is a little higher than the finished answers alone would give.

Not every unfinished answer started with reasoning. At 256 tokens, “Never finished” is higher than “Started reasoning” on every server, so at least 65 responses per run failed to finish without reasoning at all. At 1,024 tokens, the two counts nearly match.

A longer output limit does not close the gap. At 1,024 tokens, DeepGEMM still leaves 148 more answers unfinished than the default server and scores 9.6 points lower.

The server without DeepGEMM moves slightly the other way. It starts reasoning in 33 fewer responses than the default server at 256 tokens, and in 10 fewer at 1,024 tokens. That difference is close to the noise: the output limit cannot affect the first token after <think>, yet the default server’s own count changes by 23 between the two limits. So the result fits the E8M0 scales the default server still applies to its MoE inputs, but it does not show that they matter.

What I Take From This

The usual framing is simple: a model’s accuracy depends on how it is served. That is true, and it is the headline of work like Artificial Analysis’s Endpoint Accuracy Index, which compares providers serving the same open-weight model, and Yuan et al. (2025), who found up to 9% accuracy variation, and up to 9,000 tokens difference in response length, from changing only batch size, GPU count and GPU type.

But this experiment showed me something narrower and, I think, more useful:

• A kernel can change behavior while finished answers stay as accurate. DeepGEMM shifted a decision the model makes before it starts solving, and that changed how many answers reached the end.
• Whether you see the change depends on the harness. One harness turned it into nine lost points. The other, which closes the thinking block for the model, saw nothing.
• The shift still matters in production. More reasoning means more tokens, higher latency, and more truncated answers.
• Score breakdowns beat scores. The headline said “this kernel is worse”. Sorting responses by how they ended said where, and why.