AMD MI355X Beats B300 on Performance per Dollar for Kimi K3

Wafer reports running Moonshot AI's Kimi K3 — a 2.8-trillion-parameter model — on a single node of 8 AMD MI355X GPUs at 952 tok/s aggregate throughput and 118 tok/s single-stream decode. That's over 3.8× the aggregate throughput per node and 1.3× the single-stream decode of a dual-node B200 deployment (498 tok/s across 16 GPUs). At $2.50/GPU-hr for MI355X vs $6.00 for B300, the MI355X delivers 48 tok/s per dollar per GPU-hour — 7× better than B200's 7 tok/s/$ and 1.45× better than B300's 33 tok/s/$.

B300 nodes still win on raw throughput (1,568 tok/s), but at 2.4× the price, the MI355X crushes it on cost efficiency. This is one of the first frontier-scale models where AMD's HBM capacity (288GB per GPU, same as B300) gives a practical edge over B200's 192GB. Kimi K3 won't fit weights plus a 1M-token KV cache on a single B200 node, forcing a two-node TP16 setup. The MI355X fits it on one node (TP8), avoiding cross-node all-reduce overhead.

The Software Gap: One Missing Function, One Shape Mismatch

Kimi K3 ships with zero draft tensors (no MTP, no EAGLE), so Wafer used an external block-diffusion draft: RadixArk's Kimi-K3-DSpark. On CUDA it runs out of the box, but on ROCm it hit a scheduler crash:

NameError: name 'top_k_renorm_prob' is not defined. Did you mean: 'top_p_renorm_prob'?

SGLang's accept-sampling verifier has two paths: a dense path that calls top_k_renorm_prob, and a sparse fast path that routes through torch.topk. The CUDA build imports top_k_renorm_prob from sgl_kernel; the ROCm build only aliases a Triton top-p kernel, leaving top_k_renorm_prob undefined for gfx950. The fix was a single PyTorch function: sort, masked_fill, divide — no custom kernel needed.

With spec decode fixed, single-stream performance jumped 2.2×, per-stream at moderate load improved 1.7×, and peak aggregate throughput rose 18%. Peak aggregate also landed at much higher concurrency (c64 vs c24 without spec).

Prefill: The Real Bottleneck

Decode tok/s is over-glorified. Time-to-first-token (TTFT) is what users feel. On a 172k-token cold prefill, MI355X took ~51s vs B300's ~23s — a 2.2× gap. The culprit was one kernel: K3 on ROCm fell back to slow generic Triton attention because AITER's fast MLA prefill kernel wouldn't load. The problem was a shape mismatch — K3 at TP8 gives 12 attention heads per rank, but AITER's MLA path only supports 4, 8, or multiples of 16. The fix: zero-pad the head count 12→16, run the fast kernel, extract the real 12 heads from the output.

Result: steady-state prefill speed jumped from ~4–7k tok/s (Triton fallback) to ~13k tok/s (AITER ASM), a 2–3× improvement. This doesn't change aggregate throughput but dramatically cuts TTFT.

Takeaways for Developers

  1. AMD's software story is improving fast. Wafer says Kimi K3 on MI355X required "fewer bugs than GLM5.2" and no custom kernels. Day-0 ROCm support from AMD helped, but the remaining issues were trivial to fix.

  2. Performance per dollar is the new battleground. At $2.50/GPU-hr, MI355X delivers 48 tok/s/$ — 1.45× better than B300 and 7× better than B200. For inference workloads, that's a massive cost advantage.

  3. Check your framework's kernel coverage. The prefill issue was a head-count mismatch, not a missing kernel. Before writing custom kernels, verify that your model's config matches the fast path's assumptions.

What You Should Do Now

If you're serving large MoE models, benchmark on MI355X. Wafer's numbers are specific to Kimi K3, but the pattern holds: AMD's HBM capacity and price point make it a serious contender for frontier-scale inference. Watch for more day-0 support from AMD and framework fixes like the top-k renorm patch.

Method Notes

Wafer used SGLang with RadixArk's DSpark draft. Benchmarks: 1,024-token input / 400-token output. B200 numbers are deflated by cross-node RoCE v2 at ~195 Gb/s all-reduce on the decode critical path. B300 uses TP8+DCP8. Prices: MI355X $2.50/GPU-hr, B300 $6.00, B200 $4.25.