Nvidia · AMD · Blackwell · Nvidia B200 · wafer.ai
GLM5.2 on AMD MI355X at 2626 tok/s/node at over 2x lower cost than Blackwell
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
The demand for inference is skyrocketing and outpacing supply.
Key facts
- The team also hit 213 tok/s on GLM5.2 on 10k input tokens / 1.5k output tokens single stream, following Artificial Analysis standards, served on AMD MI355X capacity from TensorWave
- The team tuned the MoE kernel selection ourselves on GLM’s fp4 shapes ( model_dim 6144, moe_inter 2048, E=256, topk=8 ), which allowed them to reach 2626 tok/s/node at 2.4 RPS
- Second, deep speculative decode (such as the 5/1/6 config z-ai suggests) was still blocked
- At TP8, which was the configuration optimized for single stream decode, the MI355X can run GLM5.2-MXFP4 at 1461 tok/s/node
Summary
At around 2.75x cheaper per GPU on average (MI355X vs B300) with comparable hardware specs, the solution to cheap inference is hiding in plain sight, a message they at Wafer have been preaching for months. Conversely, on the MI355X / ROCm stack SOTA performance rarely comes out of the box for these frontier models (sometimes it does!). But as agents improve at kernel and model optimization, this gap is closing in real time. And again, on a 20k in / 1k out, 60% cache hit rate workload, they hit an aggregate throughput of 2626 tok/s/node @ 2.4 rps with a defined knee of ≤5s TTFT, only 80% of the performance measured on a B200, despite being over 2x cheaper.