EAGLE-4 on AMD Instinct GPUs from day 0

Oct 20, 2026

EAGLE-4, the newest drafter from the EAGLE team, runs on AMD Instinct™ MI355X GPUs from day 0, in SGLang and ATOM. - On Qwen3.8-27B, one MI355X GPU serves a single request at ~300 tokens/s in BF16: 3.5x plain decoding and 1.6x the model’s built-in MTP. - The gain holds as load grows, from 1 to 32 concurrent conversations, and scales to four GPUs. It works out of the box: the same checkpoint and the same serving flags as on other hardware.

Figure 1: EAGLE-4 at one request
Figure 1: EAGLE-4 at one request

What is new in EAGLE-4

Speculative decoding lets a small drafter guess several tokens ahead, and the large model checks them all in one pass. EAGLE-4 adds three ideas on top of the EAGLE family:

  • One drafter for every batch size and every GPU. EAGLE-4 drafts one token at a time, so it can stop, deepen or widen its guess tree at every step. EAGLE-Harness measures what drafting and verifying cost on the actual hardware, once at start-up. During serving it picks the tree that gives the most accepted tokens per unit of cost: a wide, deep tree for a single user, short chains under heavy load.
  • A cheaper LM head for the drafter. The drafter’s vocabulary projection used to be most of its cost. A low-rank head first shortlists candidates from the full vocabulary, then the full head scores only those exactly. This cuts the draft’s cost by more than 40% without truncating the vocabulary.
  • A small drafter. One transformer layer, so it fits next to the model with little memory, from data-center GPUs down to edge devices.

Day 0 on AMD Instinct GPUs

EAGLE-4 runs on the MI355X GPU with the same checkpoint and the same launch flags as elsewhere:

To make it fast on day 0, we measured EAGLE-Harness’s cost tables on the MI355X GPU, so tree sizes fit this GPU; added tree verification for Qwen3.8’s linear-attention (Gated DeltaNet) layers: a guess tree is checked in one pass, and only the accepted branch’s state is kept; checked losslessness: every output token matches plain decoding, up to BF16 rounding at near-ties.

Results

Qwen3.8-27B, BF16 weights, one MI355X GPU, SPEED-Bench (the EAGLE-4 paper’s benchmark: thinking on, greedy).

Concurrent conversations

1

4

16

32

Plain decoding (SGLang)

82

292

872

1,285

Built-in MTP (SGLang)

178

617

1,743

2,551

EAGLE-4, SGLang

289

941

2,167

2,784

EAGLE-4, ATOM

309

976

2,309

3,243

 

Figure2: EAGLE-4 against load
Figure2: EAGLE-4 against load
  • One request: EAGLE-4 gets ~4.7 tokens accepted per check, against ~2.9 for the built-in MTP. That is where the 1.6x comes from.
  • Under load: EAGLE-4 shortens its guesses as the GPU fills up, so it stays ahead of MTP at 32 conversations.
  • More GPUs: with tensor parallelism, one request reaches 403 tok/s on two MI355X GPUs (ATOM) and 463-489 on four (SGLang / ATOM).

Four GPUs

Tensor parallel across four MI355X GPUs, same benchmark:

Figure 3: EAGLE-4 on four GPUs
Figure 3: EAGLE-4 on four GPUs

Concurrent Conversations

1

4

16

32

Plain decoding (SGLang)

156

539

1,655

2,331

Built-in MTP (SGLang)

303

1,036

2,711

3,916

EAGLE-4, SGLang

463

1,360

3,339

4,474

EAGLE-4, ATOM

489

1,347

3,381

4,796

With SGLang, EAGLE-4 is 3.0x plain decoding and 1.5x MTP for one request on four GPUs, and still 1.9x and 1.14x at 32 conversations.

What’s next

We would love to bring EAGLE-4 to more models, and we are combining it with our single-request megakernels for even lower latency(Link to blog). We welcome collaboration with the EAGLE team, model builders and the community.

Test configuration

  • Hardware: AMD Instinct MI355X GPU (gfx950); TP1 unless stated. AMD ROCm™ software 7.2.4.
  • Model: Qwen/Qwen3.8-27B in BF16; draft: the EAGLE-4 checkpoint for Qwen3.8-27B.
  • Benchmark: SPEED-Bench qualitative split, every 10th conversation of each category (88 conversations), replayed turn by turn; thinking on, temperature 0; tok/s = generated tokens over wall-clock time.
  • Engines: SGLang: the EAGLE-4 fork at main 2af5570 with the ROCm PR rebased on it; ATOM: branch eagle4_day0_support of proaditya/ATOM, on ATOM main 2479251d3, in rocm/atom-dev:nightly_202610131505 (ROCm 7.2.4, with aiter and FlyDSL).. SGLang uses EAGLE-Harness trees at every concurrency shown; ATOM uses fixed trees per batch size up to 16 conversations and chains at 32.

Reproducing the numbers

 

SGLang

ATOM

EAGLE-4, Qwen3.8-27B

Available (ROCm PR)

Available (eagle4_day0_support)

Both engines are measured with the same SPEED-Bench client and data from the EAGLE-4 repository, inference/scripts/client_speedbench.py and inference/dataset/speed_bench/question_multiturn.jsonl; for ATOM, the client reads token counts from the response’s usage field. The client replays each conversation turn by turn (thinking on, reasoning effort xhigh, temperature 0); the reported number is wall_tput, generated tokens over wall-clock time. --every 10 is the 88-conversation subset used in the tables; omit it for all 880.

ATOM: one command serves each concurrency with the recipe’s flags, runs the client and prints the results next to ours; the README has the options (TP size, concurrencies) and the recipe the flags behind them:

EAGLE4=<EAGLE-4 checkout> DRAFT=<EAGLE-4 checkpoint> bash blog_benchmark/eagle4/build_and_repro.sh

README · recipe · script

SGLang (the EAGLE-4 repository’s SGLang, on the ROCm SGLang image for the MI355X GPU):

		export PYTHONPATH=$EAGLE4/sglang-v0.5.20/python
export FLYDSL_RUNTIME_CACHE_DIR=/tmp/flydsl_cache     # writable; aiter compiles new GEMM shapes here
# 1. Once per GPU and TP size: measure EAGLE-Harness's cost tables (about 10-20 minutes)
cd $EAGLE4/inference/scripts
python engine_cost.py --base-model-path $MODEL --fast-model-path $DRAFT \
    --draft-tree-path $COST_ROOT --environment-str qwen38-eagle4-mi355x-tp1 \
    --speculative-model EAGLE4 --compute-intensity high_intensity --model-data-type bfloat16 \
    --attention-backend triton --mm-attention-backend triton_attn --page-size 1 --mamba-ssm-dtype bfloat16 \
    --speculative-draft-attention-backend fa3 --speculative-draft-model-quantization unquant \
    --tp-size 1 --start-bs 1 --max-draft-batch-size 32 --context-step 256 --context-stop 32768 \
    --enable-cuda-graph --enable-fast-measurement --mem-fraction-static 0.8
# 2. Serve (C = concurrent conversations)
SGLANG_GDN_TREE_RECORDS=1 SGLANG_GDN_TREE_CHUNK=1 SGLANG_GDN_TREE_CHUNK_FLYDSL=1 \
python -m sglang.launch_server --model-path $MODEL --tp-size 1 --dtype bfloat16 --trust-remote-code \
    --mem-fraction-static 0.80 --context-length 36864 --chunked-prefill-size 16384 \
    --attention-backend triton --mm-attention-backend triton_attn --page-size 1 --mamba-ssm-dtype bfloat16 \
    --disable-radix-cache --reasoning-parser qwen3 --random-seed 721 \
    --max-running-requests $C --cuda-graph-max-bs-decode $C \
    --disable-overlap-schedule --disable-cuda-graph-padding --disable-prefill-cuda-graph --cuda-graph-backend-decode full \
    --speculative-algorithm EAGLE4 --speculative-draft-model-path $DRAFT --speculative-draft-model-quantization unquant \
    --speculative-draft-attention-backend fa3 --speculative-attention-mode prefill --speculative-exec-mode 0 \
    --speculative-num-steps 12 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \
    --speculative-compute-intensity high_intensity --speculative-data-path $COST_ROOT/qwen38-eagle4-mi355x-tp1 --port 30000

The three SGLANG_GDN_TREE_* switches turn on the FlyDSL tree verification for the linear-attention layers (measure the cost tables with them set as well). For TP > 1, use --tp-size N in both commands and a cost table per TP size. Plain decoding: the same server command without the --speculative-* flags; built-in MTP: --speculative-algorithm EAGLE --speculative-draft-model-path $MODEL --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4.

Share:

Related Blogs

View All Blogs