Day 0 Support for Qwen 3.8 on AMD Instinct GPUs

Aug 12, 2026

AMD is excited to announce Day-0 support for Alibaba's latest Qwen 3.8 model family on AMD Instinct™ MI300X, MI325X, and MI355X GPUs. Developers can immediately deploy and evaluate Qwen 3.8 using the AMD ROCm™ software together with SGLang and vLLM. Building upon the innovations introduced in Qwen3.6, Qwen3.8 introduces a significantly expanded sparse Mixture-of-Experts (MoE) architecture featuring 512 experts, a deeper 92-layer network, and a hybrid linear-attention and full-attention design optimized for long-context workloads.

Key Takeaways

Day-0 support across the AMD Instinct GPUs
Optimized ROCm enablement
SGLang /vLLM/ATOM readiness
Performance highlights and benchmark placeholders
Quickstart deployment instructions

What’s New in Qwen 3.8?

For the first time, Qwen3.8 brings a Qwen-Max-class model to open release. Built on the architectural foundation of Qwen3.5, Qwen3.8 delivers substantial gains across coding, professional work, research, and long-horizon agentic tasks. Beyond answering harder questions, Qwen3.8 is designed to carry complex, multi-step tasks through to completion with greater reliability.

Qwen3.8 features the following enhancements:

Core Capabilities: Comprehensive improvements across coding, professional work, research, and long-horizon agentic tasks.
Agent Execution: Stronger autonomous planning and better handling of environment feedback, leading to more reliable end-to-end task completion.
Downstream Compatibility: Broader support for popular harnesses and development tools, making it easier to integrate into your existing stack.
Flexible Thinking Control: Reasoning depth can be tuned with reasoning_effort, and reasoning context from historical messages is retained via preserve_thinking.

Developer Quickstart: Deploying Qwen 3.8 on AMD Instinct GPUs

The following section outlines the steps to deploy Qwen 3.8 on AMD Instinct GPUs using vLLM and SGLang.

How to Run Qwen 3.8 with vLLM on AMD Instinct GPUs

FP8 Deployment

vLLM Docker: docker pull vllm/vllm-openai-rocm:qwen38

Configuration:

Two MI355X Nodes:
Parallelism: TP=8, PP=2
Ray: 2 nodes, 16 GPUs
API: http://10.24.112.181:8000/v1

Env args:

		 export RAY_ADDRESS=10.24.112.181:6380 
 export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 
 export GLOO_SOCKET_IFNAME=eno0 
 export NCCL_SOCKET_IFNAME=eno0 
 export RCCL_SOCKET_IFNAME=eno0 
 export VLLM_HOST_IP=10.24.112.181 
 export VLLM_ROCM_USE_AITER=1 
 export SAFETENSORS_FAST_GPU=1

vLLM Serve Command

		vllm serve "$MODEL" \ 
   --host 0.0.0.0 \ 
   --port 8000 \ 
   --trust-remote-code \ 
   --tensor-parallel-size 8 \ 
   --pipeline-parallel-size 2 \ 
   --distributed-executor-backend ray \ 
   --max-model-len 32768 \ 
   --max-num-batched-tokens 32768 \ 
   --gpu-memory-utilization 0.8 \ 
   --reasoning-parser qwen3 \ 
   --served-model-name Qwen/Qwen3.8-2.4T-A95B-FP8 \ 
   --language-model-only \ 
   --no-enable-prefix-caching \ 
   --mamba-cache-mode none

Examples:

Endpoint: /v1/chat/completions
Model: Qwen/Qwen3.8-2.4T-A95B-FP8
Streaming: enabled

Prompt:
Say hello in one short sentence.

Reasoning:
The user wants me to say hello in one short sentence.

Final response:
Hello, it's great to meet you!

Accuracy evaluation:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.9591	±	0.0055
		strict-match	5	exact_match	↑	0.9591	±	0.0055

MXFP4 deployment

Configuration:

One MI355X Node:
Parallelism: TP=8
API: http://10.24.112.181:8000/v1

Env args:

		'export VLLM_ROCM_USE_AITER=1;  

export SAFETENSORS_FAST_GPU=1; 

 export VLLM_USE_BREAKABLE_CUDAGRAPH=0;  

export VLLM_ROCM_USE_AITER_MOE=1;  

export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1;

Model Path:

		MODEL=/mnt/hf_hub_cache/Qwen3.8-2.4T-A95B- Quark-MXFP4;

vLLM Serve Command:

		vllm serve \"\$MODEL\" --host 0.0.0.0 --port 8000 --trust-remote-code --tensor-parallel-size 8 --max-model-len auto --max-num-batched-tokens 32768 --gpu-memory-utilization 0.9 --reasoning-parser qwen3 --no-enable-prefix-caching --mamba-cache-mode none --no-async-scheduling

Examples:

Qwen3.8 MXFP4 is running successfully:

Container: vllm-qwen-head
Endpoint: http://mia1-p01-g05:8000
TP: 8 GPUs
Health: HTTP 200
Model listing: HTTP 200
Chat completion: HTTP 200
Log: /tmp/vllm-qwen38-mxfp4-final.log

Sample request succeeded with HTTP 200.

Prompt: “What is 2 + 2? Answer briefly.”

Response: 4

The Qwen3.8 MXFP4 server is generating correctly.

Accuracy evaluation:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.956	±	0.0056
gsm8k	3	strict-match	5	exact_match	↑	0.9568	±	0.0056

How to Run Qwen 3.8 with SGLang on AMD Instinct GPUs

Prerequisites

AMD Instinct MI300X or MI355X with ROCm configured
Model checkpoints:
- MXFP4: /data/models/Q Qwen3.8-2.4T-A95B- Quark-MXFP4 (Quark-produced)
- FP8: /mnt/models/Qwen/Qwen3.8-2.4T-A95B-FP8
ROCm-optimized SGLang container (match image to GPU)

Step 1: Launch Docker Container

MI355X (recommended for MXFP4):

		docker pull rocm/sgl-dev:v0.5.8.post1-rocm720-mi35x-<date> 
docker run -it \ 
--device /dev/dri --device /dev/kfd \ 
--network host --ipc host \ 
--group-add video \
--security-opt seccomp=unconfined \ 
-v /data/models:/data/models \ 
-v $(pwd):/workspace \ 
rocm/sgl-dev:v0.5.8.post1-rocm720-mi35x-<date> /bin/bash

MI300X (for FP8 two-node setup):

		docker pull rocm/sgl-dev:v0.5.8.post1-rocm720-mi30x-20260215 
docker run -it \ 
--device /dev/dri --device /dev/kfd \ 
--network host --ipc host \ 
--group-add video \ 
--security-opt seccomp=unconfined \ 
-v /mnt/models:/mnt/models \ 
-v $(pwd):/workspace \ 
rocm/sgl-dev:v0.5.8.post1-rocm720-mi30x-20260215 /bin/bash

Step 2: Start the SGLang Server

Option A — MXFP4 on MI355X (single node, TP8) ✅ Recommended

Qwen 3.8 uses the AITER attention backend (not Triton) on MI355X, with unified attention and FlyDSL enabled:

		#!/usr/bin/env bash

  export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

  cd /sgl-workspace/sglang

  python3 -m sglang.launch_server \
    --model-path /mnt/models/Qwen/Qwen3.8-2.4T-A95B-FP8-MXFP4 \
    --served-model-name /mnt/models/Qwen/Qwen3.8-2.4T-A95B-FP8-MXFP4 \
    --tp-size 8 \
    --attention-backend aiter \
    --page-size 1 \
    --kv-cache-dtype auto \
    --chunked-prefill-size 16384 \
    --watchdog-timeout 1200 \
    --mem-fraction-static 0.9 \
    --model-loader-extra-config '{"enable_multithread_load": true}' \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 9001

Option B — FP8 on 2× MI300X (TP8 PP2)

FP8 weights require two nodes (16 GPUs total). Use TP8 PP2, not TP16 across nodes (cross-node all-reduce is very slow).

		#!/usr/bin/env bash 
Rank0 
# On MI300X, ensure MI30x Docker image is used. 
  export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 
  export GLOO_SOCKET_IFNAME=eno8303 
  export NCCL_SOCKET_IFNAME=eno8303 
  export RCCL_SOCKET_IFNAME=eno8303 
  
  cd /sgl-workspace/sglang 
   
  python3 -m sglang.launch_server \ 
    --model-path /mnt/models/Qwen/Qwen3.8-2.4T-A95B-FP8 \ 
    --tp-size 8 \ 
    --pp-size 2 \ 
    --nnodes 2 \ 
    --node-rank 0 \ 
    --dist-init-addr 10.194.129.192:25000 \ 
    --dist-timeout 3600 \ 
    --attention-backend aiter \ 
    --chunked-prefill-size 8192 \ 
    --watchdog-timeout 1200 \ 
    --mem-fraction-static 1.0 \ 
    --host 0.0.0.0 \ 
    --port 9001 \ 
    --disable-radix-cache \ 
    --disable-custom-all-reduce \ 
    --page-size 16 \ 
    --kv-cache-dtype fp8_e4m3 \ 
    --cuda-graph-max-bs-decode 64 \ 
    --reasoning-parser qwen3 \ 
    --tool-call-parser qwen3_coder 
 
Rank1 
  export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 
  export GLOO_SOCKET_IFNAME=eno8303 
  export NCCL_SOCKET_IFNAME=eno8303 
  export RCCL_SOCKET_IFNAME=eno8303 

  cd /sgl-workspace/sglang 

  python3 -m sglang.launch_server \ 
    --model-path /mnt/models/Qwen/Qwen3.8-2.4T-A95B-FP8 \ 
    --tp-size 8 \ 
    --pp-size 2 \ 
    --nnodes 2 \ 
    --node-rank 1 \ 
    --dist-init-addr 10.194.129.192:25000 \ 
    --dist-timeout 3600 \ 
    --attention-backend aiter \ 
    --chunked-prefill-size 8192 \ 
    --watchdog-timeout 1200 \ 
    --mem-fraction-static 1.0 \ 
    --host 0.0.0.0 \ 
    --port 9001 \ 
    --disable-radix-cache \ 
    --disable-custom-all-reduce \ 
    --max-running-requests 64 \ 
    --page-size 16 \ 
    --kv-cache-dtype fp8_e4m3 \ 
    --cuda-graph-max-bs-decode 64 \ 
    --reasoning-parser qwen3 \ 
    --tool-call-parser qwen3_coder

		curl http://localhost:8000/v1/completions \ 
-H "Content-Type: application/json" \ 
-d '{ 
"model": "/data/models/Qwen3.8-2.4T-A95B- Quark-MXFP4/",
"prompt": "Question: Janet'\''s ducks lay 16 eggs per day. She eats 3 for breakfast and bakes 4 into muffins. How many eggs does she sell?\nAnswer:", 
"max_tokens": 64, 
"temperature": 0

Step 4 (Optional) — Accuracy Evaluation (GSM8K)

MXFP4 on MI355X (TP8, single node)

SGLang built-in benchmark:

		  python3 -m sglang.test.run_eval \ 
    --port 9001 \ 
    --model /mnt/models/Qwen3.8-2.4T-A95B-Quark-MXFP4 \
    --eval-name gsm8k \ 
    --num-examples 1319 \ 
    --num-threads 512 \ 
    --max-tokens 2048 \ 
    --chat-template-kwargs '{"enable_thinking": false}'

Result (validated):

Metric	Value
Accuracy	97.4886%
Invalid	0.1%
Runtime	168.5 s (1319 questions)
Output throughput	1060.45 tok/s

lm-eval command:

		  python3 -m lm_eval \ 
    --model local-completions \ 
    --model_args '{"base_url":"http://localhost:9001/v1/completions","model":"/mnt/models/Qwen/Qwen3.8-2.4T-A95B-FP8- 
 MXFP4","num_concurrent":256,"max_retries":10,"max_gen_toks":2048,"timeout":3600}' \ 
    --tasks gsm8k \ 
    --batch_size auto \ 
    --num_fewshot 5 \ 
    --trust_remote_code \ 
    --log_samples \ 
    --output_path <OUTPUT_DIR>

Metric	Value
Strict exact match	95.30%
Flexible exact match	95.30%
Config	TP8, MXFP4 checkpoint

FP8 on 2× MI300X (TP8 PP2)

SGLang built-in benchmark:

		python3 -m sglang.test.run_eval \ 
    --port 9001 \ 
    --model /mnt/models/Qwen/Qwen3.8-2.4T-A95B-FP8 \ 
    --eval-name gsm8k \ 
    --num-examples 1319 \ 
    --num-threads 512 \ 
    --max-tokens 2048 \ 
    --chat-template-kwargs '{"enable_thinking": false}'

Metric	Value
Accuracy	97.4886%
Invalid	0.0%
Runtime	1584.435s (1319 questions)
Output throughput	112.693 tok/s

How to Run Qwen 3.8 MXFP4 with ATOM on AMD Instinct GPUs

Prerequisites

AMD Instinct MI355X with ROCm configured
Model checkpoints:
MXFP4: /data/models/Qwen/Qwen3.8-Max-Dummy-MXFP4/ (Quark-produced)
ROCm-optimized SGLang container (match image to GPU)

Step 1: Launch Docker Container

		docker pull rocm/atom-dev:nightly_qwen3.8_day0 
docker run -it \  
--device /dev/dri --device /dev/kfd \  
--network host --ipc host \  
--group-add video \  
--security-opt seccomp=unconfined \  
-v /data/models:/data/models \  
--cap-add=SYS_PTRACE \  
--shm-size=16G --ulimit memlock=-1 --ulimit stack=67108864 \  
rocm/atom-dev:nightly_qwen3.8_day0

Step 2: Start the ATOM Server

		bash /app/ATOM/scripts/start_atom_server.sh /data/models/Qwen3.8-2.4T-A95B- Quark-MXFP4/ 8 8001 --no-enable_prefix_caching

Step 3 (Optional) — Accuracy Evaluation (GSM8K)

		lm_eval --model local-completions \ 
    --model_args '{"base_url":"http://localhost:8001/v1/completions","model":"/data/models/Qwen3.8-2.4T-A95B- Quark-MXFP4 Qwen3.8-2.4T-A95B- Quark-MXFP4 ","num_concurrent":256,"max_retries":10,"max_gen_toks":2048,"max_length":8192,"tokenized_requests":false,"timeout":3600}' \ 
    --tasks gsm8k \ 
    --num_fewshot 5 \ 
    --trust_remote_code

Metric	Value
Strict exact match	94.92% ± 0.006%
Flexible exact match	95.00% ± 0.006%
Config	TP8, MXFP4 checkpoint

Quark: MXFP4 Quantization

AMD Quark provides broad quantization support across the Qwen model family, spanning dense and MoE models, text and multimodal architectures, and deployment formats including INT4, FP8 and MXFP4. Building on this support, Quark was used to enable the MXFP4 checkpoint for Qwen3.8-Max-FP8.

Starting from the FP8 checkpoint, Quark selectively quantizes the MoE routed experts in both the main model and MTP layers to MXFP4, while keeping the remaining layers in BF16. Quark's file-to-file workflow processes the model checkpoint shard by shard, enabling scalable conversion from the pre-quantized FP8 checkpoint to MXFP4 without loading the full model into memory. Quark's MXFP4 rounding method further helps minimize quantization error, enabling high-fidelity conversion to the lower-precision format.

Quark also supports flexible mixed-precision configurations, creating room for further performance optimization as new kernels and inference capabilities become available on AMD GPUs.

The MXFP4 checkpoint can be reproduced with the following Quark quantization flow:

		from quark.torch import LLMTemplate, ModelQuantizer  

qwen3_5_moe_text_template = LLMTemplate( 
    model_type="qwen3_5_moe_text", 
    kv_layers_name=["*k_proj", "*v_proj"], 
    q_layer_name="*q_proj" 
) 
LLMTemplate.register_template(qwen3_5_moe_text_template) 

# === NOTE: Change the following source and output dir as needed === 
src_ckpt_path = " Qwen/Qwen3.8-2.4T-A95B-FP8" 
output_dir = "output/Qwen3.8-2.4T-MXFP4" 

quant_scheme = "mxfp4" 
exclude_layers = [ 
    "lm_head", 
    "*embed_tokens", 
    "*self_attn*", 
    "*linear_attn*", 
    "*mlp.gate", 
    "*shared_expert*", 
    "mtp.fc",              # Exclude BF16 layers in MTP 
    "*mtp*self_attn*",     # Exclude BF16 layers in MTP 
    "*mtp*mlp.gate",       # Exclude BF16 layers in MTP 
    "*mtp*shared_expert*", # Exclude BF16 layers in MTP 
] 
template = LLMTemplate.get("qwen3_5_moe_text") 

quant_config = template.get_config(scheme=quant_scheme, exclude_layers=exclude_layers) 

# Quantize with File2File mode to avoid loading full model 
quantizer = ModelQuantizer(quant_config) 
quantizer.direct_quantize_checkpoint( 
    pretrained_model_path=src_ckpt_path, 
    save_path=output_dir, 
)

Conclusion

With the release of Qwen 3.8, Alibaba continues to push the boundaries of open-weight models. By providing Day 0 support on AMD Instinct™ GPUs with both SGLang and vLLM, we ensure that developers have the compute power and optimized software stack needed to run these massive, high-context models at production scale.

Additional Resources

Join AMD AI Developer Program to access AMD developer cloud credits, expert support, exclusive training, and community.
Visit the ⁠ROCm AI Developer Hub for additional tutorials, open-source projects, blogs, and other resources for AI development on AMD GPUs.
Explore AMD ROCm Software.
Learn more about AMD Instinct GPUs.
Download the model and code
- Hugging Face: Qwen/Qwen3.8-2.4T-A95B · Hugging Face
- Modelscope: https://www.modelscope.ai/models/Qwen/Qwen3.8-2.4T