469 tok/s/user on Million-Token Agent Sessions: TileRT Takes the Top Spot on AgentX with AMD Instinct™ MI355X GPUs

In the latest InferenceX AgentX rankings, TileRT powered GLM-5.3 to 469 tok/s of single-user generation throughput on 8× AMD Instinct™ MI355X GPUs, taking the top spot on the AgentX leaderboard and leading the second-place NVIDIA GB300 NVL72 system by more than 100 tok/s.

AgentX: 469 tok/s on Real-World, Long-Horizon Agent Workloads

What Does AgentX Measure?

InferenceX (formerly InferenceMAX), led by SemiAnalysis, is an open, vendor-neutral, reproducible inference benchmark. AgentX focuses on inference performance under workloads that reflect how coding agents actually operate.

This is very different from conventional benchmarks with fixed prompt lengths.

AgentX workloads are derived from real coding-agent trajectories. To make the benchmark reproducible, the original content is removed and reconstructed using deterministic synthetic tokens, while preserving the structure of the original sessions: multi-turn interactions, continuously growing shared prefixes, pauses for tool calls, and dependencies between the main agent and its sub-agents.

In other words, AgentX measures performance across complete agent sessions. As a task progresses and the context keeps growing, the ability to sustain high generation speed becomes increasingly important.

469 tok/s: #1 in Single-User Performance on AgentX

On the AgentX workload, TileRT running GLM-5.3 on 8x AMD Instinct MI355X GPUs achieves: 469 tok/s single-user generation throughput

For long-horizon agent tasks, TileRT chooses not to trade numerical precision for speed. With the original FP8 weight precision and BF16 KV cache, TileRT still delivers the fastest single-user result on the AgentX leaderboard, more than 100 tok/s ahead of the second-place NVIDIA GB300 NVL72 submission running FP4.

From 1K to 1M Context: 1,000× Longer, ~2/3 of the Speed Retained

Beyond the final AgentX result of 469 tok/s, there is another metric we care deeply about: how much performance is retained as the context grows?

AgentX sessions can grow to millions of tokens. As context length increases, generating each new token requires accessing an increasingly large KV cache, making memory access progressively more expensive during decode. Strong performance at short context therefore does not necessarily translate into the same level of interactivity over a long-running agent session.

As Figure 2 shows, when the input context grows from 1K to 1M tokens - a 1,000x increase, single-user generation speed decreases from 648 tok/s to 425 tok/s. Even at a one-million-token context, TileRT retains roughly two-thirds of its short-context performance.

This matters for agent workloads. Real agent sessions continuously accumulate context, and as tasks run longer, long-context decode performance increasingly determines the interactivity experienced by the user.

Figure 2 comes from our development testing on 8x MI350X, while the AgentX leaderboard result was measured on 8x MI355X GPUs. The MI350X and MI355X GPUs are both based on CDNA 4, but MI350X GPU runs at lower clock frequencies. The results in Figure 2 therefore provide a relatively conservative reference compared with the MI355X configuration used for AgentX.

Test configuration: The AgentX leaderboard result uses 8× MI355X GPUs, while the long-context decode measurements in Figure 2 use 8x MI350X GPUs. Both configurations use TP8, the publicly released GLM-5.3 weights, FP8, and speculative decoding with MTP K=3. See our InferenceX submission for the full leaderboard configuration.

How TileRT Gets There

TileRT was designed for low latency from day one.

When people talk about scaling laws, they usually talk about model parameters and training data. For agents, we believe there is another important dimension: speed. Faster inference means an agent can complete more rounds of reasoning, tool use, and environment interaction in the same amount of time. In that sense, speed itself is a dimension of scaling intelligence.

In latency-first workloads, as hardware compute and memory bandwidth continue to increase, the time actually spent on computation during a decode forward pass has become increasingly small. Other overheads - kernel launches, cross-kernel synchronization, and round trips to global memory, start to enter the critical path.

These costs are often hidden behind computation in throughput-oriented workloads. At extremely low latency, they become increasingly visible. We call this the execution gap.

TileRT's core approach is to statically unfold the entire model at compile time into a persistent Engine Kernel on the GPU. Compute, communication, and asynchronous I/O all progress continuously within this engine. There is no kernel-by-kernel launch sequence, and optimization is no longer constrained by kernel boundaries: the scope expands from individual kernels to the entire forward pass.

On the MI355X GPU, we focus on three areas to further close the execution gap.

1.  From Global Barriers to Streaming Synchronization

Once execution moves into a persistent kernel, kernel boundaries disappear, and so does the global synchronization they implicitly provide.

A conventional approach is to use a global barrier: wait until every block has finished computing and writing its results before allowing the next stage to proceed.

TileRT avoids this global alignment. As soon as the upstream stage produces part of its output, that data can be handed off to the downstream stage. The consumer can wait for and process data incrementally as it becomes ready.

Through a streaming producer-consumer handshake, stages that would otherwise execute sequentially can continuously overlap, reducing the latency introduced by global synchronization.

This optimization is particularly effective on AMD architectures.

2. More Aggressive Prefetching with a Persistent Engine

Prefetching is one of the key techniques for hiding memory-access latency. NVIDIA's PDL allows a subsequent kernel to launch before the preceding kernel has fully completed, enabling data preparation to begin earlier.

TileRT's persistent Engine goes further by eliminating the kernel boundary itself. The entire forward pass executes continuously inside the Engine Kernel, allowing data required by later stages to start moving earlier without waiting for a "next kernel" to be scheduled.

This gives TileRT control over when to prefetch, how far ahead to prefetch, and what data to move across the entire forward pass, enabling more aggressive overlap between data movement and computation than PDL.

3. Fusing Communication with Compute

With TP8, all-reduce is also on the critical path of decode latency.

Conventional implementations typically insert collective communication as a separate operation between compute kernels, using algorithms such as ring all-reduce or reduce-scatter + all-gather. These approaches are effective at utilizing link bandwidth, but require multiple rounds of communication and synchronization.

For the small-message, latency-sensitive communication patterns found in decode, however, the number of communication rounds can matter more than peak bandwidth.

TileRT instead uses a one-shot push followed by an ordered local reduction: each rank writes its data directly into symmetric buffers on every peer, then performs the reduction locally in a fixed order. This removes multiple rounds of data transfer as well as a separate synchronization stage.

More importantly, communication in TileRT is not implemented as a standalone kernel. It is integrated directly into the execution pipeline, allowing compute and communication to continuously overlap and further shortening the critical path of each forward pass.

Why This Approach Goes Further on AMD

Latency optimization at this level requires moving beyond many existing software abstractions and working directly with the hardware. How far that can go depends heavily on how much control the architecture exposes to software.

The AMD CDNA™ 4 architecture behind the MI355X GPU provides several capabilities that align particularly well with TileRT's execution model.

• Register capacity: Prefetching is fundamentally a space-for-time tradeoff: the earlier data is moved, the more in-flight state must be retained, and the greater the register pressure. CDNA provides a larger register file per compute unit and a higher number of addressable registers per thread. For TileRT's persistent Engine, this means more in-flight data can remain resident at the same occupancy, enabling deeper prefetching and more execution stages to overlap.
• Partitioned caches and fine-grained memory control: The CDNA architecture divides the GPU into multiple compute partitions, each with its own local cache, while memory instructions allow software to control visibility scope and caching behavior. This enables cross-block data exchange within a partition using lightweight mechanisms rather than frequently relying on device-wide synchronization. It maps naturally onto TileRT's producer-consumer execution model: the runtime can explicitly decide which blocks are grouped together and which blocks exchange data. Hardware partitioning becomes a locality hierarchy that software can actively exploit.
• ISA-level control: Once optimization reaches the level of persistent execution, details such as instruction scheduling, memory scope, and prefetch distance can directly affect end-to-end latency. These decisions are difficult to leave entirely to the compiler.

The CDNA architecture allows us to control the final ISA directly through inline assembly. Combined with an open and disassemblable toolchain, this lets us inspect exactly what the compiler generates and make targeted changes on performance-critical paths. This gives TileRT substantial room for software optimization: the hardware provides the capabilities; software decides how to schedule and compose them.

We increasingly believe that exposing more architectural capabilities to software is a powerful direction for rapidly evolving AI workloads. As runtimes and compilers gain more control over hardware, more decisions that were once fixed in hardware can instead be made and iterated on in software.

TileRT's results on the MI355X GPUs are one validation of that approach.

Getting Started and What's Next

TileRT is already available on PyPI:

Next, we are moving forward along several directions:

• Official the AMD ROCm™ Software and MI355X support: We will release ROCm support on GitHub, with MI355X becoming one of TileRT's officially supported platforms.
• More batch sizes: TileRT currently focuses on pushing batch-1 interactivity to the limit. Batch 2, 4, and larger batch sizes are also on our roadmap, and we will expand support based on real workloads and user demand.

TileRT and AMD will also continue to collaborate on high-performance inference.

As agent workloads become increasingly complex, we believe inference speed itself will become an increasingly important dimension of scaling.

We will keep pushing further: lower latency, longer context, and better utilization of the capabilities exposed by the hardware.

Reference

• Website: https://www.tilert.ai/
•  GitHub: https://github.com/tile-ai/TileRT