FastFlow now runs on AMD GPUs: two flows, one umbrella

Nov 19, 2026

wavy dotted lines

Highlights

  • First HIP EP integration: FastFlow v1.0.13 brings HIP EP v0.5.3 into FLM through the new oga-hipep backend.
  • Two flows, one umbrella: NPU and GPU execution now sit behind the same CLI, local API, and conversation format.
  • Faster first token: At 16K context, the latest HIP EP build reaches the first token about 43% sooner than llama.cpp (Vulkan), with decode throughput about 7% higher.
  • Three commands to start: flm pull, flm run, and flm serve are all it takes

The iGPU flow powered by HIP EP in FastFlow v1.0.13 is an early access release, now available for wider adoption and feedback. Features, performance, and supported configurations may change as we learn from your use. Please share your feedback through the FastFlow repository.

Today, FastFlow takes a big step forward. With v1.0.13, released on November 13, 2026, it brings two inference flows under one umbrella: the NPU path developers already know, and a new high-performance GPU path powered by HIP EP. One single local API and CLI. More ways to run AI on AMD hardware.

FastFlow is a lightweight runtime of about 17 MB for AMD Ryzen™ AI systems. It offers a single-command CLI and an OpenAI-compatible server, so existing tools and applications connect without rewriting. On Ryzen AI NPUs, it demonstrates strong throughput and power efficiency. Now the GPU joins the party.

Meet HIP EP

HIP EP is an open-source ONNX Runtime execution provider built for performance on AMD GPUs. It compiles an ONNX graph through an MLIR pipeline and runs it with hipBLASLt and custom HIP kernels. As highlighted in the HIP EP site , it supports more than 20 model architectures and context lengths up to 128K tokens on the Ryzen AI Max, Ryzen AI 400, and Ryzen AI 300 series. FastFlow now brings the GPU performance to everyday developers through the oga-hipep backend, built on ONNX Runtime GenAI. The code and releases are in the HIP EP repository.

HIP EP's benchmark tracking on a Ryzen™ AI Max (Strix Halo) system shows the latest build ahead of llama.cpp (Vulkan) at 16K context. Time to first token (TTFT) is about 43% lower and decode throughput (TPS) is about 7% higher. The table shows results normalized to llama.cpp (Vulkan) as 1.00.

Configuration

TTFT at 16K (lower is better)

Decode TPS at 16K (higher is better)

llama.cpp (Vulkan) (baseline)

1.00

1.00

HIP EP build (11/10/2026)

0.57

1.07

For absolute numbers, the HIP EP site also reports decode throughput for selected models on a Ryzen AI Max (Strix Halo) system. These results use HIP EP v0.5.3 with a 2,048-token prompt. They are HIP EP results rather than end-to-end FastFlow results. See the HIP EP benchmarks page for the full data.

Model

Input type

Decode (tokens/s)

Qwen2.5 14B Instruct

Text

26.10

Qwen3.5 9B

Text and images

38.94

Qwen3.5 35B-A3B

Text and images

67.04

Gemma 3 4B IT

Text

74.98

GPT-OSS 20B

Text

85.96

One software, two hardware flows

FastFlow keeps language-model behavior in shared code. Chat templates, tokenization, sampling, and the generation loop sit above the backend boundary. Whether a model runs on the NPU or the GPU, developers get the same CLI, the same local API, and the same conversation format. The model tag simply selects the backend.

Figure 1: FastFlow shares request and generation services across the device paths. Each backend owns its execution and device state. Model-family hooks still handle differences in prompts and media.

Try it today

On a supported Windows 11 system with FLM v1.0.13 or later, three commands take you from download to a running GPU-backed model. The OGA catalog covers 13 OGA models from Qwen 2.5, 3.5, 3.6, and 3.8, Gemma 3 and 4, and GPT-OSS. The Qwen 3.5, 3.6, and 3.8 models also accept images.

		flm list
flm pull qwen3.5-oga-hipep:9b
flm run qwen3.5-oga-hipep:9b
flm serve qwen3.5-oga-hipep:9b
	

In the flm list output, models with -oga-hipep: in the name, such as qwen3.5-oga-hipep:9b, run on the HIP EP flow, pull downloads the model, run opens a chat in the terminal, and serve exposes the model through the OpenAI-compatible local API. The first OGA model load installs the HIP EP runtime if it is not already present. The current path still requires a supported NPU on the host, even when the model runs on the GPU. The Windows installation guide and the CLI documentation cover setup, runtime options, and model notes.

Summary

FastFlow v1.0.13 is a milestone: for the first time, the software runs on both the NPU and the GPU under one umbrella. The new oga-hipep backend, powered by HIP EP v0.5.3, brings high-performance GPU execution to the CLI and local API that developers already use. Shared code keeps tokenization, sampling, and generation consistent across both flows. This is a great step for the product and for everyone building local AI on AMD hardware. Pull a model, run it, and see it for yourself.

This release would not have been possible without the close collaboration and support of the FastFlow team, including Tao Wei. Their partnership made the new backend a success.

Resources

Related Blogs