The Resilience as an Architectural Structure

Jul 08, 2026

Long-context, multi-user LLM serving is increasingly bottlenecked by KV-cache growth, not just raw compute. This blog post examines the Topological Ghost Protocol (TGP), developed by ARUY TECHNOLOGIES SAS (https://aruy.net), an experimental architecture that treats the KV cache as segment-local and recyclable rather than session-long, preserving continuity through a compact state payload instead of retaining the full attention history. In the highest-pressure test on an AMD Instinct™ MI300X, with 256 concurrent users and 250 output tokens, TGP sustained approximately 431 tokens/sec at a 100% success rate, while the tested vLLM baseline fell to approximately 42.7 tokens/sec at a ~12% success rate — roughly a 10× throughput gap in that specific high-pressure regime. That advantage is not universal: at moderate load, vLLM’s mature batching and scheduling can still win, and TGP’s benefit emerges only as concurrency, output length, and model size climb together and active memory becomes the limiting factor.

We also walk through the architecture, hardware setup, methodology, local and MI300X benchmark results, and close with where TGP does and does not fit in a production stack, for both developers building serving layers and decision-makers evaluating infrastructure trade-offs.

1. Current State of the Art

In modern systems, the deployment of Large Language Models (LLMs) and even Small Language Models (SLMs) presents a marked architectural duality. On the one hand, solutions such as vLLM are highly efficient in both single-user and multi-user environments, and their deployment is particularly meaningful in contexts where hardware resources are abundant. Typically, vLLM reserves a considerable percentage of the GPU to optimize its architecture, between 85% and 90%. 

At the other extreme, architectures such as StreamingLLM solve a different problem: maintaining long sessions without VRAM collapsing or the model becoming unstable.

In both cases, there is a cost: either the KV cache accumulates, as in vLLM, or information is lost once it is no longer part of the context window or of the tokens the model uses to avoid hallucinations, as in StreamingLLM.

2. What TGP Actually Changes

TGP does not primarily rely on a faster model or prompt compression. It changes the relationship between inference, active memory, and contextual continuity.

In conventional autoregressive serving, generation continuity tends to depend on preserving accumulated internal state, especially the KV cache. TGP starts from a different premise: semantic continuity can be preserved without keeping the full computational history resident throughout the session.

The mechanism is simple at a high level:

  1. The task is split into semantically delimited segments.
  2. Each segment generates its own local, transient attention state.
  3. Once the segment finishes, its KV cache becomes eligible for release or recycling.
  4. Continuity is carried forward through a compact state payload: contextual indexes, structural metadata, and, where useful, latent vectors or condensed state instructions.

Not everything is represented in the same way. Facts, numbers, names, and syntactically rigid content are preserved through references, indexes, or metadata because they are too fragile to compress lossily. More diffuse attributes, such as tone, style, narrative identity, or generation direction, can be represented as compact control vectors: small numerical steering signals rather than text summaries.

For example, the sentence “an AI named Aster-7 awakens on Sol 42 inside Habitation Module Eos, while the habitat remains at 21.4°C and 68.2 kPa” splits into two kinds of state. The exact facts — Sol 42, Habitation Module Eos, 21.4°C, and 68.2 kPa — are preserved as structured references. The calm, melancholic science-fiction tone and Aster-7’s emerging protective identity can be preserved as compact control vectors. The next segment does not need the full previous paragraph or its KV cache; it needs the exact facts, the current semantic position, and a steering signal for how to continue coherently.

The net effect is that TGP reduces the need to keep the full computational history resident in fast memory. Instead, it retains enough compact structure to rehydrate context at each new segment. This makes the architecture relevant to long sessions, multi-user serving, and environments where available VRAM per instance is the limiting factor.

Image Zoom
Figure 1. TGP Runtime Continuity Loop
Figure 1. TGP Runtime Continuity Loop
Image Zoom
Figure 2. Segment-level TGP memory trace showing repeated rehydrate and flush cycles.
Figure 2. Segment-level TGP memory trace showing repeated rehydrate and flush cycles.

 

3. The Memory-Residency Model 

The flow diagram describes the logical operation of TGP. The effect becomes clearer when viewed as a memory-residency problem.

In a conventional autoregressive inference server, each active sequence carries a growing KV-cache. At a high level, the active KV memory of one sequence scales with the number of resident tokens:

M_KV ≈ 2 × L × H_KV × D_head × T_active × bytes

where L is the number of transformer layers, H_KV is the number of key/value heads, D_head is the head dimension, and T_active is the number of tokens that must remain resident for attention. Under multi-user serving, the pressure is multiplied by the number of active sessions:

M_active ≈ M_weights + U_active × 
M_KV(T_active) + M_runtime

This is the regime in which long outputs and high concurrency interact poorly: even if raw compute is available, the scheduler eventually has to keep too much active attention state resident at the same time.

TGP changes the residency rule. The KV-cache is treated as local to a semantic segment, not as the persistent memory of the whole session. After a segment is decoded, TGP extracts a compact continuity payload, updates the session state, and releases the KV-cache associated with that segment. The next segment is then rehydrated from the compact state rather than from the full previous attention history.

In simplified form:

		or each request:
  plan = topological_segment(prompt)

  for each segment in plan:
    cdm = estimate_density(segment)
    worker = assign_worker(cdm, available_resources)
    output,local_kv = decode(segment, compact_state)
    delta_state = encode_transition(
           output,
           cdm,
           factual_indexes,
           structural_metadata,
           latent_vectors_or_state_prompt
    )
    compact_state = merge(compact_state, delta_state)
    flush(local_kv)

The practical difference is that active memory is now bounded primarily by the maximum live segment, not by the full generated session:

M_active_TGP ≈ M_weights

              + U_active × M_KV(S_segment)

              + U_sessions × M_payload

              + M_runtime

where S_segment is the current segment length, and M_payload is the compact state used to preserve continuity. The payload may include contextual indexes for rigid facts, structural metadata for planning and assembly, and latent vectors or condensed state instructions for style, identity, and generation direction.

TGP introduces additional work: segmentation, state extraction, payload merging, and rehydration. It can also increase memory movement because “the” state must be explicitly materialized and reinjected between segments. However, the trade-off differs from a standard long-context serving loop. TGP performs additional control work to avoid unbounded growth of active KV-cache residency.

This is why the architecture becomes more relevant under high concurrency and longer outputs. At moderate loads, a mature engine such as vLLM may remain faster, because its continuous batching and scheduling are highly optimized. But when concurrency, output length, model size, and VRAM pressure increase together, the limiting factor is no longer only tokens per second. It becomes whether the system can keep admitting and completing work without crossing a memory-driven degradation threshold.

As we will observe later in this blog post, for the Qwen 72B stress test, this distinction appeared in the high-pressure regime. With 256 concurrent users and 250 output tokens, TGP completed the workload at approximately 431 tokens per second with a 100% success rate, with minimal model degradation, while vLLM degraded to approximately 42.7 tokens per second with a success rate close to 12% in the same tested scenario. The relevant point is not that TGP is universally faster, but that its segmented memory-residency model preserved completion stability after the conventional serving path crossed an operational threshold.

4. Current Status of TGP and Effective Deployment

The evolution of TGP has taken the architecture from single-user experimental runs to multi-user deployments on the AMD Instinct MI300X GPUs. In its current experimental stage, TGP has been tested with concurrent loads of 128, 192, and 256 users, including stress tests with Qwen 2.5 72B Instruct under segmented execution.

These tests do not yet represent a fully optimized production server. They should be read as architectural stress tests with a protocol such as TGP that is not yet fully optimized, which, ironically, hints at the technology’s potential.

The following sections evaluate whether the runtime behavior described above appears in practice when TGP is exposed to larger models, longer outputs, and higher concurrency. 

5. The Hardware: Why AMD MI300X?

The AMD Instinct MI300X GPU was selected for this evaluation because its large HBM3 capacity and high memory bandwidth made it suitable for testing TGP under a demanding multi-model, multi-user workload. The experimental setup required hosting the 24B main model, the 3B auxiliary model, the 72B quantized reserve model, runtime buffers, active key-value/state buffers, and the additional overhead associated with compact continuity state across up to 256 concurrent sessions.

This memory profile made the MI300X GPU particularly relevant for the experiment. TGP does not eliminate the cost of attention, nor does it make each token intrinsically cheaper. Instead, it changes the lifetime of the active inference state by treating the KV cache as segment-local and recyclable. That design introduces additional memory movement at segment boundaries, including segment closure, compact-state extraction, rehydration, and KV-cache recycling.

The implementation ran on the AMD ROCm™ Software 7.2- host and used the upstream FlashAttention-2 ROCm backend with AMD Composable Kernel (CK), as provided by the official FlashAttention repository (https://github.com/dao-ailab/flash-attention). In this setting, the MI300X GPU memory system was critical: its HBM3 capacity allowed the full experimental configuration to remain resident, while its bandwidth made it practical to evaluate TGP’s segmented execution pattern under high concurrency. The goal was therefore not only to measure raw throughput. Instead, we test whether a segmented memory-residency model could remain operationally stable when model size, output length, and user concurrency jointly increased pressure on GPU memory.

Component

Detail

GPU

AMD Instinct MI300X — up to 192 GB HBM3 — 5.3 TB/s peak theoretical bandwidth — 750 W maximum TBP

ROCm

ROCm ROCm 7.2 host ; ROCm Docker image/container used for execution

Primary model

Mistral Small 3 / 24B Instruct 2501 — BF16 — ~48 GB weights, excluding KV/cache/runtime overhead

Small model

Qwen2.5-3B-Instruct — BF16 — ~6 GB measured/reported memory at short context; higher with longer context

Large model

Qwen2.5-72B-Instruct — 4-bit quantized — theoretical weights ~36 GB; practical footprint commonly closer to ~40–46+ GB depending quantization, context length, KV/cache, and runtime

FlashAttention

FlashAttention-2 ROCm backend via AMD Composable Kernel / CK

Comparison engine

 vLLM — PagedAttention — same model and GPU, with quantization/context settings explicitly matched

Table 1. MI300X GPU experimental hardware and software configuration.

The experiments reported here compare a TGP prototype against the specific vLLM implementation and configuration used as a mature serving baseline. We intend to highlight the properties of ARUY’s TGP technology in light of the aforementioned restrictions. 

vLLM is an optimized inference engine, while TGP is an architectural technique for changing the lifetime of active inference state, especially the KV cache, under long-context and multi-user pressure.

In a production setting, these approaches are not necessarily mutually exclusive. A TGP-enabled serving engine could incorporate segmented execution, compact continuity state, and segment-local KV-cache recycling into an optimized runtime derived from, or conceptually similar to, vLLM. In that sense, the comparison should be read less as “TGP versus vLLM as a framework” and more as “TGP-style memory residency against the current continuous-serving baseline evaluated in this experiment.”

The practical question is therefore not whether TGP replaces mature inference engines, but whether TGP techniques can extend them. With sufficient engineering resources, a production implementation could resemble a “vLLM++” system: an inference engine retaining the batching, scheduling, and deployment maturity of existing runtimes, while adding TGP-style segmentation and continuity management for regimes where concurrency, model size, output length, and active memory pressure jointly stress the hardware.

6. Local Benchmarks 

Definition of segment in these tests. In the context of the local benchmarks, a segment is one bounded execution unit of TGP: the model decodes a limited portion of the answer, then the runtime closes that unit, extracts the continuity state required for the next unit, and releases the segment-local KV cache. A segment should therefore not be understood as a simple text chunk cut out after generation, but as a runtime boundary at which TGP can recycle active memory.

In the local tests below, segments were configured with a target length of approximately 150 generated tokens. This token target was used to make the comparison measurable and repeatable. Architecturally, however, a TGP segment is intended to be semantically delimited: in production, the boundary may correspond to a paragraph, subtask, reasoning step, scene, section, or other logical unit, rather than a fixed token count.

6.1 TGP vs StreamingLLM

In one test, the task consisted of having a Qwen 2.5 model with 1.5B parameters create a story based on the following prompt: “Write a science fiction story about an artificial intelligence that develops its own emotions while helping to colonize Mars. Begin with the AI’s first awakening.” In both cases, the systems had to meet the maximum target of 4,500 tokens. While StreamingLLM did so continuously, TGP did so in increments of 150 tokens. At the end, the values were averaged for StreamingLLM, while for TGP, the metrics were obtained per segment, yielding the results presented below:

Metric

TGP

StreamingLLM

Inference speed (TPS)

53.85

37.77

Total time (Wall Clock)

141.95

121.23

Energy per 1k tokens (Infer)

2,641.31

3,190.34

Total energy consumed

13,243.88

14,451.23

Table 2. Local Qwen 2.5 1.5B comparison between TGP and StreamingLLM.

6.2 ARUY’s TGP vs vLLM

In a second local test, TGP was compared against vLLM using Qwen 2.5 1.5B Instruct, 30 segments, and a target of approximately 150 tokens per segment. This test served as a local evaluation of efficiency, memory pressure, and generation capacity under a controlled long-running task.

In that regime, TGP showed advantages in the core metrics of the test, including higher effective throughput, lower average latency per segment, lower peak VRAM, lower energy per token, and a higher number of completed tokens. The most significant difference does not lie in a single metric, but in the combination of speed, memory usage, and efficiency under the same extensive generation prompt.

Metric

TGP

vLLM

Throughput (TPS)

77.26

68.46

Average latency (s)

1.77

1.87

Peak VRAM (MB)

4,810.69

7,952.66

Efficiency (J/token)

1.59

1.84

Total tokens

4,495.00

3,936.00

Table 3. Local Qwen 2.5 1.5B comparison between TGP and vLLM

Image Zoom
Figure 3. Qwen 2.5 1.5B local benchmark: GPU VRAM by segment.
Figure 3. Qwen 2.5 1.5B local benchmark: GPU VRAM by segment.

7. Scaling Up: Execution on AMD Instinct MI300X 

The MI300X experiment was designed to test whether the memory-residency behavior observed in local runs would remain visible under a substantially heavier serving regime: a 72B-class model, high user concurrency, and longer generated outputs. In this setting, the relevant workload is not a single long prompt, but the aggregate pressure created by many live sessions competing for the same HBM capacity, runtime buffers, and scheduler slots.

For this reason, the experiment should be read as a high-pressure inference-serving test rather than a pure tokens-per-second benchmark. The tested vLLM baseline represents a mature continuous-serving implementation, while the TGP prototype represents a segmented execution strategy in which the most memory-sensitive runtime object, the KV cache, is not treated as a session-long structure.

In the TGP run, each user request is represented as a sequence of segment-local decoding cycles. During a segment, the model still performs ordinary autoregressive decoding and uses attention normally. The architectural difference appears at the segment boundary: once the segment is completed, the runtime closes that decoding unit, preserves the compact continuity state required for the next unit, and makes the segment-local KV cache eligible for release. The next segment is then scheduled as a new decode unit conditioned by the preserved state, rather than by retaining the full attention history of the previous segment.

At a high level, this changes the memory-residency target from:

model weights

+ runtime buffers

+ KV state for many growing active sessions

to:

model weights

+ runtime buffers

+ KV state for the currently live segments

+ compact per-session continuity state

The expected advantage of TGP appears only when the workload approaches a memory-driven scheduling threshold. As concurrency and output length increase together, the system is forced to keep more active state resident at the same time. Under those conditions, reducing the lifetime of KV-cache blocks becomes more important than minimizing overhead in the simpler case. The relevant question is therefore not whether TGP is always faster, but whether it can keep completing requests after active memory pressure becomes the limiting variable.

The stress test in the next section evaluates exactly that operating region. It compares the TGP prototype and the tested vLLM configuration under increasing concurrency and output length, using the same GPU environment and matched model settings. The purpose is to identify whether segmented memory residency produces a different degradation profile when the workload moves from a throughput-limited regime into a memory-pressure regime.

8. Stress Test with Qwen 72B: Resilience Under High Concurrency

The stress test was designed to evaluate whether TGP’s segmented memory-residency model remains stable when concurrency, output length, and model size increase together. The benchmark used concurrent loads of 128, 192, and 256 users, with output lengths of 192 and 250 generated tokens. For TGP, the workload was executed through segment-local decoding cycles. For the tested vLLM baseline, generation was continuous under the matched model and GPU configuration.

The purpose of this test was not to show that TGP is universally faster than an optimized serving engine. At moderate loads, the tested vLLM baseline remained competitive and, in some cases, faster. The relevant question was different: when the workload approaches a memory-pressure regime, which system continues completing requests?

 

Image Zoom
Figure 4. Conceptual memory residency over time.
Figure 4. Conceptual memory residency over time.

The conceptual memory-residency figure illustrates the runtime difference measured by the stress test. In a conventional serving loop, active KV-cache residency tends to grow as each session accumulates more generated tokens. In TGP, the KV cache is treated as local to the current segment: memory rises during segment execution, is released at the segment boundary, and then rises again for the next segment. The compact continuity state remains comparatively small and stable across the session.

This figure is not a direct benchmark trace; it is a conceptual representation of the memory-lifetime rule being tested. The benchmark below evaluates whether that rule produces a different degradation profile when many users, a large model, and longer outputs compete for the same GPU memory budget.

Load / tokens

Engine

TPS

Peak VRAM approx.

GPU W avg

J/token approx.

P50 seg. latency

P95 seg. latency

Success rate

128 / 192

TGP

338.4

161.0 GB

715.1 W

2.11

44.0 s

44.6 s

100%

128 / 192

vLLM

355.7

170.5 GB

749.8 W

2.11

66.3 s

118.1 s

100%

128 / 250

TGP

343.5

167.9 GB

735.5 W

2.14

65.8 s

66.9 s

100%

128 / 250

vLLM

343.3

170.5 GB

749.9 W

2.18

80.0 s

157.8 s

100%

192 / 192

TGP

381.5

168.2 GB

713.8 W

1.87

56.7 s

58.0 s

100%

192 / 192

vLLM

350.1

170.5 GB

749.9 W

2.14

95.8 s

178.6 s

100%

192 / 250

TGP

405.6

168.2 GB

719.1 W

1.77

70.3 s

71.5 s

100%

192 / 250

vLLM

340.0

170.5 GB

749.9 W

2.21

119.2 s

239.0 s

100%

256 / 192

TGP

376.2

160.8 GB

713.5 W

1.90

47.9 s

48.6 s

100%

256 / 192

vLLM

346.7

170.5 GB

749.9 W

2.16

127.5 s

241.0 s

100%

256 / 250

TGP

431.4

167.8 GB

721.1 W

1.67

74.7 s

75.8 s

100%

256 / 250

vLLM

42.7

170.5 GB

749.9 W

17.55

158.0 s

289.9 s

12.1%

Table 4. Qwen 72B stress test results by users, output length, throughput, and success rate.

Table 4 shows that TGP’s advantage is not uniform across all regimes. In lower-pressure scenarios, the tested vLLM baseline can remain competitive because it benefits from mature batching, paging, and scheduling. The relevant behavior appears as concurrency and output length increase together. At that point, the test moves from a pure throughput comparison toward a memory-residency and completion-stability comparison.

The strongest result occurs in the highest-pressure scenario.

Image Zoom
Figure 5. Highest-pressure scenario: 256 concurrent users and 250 output tokens.
Figure 5. Highest-pressure scenario: 256 concurrent users and 250 output tokens.

At 256 concurrent users and 250 output tokens, the TGP prototype sustained approximately 431 tokens per second with a 100% success rate. Under the same tested scenario, the tested vLLM baseline fell to approximately 42.7 tokens per second, with a success rate close to 12%. This corresponds to approximately 10.1× higher throughput for TGP in that specific scenario.

This result should be interpreted as a degradation-profile difference, not as a universal speed claim. TGP did not win by making each token intrinsically cheaper. It remained stable because its execution model reduced the amount of session state that had to remain resident in active GPU memory at the point of maximum pressure.

The architectural significance is therefore resilience. In this stress test, TGP showed that segmented execution and recyclable KV-cache residency can preserve throughput and completion rate when model size, user concurrency, generated length, and active memory pressure jointly stress the hardware.

9. Where This Leaves Things

For developers: the core hypothesis is that the KV cache doesn't have to be treated as an untouchable cumulative history. Making it local, transient, and replaceable, provided continuity can be preserved through compact representations, opens a design space for long-session and high-concurrency serving that a fixed-window approach (StreamingLLM) or a fixed-memory-budget approach (vLLM alone) doesn't cover on its own.

For decision-makers: the Qwen 72B results narrow, rather than inflate, the claim. This does not displace vLLM as a mature serving engine on throughput, maturity, or scheduling sophistication;  vLLM still wins or ties at moderate load. What it identifies is a specific, well-defined operating region,  large models, high concurrency, long or segmented generations, VRAM near its practical limit; where TGP-style segmentation preserves throughput and completion stability while the conventional stack degrades sharply. If your serving workload lives comfortably below that pressure threshold, this likely isn't yet the architecture to prioritize. If concurrency and session length are trending upward against a fixed VRAM budget, it's worth evaluating directly against your own traffic shape.

The most realistic near-term path isn't "TGP replaces vLLM"; it's TGP-style segmentation and continuity management layered onto an existing serving engine, retaining vLLM's batching and scheduling maturity while adding bounded memory residency for the regimes that need it.

Get involved

TGP is still an experimental, actively evolving protocol — the local and AMD Instinct MI300X results above are stress tests, not a finished production server. If you're evaluating long-context or high-concurrency inference infrastructure and want to discuss the architecture, benchmark methodology, or a pilot against your own workload, reach out to the ARUY team at https://www.aruy.net

Share:

Related Blogs

View All Blogs