Dismantling the Memory Wall: The Evolution of LLM Inference Architectures

2026-09-17

The commercialization of Generative Artificial Intelligence has encountered a severe bottleneck: the memory wall of autoregressive text generation. While computation remains critical, the dynamic state memory required to process tokens—known as the Key-Value (KV) cache—scales linearly and unpredictably, historically resulting in up to 80% memory waste on expensive GPU hardware. This report provides a definitive analysis of the structural evolution of LLM inference architectures, demonstrating how infrastructure engineers are dismantling this memory wall by adopting decades-old Operating System paradigms. The fundamental recommendation for modern deployments is to abandon naive static batching in favor of iteration-level scheduling (Continuous Batching) paired with advanced memory orchestration like PagedAttention and RadixAttention. Ultimately, transitioning to these frameworks ensures near-optimal compute utilization, vastly reduces latency spikes for end-users, and drops operational costs by maximizing the concurrent throughput of existing physical hardware.

Key Points:

Research indicates that naive Large Language Model (LLM) serving systems waste up to 80% of GPU memory due to architectural fragmentation and over-reservation constraints.
The application of operating system virtual memory paradigms—specifically non-contiguous paging—has emerged as a structural solution to the Key-Value (KV) cache memory wall.
Advanced iteration-level scheduling, commonly referred to as continuous batching, fundamentally shifts inference from static, queue-based processing to dynamic, token-by-token resource allocation.
Evidence suggests that tree-based caching mechanisms (such as RadixAttention) provide near-zero-cost memory sharing for agentic swarms and multi-turn workflows, resolving the redundancy limits of flat memory paging.

The commercialization and widespread deployment of Generative Artificial Intelligence (AI) have exposed a critical bottleneck in modern computing infrastructure: the memory wall of autoregressive decoding. While model weights capture the static knowledge of a Large Language Model (LLM), the dynamic state required to generate coherent text—known as the Key-Value (KV) cache—scales unpredictably and consumes massive amounts of High Bandwidth Memory (HBM) (HBM - a specialized RAM architecture stacked closely to the GPU die for massive parallel bandwidth, boasting hardware capacities like the 80GB found in NVIDIA A100s or 144GB in Grace Hopper configurations). Without intelligent orchestration, the sheer volume of this transient data chokes graphics processing units (GPUs), plummeting hardware utilization to fractional percentages (often plummeting to a baseline utilization percentage of merely 20% to 30%). However, a profound paradigm shift is underway. By borrowing decades-old resource management concepts from traditional Operating Systems (OS)—such as virtual memory, paging, and prefix trees—AI infrastructure engineers are systematically dismantling the memory wall. This report deconstructs the structural evolution of LLM inference serving, analyzing the compounding effects of Continuous Batching, PagedAttention, Chunked Prefill, and RadixAttention.

The Anatomy of Autoregressive Inference and the KV Cache Bottleneck

To understand the paradigm shift in AI infrastructure, one must first decouple the mechanics of autoregressive text generation. LLM inference operates fundamentally in two distinct phases: the Prefill phase and the Decode phase [cite: 1].

The Prefill phase is highly compute-bound. When a user submits a prompt, the model processes the entire token sequence in a single, massive forward pass. This phase exhibits high arithmetic intensity, fully saturating the GPU's matrix multiplication units to construct the initial context [cite: 1, 2]. The output of this phase is not just the first generated token, but the initial KV cache.

The Decode phase, conversely, is heavily memory-bound. The model generates one token at a time. To avoid recalculating the attention scores for all preceding tokens at every step, the system caches the Key and Value vectors of historical tokens [cite: 1, 2]. Mathematically, the Query (Q) vector for a new token is immediately discarded after its forward pass because it is only required once to compute the attention weights (dot products) against historical keys. Conversely, the historical Key (K) and Value (V) vectors must be retained because every future Query vector will need to compute attention scores against them to maintain contextual awareness. While mathematically elegant, this caching mechanism introduces a severe logistical crisis in production environments.

The KV cache is exceptionally large and highly dynamic. For instance, serving a single sequence on a LLaMA-13B model can consume up to 1.7GB of VRAM (Video Random Access Memory, the physical memory natively attached to the GPU) just for the cache [cite: 3]. In early, naive implementations (such as standard PyTorch inference loops or early HuggingFace Transformers), memory was managed through static, contiguous pre-allocation. If a system allowed a maximum context length of 2048 tokens, it blindly reserved 2048 tokens' worth of memory for every incoming request, regardless of whether the user asked a five-word question or submitted a massive document [cite: 2, 4].

This rudimentary approach birthed three distinct vectors of memory waste:

Internal Fragmentation: Memory reserved for future tokens that are never actually generated because the model outputs an end-of-sequence token earlier than expected [cite: 4]. Historically, this accounted for approximately 2% to 5% of total memory waste.
External Fragmentation: Gaps of free memory scattered across the GPU that are too small to fit the massive contiguous tensors required for new requests [cite: 2, 4]. This represented a massive 30% to 50% of memory waste due to contiguous pre-allocation chunking limits.
Redundant Storage: Identical system prompts or shared conversational histories being recalculated and stored repeatedly across thousands of parallel users [cite: 4, 5]. This accounted for roughly 20% to 30% of aggregate memory waste.

Research confirms that these inefficiencies resulted in systems wasting 60% to 80% of their available GPU memory, artificially capping the maximum concurrent batch size and leaving compute units idle [cite: 3, 6]. Resolving this required completely abandoning static execution graphs.

The Orchestration Shift: Orca and Continuous Batching

Before the optimization of the memory layer, researchers had to fix the scheduling layer. Early inference engines relied on static batching. A batch of user requests would be grouped, padded to match the length of the longest sequence, and pushed through the GPU. The entire batch had to wait for the single longest request to finish generating before the GPU could accept new work [cite: 7, 8]. This "head-of-line blocking" meant that if one user requested a 1,000-word essay and seven others requested single-sentence answers, the GPU spent the vast majority of its cycles processing padded (empty) tokens for the seven completed requests while waiting for the essay to finish.

The solution arrived with the Orca framework, presented at OSDI '22, which introduced the concept of Continuous Batching (also known as iteration-level scheduling or dynamic batching) [cite: 9, 10]. Orca Framework Profile: Functional Scope: Advanced iteration-level inference scheduler. Current Price/Cost: Open Source academic research. Availability: Described originally in OSDI '22, its underlying concepts are now integrated into almost all modern engines. Real-World Context: Ideal for understanding baseline continuous batching architecture; anti-use case: it is not typically deployed as a production-ready standalone server today, but rather serves as foundational theory.

Continuous batching fundamentally alters the lifespan of a sequence within a batch. Instead of freezing the batch until all sequences complete, the system evaluates the batch at every single decode step (iteration). When a sequence emits a stop token, the scheduler immediately evicts it, reclaims its memory, and slots a new waiting request into the now-empty space for the very next token generation step [cite: 7, 10].

To contextualize the performance multiplier of continuous batching, consider the following performance scaling dynamics observed in foundational studies.

Scheduling Paradigm	Mechanism	Throughput Multiplier	GPU Utilization
Static Batching	Fixed batch size; waits for longest sequence.	1.0x (Baseline)	~30-40%
Continuous Batching (Orca)	Token-level eviction and injection.	Up to 36.9x over standard baselines.	Near optimal

The data clearly illustrates a structural leap. Orca reported up to a 36.9x throughput improvement over NVIDIA's FasterTransformer on a GPT-3 175B parameter model [cite: 7, 10]. Later implementations, such as those by Anyscale using vLLM, demonstrated up to 23x improvements in throughput over naive static batching by ensuring the GPU acts as a continuous assembly line [cite: 7]. However, continuous batching is only viable if the system can dynamically allocate memory for incoming requests on the fly. This requirement catalyzed the invention of PagedAttention.

PagedAttention: The Virtual Memory Analogy for GPUs

While Orca proved that iteration-level scheduling could maximize throughput, it remained bottlenecked by how PyTorch and CUDA handle memory contiguousness. If an incoming request needed more memory, but the GPU only had fragmented free space, the engine would crash with an Out of Memory (OOM) error. The vLLM team solved this by looking backward—specifically, to 1960s operating system design [cite: 11, 12].

vLLM Framework Profile: Functional Scope: High-throughput memory-efficient LLM serving engine built around PagedAttention. Current Price/Cost: Free/Open Source. Availability: Available via GitHub and PyPi. Real-World Context: Ideal for high-concurrency API backends and enterprise serving; anti-use case: not suitable for edge/mobile client-side deployments with strict hardware limits.

In standard computing, OS virtual memory separates how a program sees memory from how the RAM physically stores it. Memory is broken into fixed-size "pages." The program operates as if it has one long, continuous block of memory (logical space), while the OS scatters these pages wherever they fit in physical RAM, using a page table to map the relationships [cite: 6, 11, 13].

PagedAttention directly ports this concept to the KV cache [cite: 3]. Instead of forcing a sequence's KV cache to reside in one massive, unbroken tensor, PagedAttention partitions the cache into small, fixed-size KV Blocks (usually containing the keys and values for 16 tokens) [cite: 13]. These physical blocks can be scattered anywhere in the GPU's VRAM. The system utilizes a Block Table to translate the model's logical token sequence into physical GPU memory addresses [cite: 4, 13, 14].

To ground this in software engineering terms, observing the core data structure of PagedAttention demystifies its magic. Below is an illustrative extraction of a Python Block Table conceptualization used to map virtual tokens to physical GPU blocks.

class BlockTable:
 """
 Per-request 'page table': maps logical block index -> physical block.
 Logical blocks are contiguous (0,1,2...), physical blocks can be scattered.
 """
 def __init__(self):
 self.logical_to_physical: List[KVCacheBlock] = []
 
 def __len__(self):
 return len(self.logical_to_physical)
 
 def get_block_for_token(self, token_idx: int, block_size: int):
 logical_block_idx = token_idx // block_size
 return self.logical_to_physical[logical_block_idx]

This decoupling of logical sequences from physical memory addresses has profound implications for system stability and throughput. Because blocks are allocated strictly on-demand, and because all blocks are identically sized, external fragmentation is virtually eliminated. Furthermore, internal fragmentation is isolated strictly to the final block of a sequence (wasting at most 15 tokens' worth of memory). Research indicates that PagedAttention reduces KV cache memory waste to under 4%, allowing the engine to safely pack 2x to 4x more concurrent requests onto the exact same hardware [cite: 3, 6].

Implementation: Deploying vLLM with Continuous Batching

To practically realize these gains, deploying an optimized engine requires minimal logistical overhead. The following numbered guide details how a developer configures and spins up vLLM with continuous batching natively enabled:

Install the framework: Execute package installation via pip, ensuring compatibility with your system's CUDA version.
```
pip install vllm
```

Initialize the Async Engine: Import the library and initialize the AsyncLLMEngine. Continuous batching and PagedAttention are active by default.

from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.engine.async_llm_engine import AsyncLLMEngine

engine_args = AsyncEngineArgs(
 model="meta-llama/Llama-2-7b-chat-hf",
 gpu_memory_utilization=0.90, # Dedicate 90% of VRAM to KV Cache/Weights
 max_num_batched_tokens=4096 # Enforce chunked/continuous limits
)
engine = AsyncLLMEngine.from_engine_args(engine_args)

Generate Token-by-Token: Yield requests asynchronously. The underlying scheduler will automatically evict finished sequences and inject new ones at the iteration level without stalling the event loop.

import uuid
from vllm import SamplingParams

async def generate(prompt):
 request_id = str(uuid.uuid4())
 sampling_params = SamplingParams(temperature=0.7, max_tokens=100)
 
 # Asynchronously stream output as the scheduler dynamically batches
 async for request_output in engine.generate(prompt, sampling_params, request_id):
 print(request_output.outputs[0].text)

Copy-on-Write and Advanced Decoding Strategies

The architectural genius of PagedAttention extends beyond simple memory density; it natively unlocks complex, memory-heavy decoding algorithms like Beam Search and Parallel Sampling [cite: 3, 15].

In Beam Search, a model explores multiple distinct conversational paths (beams) simultaneously to find the highest probability response. To define it further: Beam Search acts much like a grandmaster chess player evaluating multiple potential future move sequences concurrently before committing to the absolute best overarching path, rather than just greedily choosing the most obvious immediate move. In traditional systems, each beam required copying the entire KV cache history, rapidly exhausting VRAM [cite: 13]. With PagedAttention, the system utilizes Copy-on-Write (CoW) semantics [cite: 11]. All parallel beams start by referencing the exact same physical blocks for the initial prompt, incrementing a reference count. When a specific beam diverges and generates a unique token, only then does the system allocate a new physical block for that specific branch [cite: 3, 13]. In the context of GPU memory constraints, this is profoundly relevant as it prevents the exponential explosion of cache copies. This deduplication cuts memory usage for complex decoding by up to 55%, translating directly to a 2.2x improvement in throughput [cite: 3].

Chunked Prefill: Eradicating Head-of-Line Blocking

While Continuous Batching and PagedAttention solved the memory and throughput crises, a critical latency issue remained for streaming applications. In real-world workloads, servers handle a mixture of short, interactive queries and massive Document Retrieval-Augmented Generation (RAG) prompts (RAG - a technique injecting external database context, such as corporate PDFs, directly into a user's prompt before inference) [cite: 16, 17].

If the engine is currently streaming responses token-by-token to ten users, and an eleventh user submits a 32,000-token prompt, the system must execute the compute-heavy Prefill phase for those 32K tokens in a single forward pass. On an Nvidia H100 running a 70B parameter model, this computation monopolizes the GPU for 200 to 400 milliseconds [cite: 17, 18]. During this window, the ten users currently receiving tokens experience a complete freeze. This phenomenon, known as head-of-line blocking, severely degrades the Inter-Token Latency (ITL, the time delay between each generated word) and creates unacceptable spikes in Time to First Token (TTFT, the initial wait time before text generation begins) at the 95th percentile (p95, meaning 95% of requests are faster than this threshold, highlighting the worst-case tail delays) [cite: 17, 18, 19].

Chunked Prefill (sometimes integrated alongside decode-maximal batching strategies like SARATHI, a specialized inference scheduler designed to explicitly interleave compute-heavy prefill chunks with memory-bound decode steps) fundamentally redesigns inference scheduling to protect latency constraints [cite: 19]. Instead of processing a massive prompt in one uninterrupted operation, the engine divides the prompt into strict token-budget slices [cite: 16, 18].

If the scheduler has a budget of 4,096 tokens per iteration, and receives a 32,000-token prompt, it processes the first 4,096 tokens, pauses the prefill, and immediately executes a decode step to serve the active users. It then resumes the prefill for the next chunk [cite: 17, 19]. By interleaving prefill slices with decode tokens, the system ensures that long prompts never starve active requests. Empirical data demonstrates that utilizing chunked prefill on mixed workloads can reduce p95 TTFT spikes by up to 68% (e.g., dropping latency from 2,800 ms down to 890 ms) while maintaining perfectly smooth token streams for active clients [cite: 17].

RadixAttention: The Tree of Life for Agentic Swarms

PagedAttention mastered memory sharing within the exact same request (e.g., beam search), but it natively treats every distinct user request as an isolated entity. In modern AI applications—particularly system prompt heavy LLMs, multi-turn chat applications, and agentic swarms exploring vast logical trees—upwards of 80% of the input tokens may be identical across entirely different API calls [cite: 20, 21].

For example, if 1,000 users interact with a custom customer service bot, all 1,000 requests begin with the exact same 2,000-token hidden system prompt dictating rules and tool definitions [cite: 20, 21]. Standard inference servers, even those running PagedAttention, recompute and store the KV cache for this identical prefix 1,000 separate times [cite: 20].

The RadixAttention paradigm, introduced by the SGLang framework, solves this global redundancy by abandoning flat mapping in favor of a Radix Tree (a compressed trie) data structure [cite: 5, 22, 23].

SGLang Framework Profile: Functional Scope: Flexible prompt programming language and highly optimized runtime utilizing RadixAttention. Current Price/Cost: Open Source. Availability: Available via GitHub. Real-World Context: Ideal for complex multi-agent systems, structured JSON generation, and environments with heavy prompt reuse; anti-use case: less beneficial for single-turn, isolated queries with zero historical overlap.

A Radix Tree is a space-optimized prefix tree where nodes with only one child are merged with their parents. As a real-world analogy, imagine a library classification system where instead of having separate massive folders for 'Science-Biology-Genetics' and 'Science-Biology-Anatomy', the system shares a single continuous 'Science-Biology' pathway and only branches at the specific sub-discipline. In the context of GPU memory, this means instead of discarding the KV cache after a generation request completes, RadixAttention permanently retains the cached activations in a global tree [cite: 23, 24]. The edges of this tree represent sequences of tokens. When a new request enters the system, SGLang performs a longest-prefix match against the Radix Tree. If a request shares a prefix with historical data (such as the 2,000-token system prompt), it directly references the existing memory locations and only computes the novel tokens, executing zero-cost memory sharing [cite: 5, 21, 22].

To illustrate how this state logic is physically maintained in SGLang, analyzing the node-level data structure provides a clear picture of tree-based memory management.

class TreeNode:
 def __init__(self, id=None, priority=0):
 self.children = defaultdict(TreeNode)
 self.parent: TreeNode = None
 self.key: RadixKey = None # Token sequence
 self.value: torch.Tensor = None # KV cache indices
 self.lock_ref = 0 # Reference count to prevent eviction
 self.last_access_time = time.time()

The snippet above reveals the operational safety mechanisms inherent to RadixAttention. The lock_ref integer acts as a critical concurrency lock. When a request traverses the tree and utilizes a specific sequence of tokens, it increments the reference counter for that node and all its ancestors [cite: 22, 24, 25]. This mathematically guarantees that the memory cannot be inadvertently overwritten while in use.

However, GPU memory is finite. When the system exhausts its VRAM budget, it must intelligently prune the tree. RadixAttention employs a highly customized Least Recently Used (LRU) eviction algorithm [cite: 23, 24]. Crucially, the eviction logic only targets leaf nodes (nodes with no children) that possess a lock_ref of zero [cite: 24, 25]. Striking at the leaves ensures that foundational shared prefixes (like the root system prompts) remain intact and highly accessible, while unique, conversational dead-ends are continuously pruned to free up physical cache space [cite: 24, 25].

The Calculus of Multi-Turn and Agentic Scaling

The throughput implications of RadixAttention are most aggressively realized in multi-turn and agentic paradigms. In a multi-turn chat, each subsequent user message contains the entire history of the preceding conversation. Under standard engines, this history is recalculated repetitively. RadixAttention ensures that Turn 5 simply appends to the leaf node created by Turn 4, requiring computation only for the user's newest sentence [cite: 20, 21].

Similarly, for autonomous agents exploring branching logical paths (e.g., Tree of Thoughts programming), an agent can fork its state, explore Branch A, retreat to the root of the fork in memory, and execute Branch B with zero recomputation of the shared context [cite: 5]. Workloads exhibiting 60% or higher prefix overlaps routinely achieve 75% to 95% cache hit rates, allowing SGLang to achieve up to 5 times higher throughput than non-prefix-aware systems like earlier versions of vLLM or Microsoft Guidance [cite: 20, 23].

To ground this in reality, consider the real-world case study of the LMSYS Chatbot Arena. When LMSYS migrated their backend inference engine to utilize SGLang and RadixAttention, the platform—which heavily relies on overlapping system prompts, user multi-turn dialog history, and repetitive moderation guardrail evaluations—observed extraordinary cache hit rates in production. By intelligently caching shared conversational branches across thousands of simultaneous judge interactions, the system bypassed the compute-heavy prefill phase for the majority of requests, actively realizing the 75%+ hit rates previously observed only in academic benchmarks.

Memory Paradigm Comparison

To consolidate these innovations, the following table contrasts the functional and structural efficiencies of these three dominant memory allocation paradigms:

Memory Architecture	Allocation Mechanism	Cross-Request Memory Sharing	Aggregate Memory Waste (%)
Standard PyTorch (Baseline)	Contiguous, static maximum pre-allocation.	None. Total isolation per request.	60% – 80%
PagedAttention (vLLM)	Non-contiguous, block-based virtual memory paging.	Intra-request only (e.g., Beam Search).	< 4%
RadixAttention (SGLang)	Global prefix tree (compressed trie) state caching.	Global sharing across entirely distinct users/APIs.	Near 0% (For shared prefixes)

Cross-Tier Memory Offloading: Beyond the Physical VRAM Limit

The next logical question facing infrastructure architects is an inevitable limit of physics: What happens when a single sequence's context window (e.g., pushing beyond 1 to 2 million tokens) completely exceeds the total physical VRAM limits of the GPU, rendering even PagedAttention insufficient?

To solve this, the frontier of AI infrastructure is pivoting toward Cross-Tier Memory Offloading. Frameworks such as FlexGen introduce an asynchronous memory hierarchy that actively bridges GPU VRAM, CPU System RAM, and NVMe SSD storage. If the KV cache for a massive sequence overflows the GPU, the system dynamically evicts the least recently accessed KV blocks down to the CPU RAM, or further down to the NVMe disk, acting as a massive swap space. While disk retrieval inherently introduces latency, techniques like speculative pre-fetching ensure that future required blocks are hoisted back into GPU VRAM just in time for the attention computation, effectively breaking the physical memory ceiling of individual accelerator hardware.

Synthesis: The Operating Systemification of the AI Stack

The evolution from static, contiguous memory allocation to continuous batching, PagedAttention, chunked prefill, and RadixAttention represents a terminal paradigm shift. The AI infrastructure stack is no longer merely a conduit for tensor calculus; it has biologically evolved into a hyper-specialized Operating System.

Where operating systems manage CPU cycles and RAM pages for disparate applications, modern inference engines manage GPU token iterations and KV cache blocks for massive, concurrent swarms of neural networks. The convergence of these technologies allows developers to achieve inference economics that were physically impossible just two years ago.

Looking toward the near frontier, architectures are already abstracting this concept beyond the single GPU. Nascent distributed systems, like LMCache and Mooncake, are separating the KV cache entirely from the execution node, creating global cache pools across multiple servers interconnected by ultra-high-speed networks (such as InfiniBand NDR operating at 400 Gbps) [cite: 24].

Distributed Caching Profiles (LMCache & Mooncake): Functional Scope: Disaggregated, network-level KV cache sharing and routing. Current Price/Cost: Open Source ecosystem tools. Availability: GitHub (actively researched). Real-World Context: Designed for hyperscale data centers requiring multi-node KV sharing; anti-use case: unnecessary overhead for single-node or localized deployments.

In these topologies, a centralized router utilizes an abstract view of a global Radix tree to intelligently route user requests to the specific server that currently holds the required prefix in its local VRAM [cite: 24].

Downstream Implications: Economics, Society, and Environment

The architectural solutions to the memory wall ripple far beyond the technical backend, manifesting powerful downstream implications:

Economic Impacts: By compressing memory waste from 80% to under 4%, continuous batching and PagedAttention effectively quadruple the throughput of a single GPU. For API providers, this slashes the cost of goods sold, driving inference prices down by orders of magnitude (as witnessed by the race to the bottom in open-weights API pricing) and making AI financially viable for lower-margin business integrations.
Social & Democratization Impacts: Reducing the VRAM requirement for long-context generation democratizes local AI. Hobbyists, researchers, and small enterprises can now run sophisticated agentic workflows on consumer-grade hardware or highly limited server racks, preventing an intelligence monopoly strictly held by hyperscalers.
Environmental Impacts: LLM inference is notoriously power-hungry. By ensuring that compute units are saturated (via chunked prefill and dynamic batching) rather than idling while waiting for memory allocations, these frameworks vastly increase the number of tokens generated per watt of electricity, meaningfully curbing the aggregate carbon footprint of the AI industry.

As large language models transition from stateless conversational bots to stateful, long-horizon autonomous agents, the ability to effortlessly recall, branch, and mutate context without paying a computational penalty will dictate the ceiling of AI capability. Through the intelligent fragmentation of memory and the continuous orchestration of compute, the infrastructure layer has successfully insulated the application layer from the physics of the memory wall.

Sources & References