Is memory becoming the bottleneck for AI?
In our AI infrastructure market deck, you will find everything you need to understand the market
SUMMARY
Yes. Memory is becoming the bottleneck for AI, most clearly in token-by-token inference, long-context workloads and systems trying to serve many users at once.
The important shift is not simply that models need more gigabytes. AI systems are increasingly limited by how quickly they can move weights, KV-cache data and intermediate state to the processor, which makes bandwidth and data placement as important as raw capacity.
The newest accelerators make the direction unusually obvious. NVIDIA Rubin and AMD MI455X deliver roughly 22 to 23.3 TB/s of HBM bandwidth, about an order of magnitude above the A100, showing that chipmakers are spending heavily just to keep arithmetic units supplied with data.
Inference is the sharper problem because decoding generates one token at a time. The GPU repeatedly reads a large model and a growing cache while doing relatively little calculation, so headline FLOPS can rise much faster than the speed a user actually experiences.
The KV cache is changing the economics of serving. For a long-context model, one active conversation can consume as much memory as the compressed model weights, and concurrency multiplies that temporary state across every user still generating tokens.
Reasoning models and agents intensify the pressure because they keep jobs alive longer. A short final answer may hide dozens of model turns, tool calls and recovery steps, all of which create more weight traffic and more state that must be retained, compressed or retrieved.
Many fixes move the bottleneck rather than removing it. Sharding saves local memory but increases network communication; CPU or SSD offload adds capacity but can create transfer delays; mixture-of-experts reduces computation per token while leaving a large expert pool distributed across the system.
Quantization, FlashAttention and PagedAttention have produced major gains, but the freed memory rarely stays free. Providers usually reinvest it in larger models, longer contexts, bigger batches or more simultaneous users, so efficiency improvements delay the wall without making memory strategically unimportant.
The constraint now reaches beyond the chip. HBM manufacturing, advanced packaging, GPU-to-GPU links, CXL memory, flash tiers and scheduling software all affect whether expensive processors spend their time calculating or waiting.
Training remains more conditional because networking, synchronization, power and data can become the immediate limit. Even there, memory determines how the job must be partitioned, and the resulting communication often becomes the next bottleneck.
Our conclusion is that AI is moving from a compute-first era toward a data-movement era. Future gains will depend less on theoretical FLOPS alone and more on how efficiently systems fit, move, compress, reuse and forget information.
This market map, featured in our AI infrastructure market deck, highlights top companies and startups in the AI infrastructure market
Why is AI memory becoming a bottleneck now?
AI memory is becoming a bottleneck now because the industry is adding arithmetic power faster than many important workloads can supply it with data.
The clearest clue comes from the newest accelerators. NVIDIA’s Rubin combines 288 GB of HBM4 with 22 TB/s of memory bandwidth. AMD’s MI455X reaches 432 GB and 23.3 TB/s. Compared with the 80 GB and roughly 2 TB/s of NVIDIA’s A100, the latest designs offer several times more capacity and about an order of magnitude more bandwidth. The striking part is the bandwidth jump: chipmakers are spending huge amounts of silicon, packaging capacity and power simply to keep their processors fed.
The workload has changed at the same time. Earlier chatbots mostly handled short prompts and short answers. Today’s reasoning models generate many more internal and visible tokens, while agents keep working through documents, tools and software over longer periods. OpenAI’s current GPT-5.4 supports around one million tokens of context, and NVIDIA now describes agentic inference in terms of sustained reasoning, long-context attention and large KV caches. More intelligence is increasingly being bought with more work at inference time, which raises the amount of model data and temporary state that must be moved for each useful result.
Memory has always mattered in AI. What is new is how often it now limits the commercial outcome people care about: tokens per second, users per GPU, context length, response latency and cost per completed task. The bottleneck is moving closer to the customer-facing part of AI, where small inefficiencies are multiplied across millions of requests.
What does a “memory bottleneck” actually mean for AI?
An AI memory bottleneck means the processor has usable computing power that cannot be fully exploited because the right data does not fit nearby or arrive quickly enough.
People often mix together four separate problems. Capacity decides whether model weights, training states and active conversations can fit. Bandwidth decides how quickly those numbers reach the processor. Latency matters when data has to cross a slower connection for every generated token. Supply and cost decide how many complete systems can actually be built.
Each problem calls for a different fix. Quantization helps a model fit and usually reduces traffic. Faster HBM improves bandwidth. CXL, CPU memory and SSDs add cheaper space, though with a speed penalty. More HBM production eases supply, while a poorly designed inference engine can still waste all of it.
In practice, AI data can sit in several places. The GPU keeps a small amount extremely close to its compute cores, stores much more in nearby HBM, and can reach into server RAM, SSDs or remote storage when necessary. Farther-away storage is cheaper and larger, but slower. The difficult choice is what must stay close enough to be used immediately.
| Memory problem | What the user experiences | Typical cause | Common response |
|---|---|---|---|
| Capacity | The model or context does not fit | Too many weights, activations or cached tokens | Quantization, sharding, larger HBM |
| Bandwidth | Slow token generation | Weights or KV data cannot be read quickly enough | Faster HBM, batching, data reuse |
| Latency | Pauses and poor interactivity | Repeated transfers across PCIe or a network | Keep hot data local, prefetch, better scheduling |
| Supply and cost | Scarce or expensive AI servers | Limited HBM and advanced packaging output | Long-term supply deals, more fabrication capacity |
As this chart shows, and as featured in our AI infrastructure market deck, search interest in AI infrastructure has risen sharply
Is AI training already limited by memory?
AI training is heavily constrained by memory today, although large clusters can trade that constraint for extra communication and recomputation.
A model’s weights are only the beginning. Standard mixed-precision AdamW training has commonly required about 16 bytes per parameter once weights, gradients, master weights and optimizer states are counted. That puts a 70-billion-parameter model near 1.12 TB before activations and temporary buffers. A 405-billion-parameter model approaches 6.5 TB, while a trillion-parameter model reaches roughly 16 TB. The original ZeRO work was built around this problem, and a newer 2026 study called FlashOptim still treats per-parameter optimizer memory as a major barrier.
Training frameworks get around the limit by splitting model states across GPUs. PyTorch’s Fully Sharded Data Parallel divides parameters, gradients and optimizer states across workers, allowing models that would never fit on one device to run. The price is more communication because each GPU repeatedly gathers the pieces it needs. PyTorch’s own documentation states that FSDP saves memory at the cost of communication, while ZeRO++ showed how serious that new constraint can become: cutting communication volume fourfold improved throughput by as much as 2.16 times at 384-GPU scale.
Activations create another large bill, especially with long training sequences. Teams can discard some activations and recalculate them during the backward pass, or shrink batch sizes until the run fits. Both choices reduce memory use by spending more compute or accepting lower hardware utilization.
Memory capacity decides how a large training job must be divided. Once the model is spread across many GPUs, networking and synchronization often decide how efficiently it runs. The original constraint has simply moved into another part of the system.
The estimates below show raw orders of magnitude. Actual runs vary with precision, optimizer, sharding and checkpointing.
| Model size | BF16 weights only | Four-bit weights only | Approximate AdamW training states |
|---|---|---|---|
| 70B parameters | 140 GB | 35 GB | 1.12 TB |
| 405B parameters | 810 GB | 202.5 GB | 6.48 TB |
| 671B parameters | 1.34 TB | 335.5 GB | 10.74 TB |
| 1 trillion parameters | 2 TB | 500 GB | 16 TB |
Is AI inference more memory-bound than training?
AI inference is more consistently memory-bound than training, especially while a language model generates tokens one by one.
Training processes large blocks of tokens through big matrix multiplications, giving the GPU many calculations to perform for each byte it reads. Prompt processing, usually called prefill, behaves similarly. The decode phase is much less favorable: the model generates one new token, reads a large amount of weight and cache data, performs a relatively small amount of work, then repeats the cycle.
A recent Berkeley review of memory-efficient inference reaches the same conclusion from profiling: memory bandwidth is the main LLM inference bottleneck, and the dominant source of traffic shifts from model weights to the KV cache as context grows. NVIDIA’s technical guidance also describes autoregressive decode as a phase that underuses available compute because it moves so much data for each generated token.
Batching can improve the economics. When many requests are decoded together, one weight read contributes to several users, so total throughput rises. Yet larger batches need more cache space and can make an individual user wait longer. Providers therefore face a real trade-off between low latency and high utilization.
Inference has become the sharper memory problem. A training run can be planned around large batches and a fixed cluster. A public AI service has to absorb short prompts, million-token prompts, traffic spikes, long agent sessions and impatient users at the same time. Memory decides how many requests can coexist and how quickly each one moves forward.
If you want more recent data on this point, please see our latest AI infrastructure market report.
This chart, included in our AI infrastructure market deck, shows annual VC investment in AI infrastructure startups
Why can a faster GPU still generate tokens slowly?
A faster GPU can still generate tokens slowly when its arithmetic units spend most of their time waiting for weights and cached context.
Imagine a dense 70B model stored in BF16. Its raw weights occupy about 140 GB. If one decoding step had to stream that full amount through a GPU with 3 TB/s of memory bandwidth, the transfer alone would take roughly 47 milliseconds in a perfect system. That creates a theoretical ceiling near 21 tokens per second before attention work, communication and software overhead are added.
Raise bandwidth to 8 TB/s and the same simplified floor falls to about 18 milliseconds, or 57 tokens per second. At 22 TB/s, it falls to a little over six milliseconds, or roughly 157 tokens per second. Real performance will be lower, but the calculation explains why HBM bandwidth can matter more than headline FLOPS during interactive decoding. The bandwidth figures roughly correspond to H100-class hardware, current 8 TB/s accelerators and Rubin.
Four-bit quantization changes the same calculation by shrinking the raw weights from 140 GB to about 35 GB. Batching changes it again by reusing one weight read across several requests. Those are powerful gains because they increase the useful work performed per byte moved.
Engineers call the crucial ratio arithmetic intensity: how much calculation happens for every byte fetched. Training and prompt processing usually reuse data well, whereas small-batch decoding repeatedly reads large amounts for very little work. A processor can gain several times more low-precision compute and still show only a modest improvement in chat speed when memory bandwidth barely moves.
Is the KV cache becoming a bigger problem than model weights?
The KV cache is already becoming a bigger memory problem than model weights in long-context and high-concurrency inference.
A transformer saves key and value representations from earlier tokens so it can generate the next token without recomputing the whole conversation. That cache grows with every token and every active request. Model weights remain mostly fixed after the model loads; user state keeps accumulating.
We can see the crossover with a familiar model. Meta’s Llama 3.1 70B has 80 layers, eight key-value heads, a head dimension of 128 and a 128K context window. In BF16, those dimensions produce roughly 320 KB of KV cache per token. One fully used 128K context therefore needs around 40 GB of cache. The same model’s raw four-bit weights occupy about 35 GB. One long conversation can require more memory for temporary context than for the compressed model itself.
Concurrency makes the gap much larger. Sixteen such requests would approach 640 GB of KV cache before weights, buffers and fragmentation. Production traffic rarely consists entirely of maximum-length conversations, but the example shows why context length and user count cannot be planned separately.
Current pricing provides another clue. OpenAI’s GPT-5.4 supports a context window of roughly one million tokens, while API inputs above 272,000 tokens are priced at a higher multiplier. That pricing boundary reflects the extra infrastructure cost of very large contexts, even though providers do not publish their internal serving architecture.
Recent research has moved from merely compressing the KV cache to deciding what the model should forget. Microsoft researchers described linearly growing KV state as a long-context bottleneck when introducing IndexMem, while other 2026 work explores learned eviction and tiered caches. The research agenda itself has shifted: the scarce resource is increasingly the active history of users and agents.
If you want more recent data on this point, please see our latest AI infrastructure market report.
This chart, included in our AI infrastructure market deck, shows why CoreWeave is winning in AI infrastructure
Are reasoning models and AI agents making memory pressure worse?
Reasoning models and AI agents are pushing memory pressure higher because they keep inference running longer and carry more state from one step to the next.
Reasoning models improve by spending more compute at test time. OpenAI reported that o3 continued improving when allowed to think longer, after an order-of-magnitude increase in inference-time reasoning compared with its earlier scaling path. More reasoning means more generated tokens, more repeated weight reads and more cache entries before the final answer appears.
Agents add a second multiplier. A single user request may produce a plan, several searches, tool calls, code execution, error recovery and verification. The result can require dozens of model turns even when the final response is short. OpenAI’s current models are explicitly designed for long-running agentic workflows, and GPT-5.4 combines computer use with a context window of about one million tokens.
Keeping every old token in HBM would be wasteful. Good agent systems summarize completed work, store durable facts outside the prompt and retrieve only what matters. OpenAI’s coding models now use native compaction for long-running tasks, while NVIDIA has introduced a dedicated flash-based “context memory” tier for temporary KV data in Rubin-era systems.
The real shift is what providers are serving. An increasing share of AI usage comes from processes that remain active for minutes or hours rather than isolated questions. Their memory bill depends on task duration, branching, tool use and context reuse alongside the size of the base model.
Do mixture-of-experts models actually reduce AI memory needs?
Mixture-of-experts models slash the compute used per token, while their total memory footprint remains much closer to the full model size.
DeepSeek-V3 makes the difference easy to see. It contains 671 billion parameters but activates about 37 billion for each token. The model therefore performs far less computation than a dense 671B model would, which is a major reason mixture-of-experts designs can offer strong capability at a manageable inference cost.
The full expert pool still has to live somewhere. At BF16 precision, 671 billion raw weights occupy roughly 1.34 TB. Four-bit storage lowers that to around 336 GB, before quantization metadata and runtime buffers. Several GPUs can divide the experts among themselves, though each token must then reach the machines holding its selected experts.
That routing introduces new pressure on networking and scheduling. Popular experts can become overloaded, rare experts may sit underused, and small batches may fail to reuse expert weights efficiently. DeepSeek’s architecture therefore combines expert sparsity with Multi-head Latent Attention, which compresses attention state and attacks the KV-cache side of the problem as well.
Sparse models separate compute scale from total parameter scale far more successfully than they separate intelligence from data movement. MoE reduces the math performed for each token, yet the system still needs a large memory pool and a fast way to move tokens between experts.
This chart, included in our AI infrastructure market deck, shows annual funding in AI infrastructure startups
Are quantization and better software solving the AI memory problem?
Quantization and better software are delaying the memory wall, while larger contexts and higher concurrency quickly consume much of the space they free.
Quantization delivers the most obvious saving. Moving weights from BF16 to four bits cuts their raw footprint by about 75%, allowing a 70B model to fall from around 140 GB to 35 GB. Lower precision also reduces the amount of data read during decoding. NVIDIA has extended the same idea to the KV cache: its NVFP4 format can cut KV-cache memory by up to 50%, effectively doubling the available context budget in suitable workloads.
Software has removed large amounts of waste. FlashAttention speeds attention by reducing transfers between HBM and the GPU’s small on-chip memory. PagedAttention reorganizes KV storage into blocks, producing near-zero cache-allocation waste in its original tests and raising serving throughput by two to four times at similar latency. Both results came from treating memory movement and allocation as first-class design problems.
Newer work is becoming more aggressive. Google’s TurboQuant compresses KV data through low-bit representations without requiring model retraining, while FlashOptim reports cutting AdamW training memory from 16 bytes to seven bytes per parameter, or five with gradient release. These advances attack inference and training from different directions.
The rebound is already visible. Four-bit weights make a larger model affordable. A smaller KV cache encourages longer contexts or bigger batches. Better allocation lets a server accept more simultaneous requests. Efficiency lowers the memory required for one unit of work, then providers use the saving to sell more capability from the same machine.
These techniques push the memory wall farther away, while keeping memory strategically important. Their success also reveals how much performance had been trapped behind unnecessary data movement.
Can CPU RAM, SSDs and CXL absorb the AI memory overflow?
CPU RAM, SSDs and CXL can hold the overflow, though they remain too slow for data that must be touched during every token.
Speed is the catch. Current HBM systems move several terabytes each second. A PCIe link operates at a fraction of that rate, and SSDs are slower again. A 2026 study of KV-cache offloading found cases where transfers consumed as much as 99% of latency and GPUs used only 28% of their rated power because they were waiting for data.
Offloading still makes sense for colder state. Dormant conversations, shared prompt prefixes, less-used experts and old agent history can sit in ordinary RAM or flash until needed. NVIDIA’s latest Inference Context Memory Storage platform formalizes that idea by adding a high-bandwidth flash tier specifically for temporary KV data, positioned between GPU HBM and general storage.
CXL makes server memory easier to expand and share. The CXL 4.0 specification doubles the link rate from 64 to 128 GT/s and adds bundled ports for more aggregate bandwidth. That can help pooled memory behave like a practical extension of the server, especially for datasets or states that are too large for local DRAM.
The practical design is a hierarchy. Active weights and the hottest cache entries stay in HBM, warm state moves to CPU or pooled memory, and long-term records remain on SSDs or distributed storage. Predicting what will be needed next is the difficult part, because a cache miss can leave an expensive GPU waiting.
If you want more recent data on this point, please see our latest AI infrastructure market report.
This chart, included in our AI infrastructure market deck, compares the main business model options for AI cloud infrastructure providers
Is networking becoming a bigger AI bottleneck than memory?
Networking becomes the dominant limit in some large AI clusters because sharding turns remote data into part of the memory path.
Fully sharded training is the simplest example. Each GPU stores only part of the model states, which saves local capacity, then gathers parameters and synchronizes gradients over the network. ZeRO++ improved throughput by reducing that communication volume fourfold, showing that network traffic can absorb much of the benefit created by memory sharding.
Large mixture-of-experts systems create a similar pattern during inference. Tokens travel to the GPUs holding their chosen experts, then results must be combined. Long-context serving may also move KV state between machines that handle prompt processing and machines that handle generation.
Hardware vendors are responding with links that increasingly resemble memory buses. Rubin provides 3.6 TB/s of GPU-to-GPU NVLink bandwidth per GPU, while NVIDIA’s rack-scale systems connect 72 accelerators into one large domain. That level of investment would make little sense if local HBM alone could feed distributed models efficiently.
Memory and networking now form one broader data-movement constraint. Local bandwidth dominates many single-GPU decoding jobs, while collective communication can dominate frontier training or expert routing. The workload decides where the traffic jam appears; either way, compute ends up waiting for data.
Is the HBM supply chain restricting AI growth?
HBM supply is currently restricting how quickly and cheaply AI infrastructure can expand.
Micron’s latest financial update says customers recognize that memory and storage shortages will take considerable time to improve. The company still has no clear line of sight on when supply will catch rising demand, even though industry output is expected to improve gradually. Its entire calendar 2026 HBM supply had already been covered by price-and-volume agreements, and Micron expects the HBM market to grow from roughly $35 billion in 2025 to around $100 billion in 2028.
The constraint comes from manufacturing, not a lack of orders. HBM stacks multiple memory dies, adds a logic base, requires difficult packaging beside a very large processor and consumes cleanroom capacity that could produce conventional DRAM. Micron has begun volume shipments of 12-layer HBM4 for Rubin and has already sampled a 16-layer version with 48 GB per stack, showing how quickly suppliers are being pushed toward denser products.
SK hynix offers another useful measure of urgency. After moving HBM4 into production, it was already shipping HBM4E samples by June 2026. The transition to the next memory generation is beginning before the previous one has fully spread through the market.
AI deployment will continue, but the shortage changes allocation, prices, long-term contracts and which customers receive the newest systems first. Accelerator demand cannot become installed computing capacity until the memory and packaging arrive with it, which makes HBM a real bottleneck even without a complete halt.
This chart, featured in our AI infrastructure market deck, shows the share of revenue generated by each customer segment in the AI infrastructure market
Are chipmakers now designing AI hardware around memory?
Yes. Today’s accelerator roadmaps increasingly look like memory roadmaps with more compute built around them.
These jumps go far beyond ordinary specification growth. A100 offered 80 GB and about 2 TB/s. H200 raised that to 141 GB and 4.8 TB/s without changing the basic Hopper GPU architecture, and NVIDIA reported up to 1.9 times higher Llama 2 70B inference performance than H100 in selected tests. AMD’s MI355X now supplies 288 GB and 8 TB/s.
The newest generation pushes bandwidth even harder. Rubin keeps capacity at 288 GB but raises bandwidth to 22 TB/s. AMD’s MI455X reaches 432 GB and up to 23.3 TB/s, almost three times the bandwidth of MI355X while capacity rises by 50%. Reading models and caches faster has become as important as fitting more of them.
The table uses headline vendor specifications. It shows the direction of hardware design; total real-world performance also depends on software, precision and workload.
| Accelerator | HBM capacity | Peak HBM bandwidth |
|---|---|---|
| NVIDIA A100 80GB | 80 GB | About 2 TB/s |
| NVIDIA H200 | 141 GB | 4.8 TB/s |
| AMD MI355X | 288 GB | 8 TB/s |
| NVIDIA Rubin | 288 GB | 22 TB/s |
| AMD MI455X | 432 GB | Up to 23.3 TB/s |
If you want more recent data on this point, please see our latest AI infrastructure market report.
Is memory the main barrier to running AI locally?
Memory capacity is the first major barrier to local AI, and memory bandwidth decides whether a model that fits is pleasant to use.
A 70B model needs about 140 GB for BF16 weights or roughly 35 GB at four bits before cache and runtime overhead. Most consumer GPUs cannot hold the compressed version on one card. Users then have to choose a smaller model, split it across devices or place some layers in slower system RAM.
The new generation of AI desktops is built around that limitation. NVIDIA’s DGX Spark has 128 GB of unified memory and supports inference on models up to 200 billion parameters, although its documented 273 GB/s bandwidth is far below data-center HBM. Apple’s M3 Ultra Mac Studio reaches 512 GB of unified memory and 819 GB/s, enough for Apple to claim support for models above 600 billion parameters.
Those machines reveal the difference between fitting and running fast. A heavily quantized model can sit entirely in a large unified-memory pool, yet each generated token still requires substantial traffic. Apple’s 819 GB/s is impressive for a desktop but remains far below the 8 to 23 TB/s range of current data-center accelerators.
Smaller models may offer a better local experience anyway. Distillation, task-specific tuning and retrieval can outperform a much larger general model for speech, coding, document search or private assistants. Even then, extra memory creates room for better models, longer context and several tools running together.
Local AI is exposing the memory wall to ordinary buyers. The most useful machines increasingly compete on usable shared memory and bandwidth, rather than GPU compute alone.
This chart, included in our AI infrastructure market deck, shows how GPU cloud infrastructure technology has evolved over time
Will new AI architectures make the memory wall disappear?
New architectures are softening the memory wall; none has yet matched frontier Transformers across every important task while keeping memory use flat.
State-space models offer the cleanest alternative. Mamba-style systems process sequences recurrently and can maintain a fixed-size state during generation instead of a KV cache that grows with every token. Mamba-3, released in 2026, improves retrieval and state tracking while using half the state size of Mamba-2 in some comparisons. Its researchers still acknowledge the central weakness of this family: earlier efficient models often lost quality or failed to deliver their theoretical hardware gains in practice.
Hybrid attention may move faster into production. Kimi Linear combines three linear-attention layers with one full-attention layer. Its authors report up to 75% lower KV-cache use and up to six times higher decoding throughput at a one-million-token context, while matching or beating their full-attention baseline in the reported tests.
Other teams are keeping Transformers and becoming selective about memory. DeepSeek-V3.2 uses sparse attention for long context. IndexMem learns which cache entries deserve to survive. Tiered-cache systems keep recent or valuable state in fast memory and move the rest elsewhere. Each approach accepts a different risk: lost details, retrieval mistakes, extra compute or more complicated scheduling.
We are likely to end up with several architectures. Full attention will remain valuable when exact access to a long history matters. Recurrent and hybrid models will appeal to agents, streaming data and very long sessions. The fact that memory pressure is already changing model design says more about the bottleneck than another isolated benchmark.
Is memory becoming the bottleneck for AI?
Yes, memory is becoming the bottleneck for AI, especially in inference, long-context workloads and systems serving many users at once.
Three trends now point in the same direction. Workloads retain more state through reasoning, agents and million-token contexts. Hardware roadmaps are increasing memory bandwidth at an extraordinary pace. Software research is concentrating on KV compression, eviction, paging, offloading and alternative attention because data movement is holding back available compute.
Frontier training remains a more complicated case. Power, networking, data quality or synchronization can dominate a particular run, and sharding often turns a local capacity problem into a communication problem. Memory still shapes the system, even when another constraint becomes the immediate limit.
For inference, the judgment is much firmer. Decode is frequently bandwidth-bound. Context state grows with every active token. Quantization savings are often reinvested in larger models or more users. As seen above, the latest Rubin and MI455X designs reach roughly 22 to 23.3 TB/s, yet the industry is already building cache tiers and new architectures to reduce the traffic placed on that bandwidth.
Memory is also becoming broader than HBM capacity. The real constraint covers HBM bandwidth, KV-cache management, GPU-to-GPU links, CPU offload, flash storage, packaging capacity and the supply of advanced memory stacks. All of these determine whether processors spend their time calculating or waiting.
Our judgment is that the statement is mostly true today and will probably become more true as inference accounts for more AI activity. The next gains will depend less on theoretical FLOPS alone and more on how well systems fit, move, compress and reuse data.
If you want more recent data on this point, please see our latest AI infrastructure market report.
In our AI infrastructure market deck, we identify pain points entrepreneurs should prioritize
OUR METHODOLOGY
This analysis tests whether memory is becoming a defining bottleneck for AI by looking for observable technical and commercial consequences rather than relying on industry sentiment. We examine changing workloads, training and inference behavior, accelerator design, model architecture, memory-management software, system infrastructure, supply conditions and local deployment.
We separate four different constraints: capacity, bandwidth, latency, and supply and cost. A model that does not fit, a processor waiting for data, an offloaded cache creating transfer delays, and a server delayed by scarce HBM are different problems, even though people often describe all four as a memory bottleneck.
We also distinguish direct constraints from displaced ones. When sharding reduces local memory use but increases network traffic, or when CPU and SSD offload add capacity but create transfer delays, we treat the bottleneck as moving through the system rather than disappearing.
The hardware comparison uses successive accelerator generations to see where vendors are concentrating their engineering gains. Headline capacity and bandwidth figures come from NVIDIA’s documentation for A100, H200 and Rubin, and AMD’s documentation for MI355X and MI455X. These specifications show the direction of system design, while real performance still depends on precision, software, batching and workload.
The training analysis uses transparent per-parameter estimates and the design trade-offs documented in ZeRO, ZeRO++ and PyTorch Fully Sharded Data Parallel. The inference analysis draws on the different arithmetic intensity of prefill and decode, along with research and vendor material describing weight traffic, KV-cache growth and memory-bandwidth limits.
The calculations for model weights and KV cache are order-of-magnitude checks rather than performance forecasts. They are intended to show when compressed weights, optimizer states or active context become large enough to alter hardware requirements, concurrency or system design.
We assess software improvements by asking whether they reduce data movement, allocation waste or stored precision. FlashAttention, PagedAttention and vLLM are used as established examples, while newer work on KV compression, eviction, optimizer memory and tiered caching shows where current research effort is moving.
The supply-chain section relies on company disclosures rather than demand estimates alone. Micron’s fiscal 2026 update and SK hynix’s HBM4E sampling provide evidence on contracted supply, manufacturing pressure and the speed of product transitions. CXL documentation is used to assess how pooled and expanded memory may absorb colder data outside HBM.
Key sources include NVIDIA’s Rubin architecture overview, NVIDIA’s Vera Rubin platform specifications, AMD’s MI455X specifications, AMD’s MI355X specifications, OpenAI’s GPT-5.4 announcement, OpenAI’s GPT-5.4 API documentation, the DeepSeek-V3 technical report, PyTorch FSDP documentation, the ZeRO++ paper, the FlashAttention paper, the PagedAttention and vLLM paper, the CXL 4.0 specification overview, Micron’s fiscal 2026 update, SK hynix’s HBM4E announcement, NVIDIA DGX Spark hardware documentation, and Apple’s Mac Studio specifications.
The conclusion is weighted by workload. The evidence is strongest for autoregressive inference, long-context processing and high-concurrency serving. Frontier training remains more conditional because communication, synchronization, power and data can become the immediate limit, even when memory determines how the system must be organized.
This chart, included in our AI infrastructure market deck, shows the share of revenue by region across Europe, Asia, North America, Africa, and South America in the AI infrastructure market
Related blog posts
- AI chip memory: which startup is ahead?
- Who is the best memory chipmaker now?
- Do AI data centers need fewer GPUs and more memory?
- Is GPU memory more important than GPU speed now?
