Will inference become bigger than AI training?

What does “bigger than AI training” actually mean?

AI inference is becoming bigger than AI training in the sense that matters most for the infrastructure market: total compute used, money spent and electricity consumed over time.

There is still an important measurement problem. One frontier training run can use an absurd amount of computing power, while one inference request can be tiny. But training happens in large bursts. Inference keeps running after the model launches, potentially millions or billions of times a day.

We therefore looked at four measures that can give different answers: compute, cloud spending, data-center power and the size of individual workloads.

The latest numbers are already leaning toward inference. Gartner currently expects 55% of the $42 billion AI-optimized infrastructure-as-a-service market in 2026 to support inference, rising to 59% in 2027. Deloitte expects inference to account for roughly two-thirds of AI compute this year. McKinsey's physical data-center view is more conservative: it put inference at 20.9 GW in 2025 versus 23.1 GW for training, and its October 2026 chart shows the two level at about 29% each of total data-center demand in 2026.

Those estimates are close enough to tell us where we are. The crossover is happening around now, although it has arrived earlier in compute and cloud spending than in physical power demand.

Measure	Where inference stands
AI compute	Probably already larger
AI cloud infrastructure spending	Already larger
Data-center power	Roughly level with training
Largest individual workload	Training remains much larger

Has AI inference already overtaken training, and on which measure?

AI inference has probably already overtaken AI training on aggregate compute and cloud spending. On data-center power, the two look roughly level. None of these figures is a measured census: they are estimates and forecasts.

Deloitte's latest technology outlook estimates that inference moved from about one-third of AI compute in 2023 to half in 2025 and roughly two-thirds today. Gartner's latest forecast reaches a similar conclusion from actual spending: 55% of AI-optimized IaaS spending is expected to go toward inference this year.

McKinsey gives us the useful counterpoint. Its data-center model estimated 23.1 GW of training demand and 20.9 GW of inference demand in 2025, so training was about 10% larger. Its October 2026 report puts the two level in 2026, each at about 29% of total data-center demand.

The interesting part comes next. McKinsey expects inference demand to grow at 35% a year through 2030, compared with 22% for training. Its model reaches 93.3 GW for inference and 62.2 GW for training by 2030, and its October 2026 report restates that as about 60% inference and 40% training.

We would therefore call the market at the crossover today, already past it on spending and compute, level on power. Anyone saying training still overwhelmingly dominates AI infrastructure is working from an outdated picture.

Estimate	Inference	Training	What it tells us
Deloitte AI compute	~67%	~33%	Inference already ahead
Gartner AI IaaS spending	55%	45%	Inference already ahead
McKinsey 2026 data-center demand	~29% of total	~29% of total	Roughly level
McKinsey 2030 power demand	93.3 GW (~60%)	62.2 GW (~40%)	Inference clearly ahead

Why is AI inference growing faster than training now?

AI inference is growing faster because people are asking models to do much more work after training ends.

The first wave of generative AI was relatively simple. A user typed a prompt, a model generated an answer and the job ended. That could involve hundreds or a few thousand tokens.

Today's products increasingly search the web, inspect files, generate code, call tools, read the result, correct mistakes and continue working. The amount of hidden computation behind one visible answer can be many times larger. Products that search, code and call tools are real; we separate what works today from what is still hype in our data center market report.

Microsoft's numbers show how seriously this is affecting infrastructure. Earlier in 2026, the company said it improved inference throughput for its most-used Copilot models by 40%, while adding another gigawatt of capacity and describing AI demand as capacity-constrained. In October 2026, Microsoft reported Azure growth of 48%.

Amazon is seeing the same change at larger commercial scale. Bedrock, its inference service, has hundreds of thousands of customers, and Amazon says most Bedrock inference runs on Trainium. In its October 2026 results, Amazon said total Bedrock token volume grew nearly 17 times year over year and Fortune 500 spending on it nearly 7 times. Google said its model APIs now process nearly 32 billion tokens per minute, more than four times a year earlier.

So inference demand is no longer mainly about more people opening chatbots. Existing users and applications are simply doing more computing every time they interact with AI.

Are reasoning models making AI inference much more expensive?

Reasoning models are pushing AI inference higher because they can spend far more compute solving a problem before giving the user an answer.

With older chatbots, inference was mostly about producing the next token until the response finished. Reasoning systems can keep working internally, reconsider approaches, call tools and generate much larger amounts of intermediate computation.

That changes the economics quite dramatically. A short final answer can hide a large inference job.

Coding makes this easy to see. A basic coding assistant might generate one function. A coding agent can read a repository, search through files, modify code, run tests, inspect errors, try another solution and repeat that cycle. The user may end up receiving a 30-line patch after the model processed hundreds of thousands of tokens.

Nvidia has increasingly designed its systems around this exact problem. Its Rubin platform is being marketed around a reduction of up to 10 times in inference token cost compared with Blackwell on selected workloads. Nvidia would not be making token economics one of its central hardware metrics if inference still behaved like the lightweight chatbot workload of a few years ago.

Reasoning gives the industry another scaling lever. Better answers can now come from spending more compute during deployment, not only during training.

Will AI agents make inference even bigger than reasoning models?

AI agents could push inference much higher because one human request can trigger dozens or hundreds of model actions.

Imagine asking a chatbot for three hotels in Tokyo. That is a small inference workload. Now imagine asking an agent to plan the entire trip. The agent might search flights, compare hotels, check neighborhoods, inspect train times, change the itinerary after discovering a conflict and return later to watch prices.

One user instruction has turned into a chain of model calls.

Software engineering already gives us a working version of this pattern. Coding agents can operate for long periods, repeatedly reading files, calling terminals and testing their own work. Parallel agents can increase consumption again by exploring several approaches at once.

Amazon has even started talking about the extra CPU demand created by agentic AI. In April 2026, the company disclosed that Meta committed to tens of millions of Graviton cores partly for CPU-intensive work behind real-time reasoning, code generation and multi-step agent orchestration.

This creates a much bigger ceiling than human prompting alone. People can only type so quickly. Software can keep calling models all day. IDC forecast in October 2026 that there will be 2.5 billion AI agents by 2030; that is a forecast of agents, not a measure of compute.

Could cheaper AI inference actually reduce total compute demand?

Cheaper AI inference is unlikely to reduce total compute demand anytime soon because usage is growing faster than the cost per task is falling.

The efficiency gains are already huge. Microsoft improved throughput on major Copilot models by 40% earlier this year. Amazon says Trainium2 offers roughly 30% better price-performance than comparable GPUs for its target workloads, while Trainium3 improves another 30% to 40% over Trainium2. Nvidia says Rubin can reduce inference token costs by as much as 10 times versus Blackwell on some workloads.

Yet none of these companies is talking about needing fewer data centers. Nobody is talking about needing fewer data centers, and we weigh whether the market is overbuilt or still full in our report on data centers.

Lower costs are making previously uneconomic workloads viable. If an AI coding task becomes five times cheaper, developers can run several agents in parallel. If analyzing a company's entire document archive becomes affordable, the company can stop sampling documents and process everything. If video inference costs fall enough, applications can examine every frame.

We are already seeing that effect at Amazon. Earlier in 2026, its chips business moved above a $25 billion annual revenue run rate, growing at triple-digit rates even as each chip generation cut the cost per workload. This rebound is likely, not guaranteed: if efficiency outran usage, total demand could still flatten.

Inference therefore looks more like a market where efficiency unlocks consumption than one where efficiency eliminates demand.

Does longer AI context really add much inference compute?

Longer context is quietly making AI inference heavier because models increasingly read whole working environments rather than short prompts.

Coding agents can ingest large repositories. Enterprise assistants retrieve long internal documents. Research agents accumulate histories across many searches. Multimodal systems add images, audio and eventually video to that context.

The amount of information processed can therefore be much larger than the response the user sees.

Caching prevents the cost from rising in a straight line. Providers increasingly store and reuse previously processed context instead of recomputing everything each time. Anthropic, OpenAI and other model providers now price cached input differently because repeated context has become important enough to affect inference economics.

Still, the direction is pretty clear. AI applications are moving from “answer this question” toward “understand this workspace and keep working inside it.”

That gives inference another source of growth even if the number of users eventually slows.

Could AI video and robots become bigger inference workloads than chatbots?

AI video, robotics and other continuous systems could eventually consume far more inference than text chat because they process much more information for much longer periods.

Text is remarkably compact. A chatbot can generate a useful answer with a few thousand tokens. Video generation has to create large sequences of image information while keeping movement and objects consistent across time. Video understanding works in the other direction, repeatedly processing frames rather than one short prompt.

Robotics pushes this further. A useful robot has to interpret cameras and other sensors, understand its surroundings and choose actions repeatedly while it moves. Autonomous vehicles already operate this way: inference happens continuously while the vehicle is running.

The same pattern could appear in personal AI devices. A model that occasionally answers a question creates limited demand. A model that listens, watches a screen or monitors an environment for hours creates a very different workload.

We should therefore be careful about extrapolating the future inference market from today's chatbot tokens. Text may end up being one of the cheaper forms of AI inference.

Is AI training actually slowing down?

AI training is still growing extremely fast, even as inference takes a larger share of total AI compute.

Epoch AI's research puts frontier training compute growth at roughly four to five times per year historically. Its latest power work says the largest training runs now exceed 100 MW and could require roughly 4 to 16 GW by 2030 if current scaling continues. Training runs that could need 4 to 16 GW raise hard questions; we go through the biggest unsolved problems in our report covering data centers.

Those numbers are enormous. A single frontier training operation could eventually demand several gigawatts for months.

Budgets point the same way. In October 2026, Meta reported $34.91 billion of third-quarter capital expenditures and narrowed its 2026 outlook to $135 billion to $145 billion, money it says supports both model training and consumer AI.

There are signs that the composition of model development is changing, though. More compute is going into post-training, reinforcement learning, synthetic data and reasoning rather than one giant pre-training run. Deloitte also argues that growth in traditional training compute has probably slowed from the exceptional rates seen in 2023 and 2024.

We should not confuse falling share with falling demand. Training can lose ground to inference while continuing to grow dramatically in absolute terms.

Could giant frontier training runs eventually retake the lead?

Frontier training could occasionally jump ahead after a huge new cluster comes online, but a lasting return to training dominance looks unlikely once inference spreads across enough users and applications.

The difference comes from repetition. A frontier model might be trained once over several months and then receive further post-training. After release, that same model family can serve millions of users and software agents every day.

Training also has a harder physical constraint. Thousands or hundreds of thousands of accelerators need to communicate quickly enough to behave like one machine. Epoch AI expects the largest training jobs could eventually require several gigawatts concentrated around a single workload, although distributed training across multiple sites is becoming more practical.

Inference can spread much more easily. The same model can serve requests from data centers across North America, Europe and Asia.

So a future 5 GW training run would be a spectacular infrastructure project. It would still compete against inference running continuously across an entire global fleet.

Are Amazon, Google, Microsoft and Meta building for inference-heavy AI?

Amazon, Google, Microsoft and Meta are already spending real money around inference, which is stronger evidence than another forecast. None of them publishes an inference-only hardware mix, though.

Meta is particularly clear. The company says it has deployed hundreds of thousands of MTIA chips for inference across advertising and organic content. Its next MTIA generations will cover broader workloads, but Meta says MTIA 400, 450 and 500 will primarily support generative-AI inference in production in the near term and into 2027.

Amazon gives us a second example. Bedrock runs most of its inference on Trainium, and its token volume grew nearly 17 times in a year. AWS sales rose 39% to $46.0 billion in the third quarter of 2026, and Amazon still guides to about $220 billion of 2026 cash capital expenditures. OpenAI and Anthropic have made multi-gigawatt Trainium commitments, although those span training as well as inference.

Google is the newest strong signal. Google Cloud revenue rose 103% to $30.7 billion in the third quarter of 2026, and Sundar Pichai cited strong demand for TPU systems. Those sales mix training and inference, so they are not an inference market size.

Microsoft is approaching the same problem through fleet optimization. Its 40% throughput gain on major Copilot models came while it kept adding gigawatts, and Azure grew 48% in its latest quarter.

Four companies with very different AI strategies are optimizing around the same thing: serving far more AI work at lower cost.

Company	What it is doing around inference now
Meta	Hundreds of thousands of MTIA inference chips already deployed
Amazon	Most Bedrock inference runs on Trainium; Bedrock token volume up nearly 17× in a year
Google	Cloud revenue up 103% in Q3 2026; strong TPU demand; nearly 32 billion API tokens per minute
Microsoft	Improved throughput on major Copilot inference workloads by 40%
Nvidia	Rubin targets up to 10× lower inference token cost on selected workloads

Will inference create a much bigger market for custom AI chips?

AI inference is making custom chips much more attractive because shaving a small amount off the cost of a workload becomes valuable when that workload runs billions of times.

Training changes quickly. Researchers experiment with architectures, numerical formats and communication patterns, so flexible GPUs remain extremely useful.

Production inference can be more predictable. Meta knows roughly what kinds of ranking, recommendation and generative-AI workloads its own services will run at huge scale. That makes it worthwhile to design MTIA around those workloads rather than buying only general-purpose accelerators. Custom chips for predictable workloads are one opening; we map where the biggest opportunities are in our data centers market report.

Amazon is showing how quickly that can become a serious business. Trainium2 is largely sold out, Trainium3 has been close to fully subscribed, and Trainium commitments now stretch across major AI labs and large enterprises. Meta has separately expanded its Broadcom partnership to co-develop future MTIA generations. Google reported strong TPU demand across a wide range of customers in October 2026, though it does not split those sales by workload.

This will probably be one of the biggest consequences of an inference-heavy market. A custom accelerator does not have to replace Nvidia everywhere. It only has to beat a GPU economically on a workload that its owner runs often enough.

Does more inference threaten Nvidia's AI-chip dominance?

More inference creates a real opening for Nvidia's competitors, but it is currently making Nvidia much richer at the same time.

The competitive risk comes from specialization. Google can run large workloads on TPUs, Amazon on Trainium and Meta on MTIA. Those companies control both the software and the infrastructure, so even modest savings per request can justify custom silicon.

Nvidia is responding directly rather than relying on its training lead. Fiscal 2026 Data Center revenue reached $193.7 billion, up 68%, and the company's Rubin roadmap now emphasizes inference economics heavily. Rubin promises up to a 10-fold reduction in inference token cost compared with Blackwell for selected workloads.

Nvidia has also agreed to supply Meta with millions of Blackwell and Rubin GPUs over multiple generations even as Meta scales MTIA. That combination is revealing. Custom chips can take meaningful inference workloads without removing the need for Nvidia's more flexible accelerators.

Inference probably makes the AI-chip market less uniform. Nvidia can remain the largest supplier while hyperscalers increasingly move predictable, high-volume workloads onto their own chips.

Why is inference becoming the commercially bigger side of AI?

AI inference is becoming commercially bigger because this is where companies repeatedly charge customers for using models rather than spending money to create them.

Training behaves much like extremely expensive R&D. A company can spend billions building a better model, but the financial return depends on whether customers later want to use it.

Inference is production. Every API request, Copilot task, search response, generated video or agent action can become a billable event or support a paid product.

The customer base is also much larger. Only a small number of companies will ever train frontier foundation models. Potentially millions of organizations can use them. A bank does not need to train a frontier model to run AI across fraud analysis, customer support, software development and document processing. An insurer can process millions of claims with AI without owning a giant training cluster.

Amazon's Bedrock gives us a concrete example. It has hundreds of thousands of customers, and in October 2026 Amazon said Fortune 500 spending on Bedrock grew nearly 7 times year over year. Cloud companies do not publish an inference-versus-training revenue split, so this shows direction, not size.

As AI moves from experimentation into day-to-day software, more infrastructure naturally ends up on the serving side.

Will most AI inference eventually move from data centers onto phones and PCs?

Some AI inference will move onto phones, PCs, vehicles and robots, but heavy reasoning and generative workloads still look firmly tied to data centers.

Running models locally makes sense when latency, privacy or connectivity matter. Apple, Qualcomm, AMD and Intel are all putting increasingly capable neural processors into consumer devices, and small language or vision models can already handle useful tasks without calling the cloud. Capable neural chips arriving in consumer devices raise the question of who the likely future winners are, which we weigh in our data centers report.

At the same time, frontier workloads keep getting heavier. Reasoning models need more runtime compute. Agents need long contexts and repeated tool calls. High-quality image and video generation asks for much more memory and processing than a typical phone can provide.

Deloitte currently expects most AI computation to remain on advanced, power-hungry chips located in data centers or enterprise systems even as inference becomes roughly two-thirds of AI compute.

The likely outcome is a split. Devices handle cheap and frequent tasks locally, while difficult requests move to much larger cloud models. If billions of devices begin running local models as well, edge AI could add another layer of inference demand rather than simply taking workloads away from data centers.

What could stop AI inference from becoming much bigger than training?

AI inference would fall short of today's expectations if companies fail to find enough AI applications worth running constantly.

Agents are the biggest uncertainty. They look impressive in software engineering and research, but we still do not know how reliable they will become across high-stakes business processes. If autonomous AI remains brittle, companies may use models as occasional assistants rather than continuous workers.

Economics could also bite. Inference costs can become uncomfortable when an application runs millions of long-context reasoning tasks. Companies will keep asking whether the productivity gain is worth the compute bill.

Supply is the nearer limit. IDC said in October 2026 that power, memory, packaging and site buildouts have not kept pace with AI demand, and it projects a 45 GW U.S. power shortfall by 2028. That caps how fast inference can grow, not where it ends up.

Efficiency adds another variable. Distillation, quantization, caching, sparse models, better accelerators and smarter software can cut the compute required per task dramatically. Inference demand only keeps racing ahead if new usage grows faster than those savings.

Training meanwhile has its own upside. If another round of frontier scaling produces major capability gains, AI labs could justify even larger clusters and more expensive model-development programs.

We still think inference wins, but the gap depends much more on actual AI usage than on how many GPUs the industry can manufacture.

How much bigger than AI training could inference become by 2030?

AI inference becoming roughly 1.5 to 2 times larger than training by 2030 looks like a reasonable base case today, with agents and continuous multimodal AI creating meaningful upside beyond that.

McKinsey provides the cleanest physical comparison. Its model goes from 20.9 GW of inference and 23.1 GW of training in 2025 to 93.3 GW and 62.2 GW in 2030. Its October 2026 report puts that at about 60% inference and 40% training, roughly 1.5 times larger. It is a scenario, not a measurement.

Gartner's newest forecast points in the same direction through spending. Inference represents an expected 55% of AI-optimized IaaS spending today and 59% in 2027. Its previous longer-range work had inference moving above 65% by 2029, which would imply close to twice the rest of the market.

Deloitte is already more aggressive, putting inference at about two-thirds of current AI compute.

As seen above, those forecasts come from different measurement methods, yet they all put inference on the same side of the crossover. We would need a major disappointment in AI usage or an extraordinary acceleration in frontier training for that direction to reverse.

Measure	Current / recent position	2030 direction
McKinsey data-center power	Roughly level in 2026	~1.5× training (60/40)
Gartner AI IaaS spending	55% inference	Rising share
Deloitte AI compute	~67% inference	Already around 2× training
Our base case	Around crossover now	~1.5–2× training

Will inference become bigger than AI training?

Yes. AI inference is becoming the bigger AI infrastructure workload, and on several important measures it has already crossed training.

The evidence keeps getting stronger. Gartner forecasts inference above half of AI-optimized cloud infrastructure spending in 2026. Deloitte puts it around two-thirds of AI compute. McKinsey shows the two roughly level on data-center demand in 2026 and inference at about 60% of AI demand by 2030. These are forecasts on different measures, but they point the same way.

The companies buying and designing the chips point the same way. Meta has hundreds of thousands of inference-focused MTIA accelerators deployed. Amazon says Bedrock token volume grew nearly 17 times in a year. Google Cloud revenue doubled in the third quarter of 2026 on AI infrastructure and TPU demand. Microsoft is squeezing 40% more inference throughput from major Copilot models while adding gigawatts of capacity. Nvidia's next hardware generation is being sold partly on how dramatically it can cut the cost of inference tokens. Microsoft squeezing more throughput while Nvidia sells on cheaper tokens is the kind of change we go through across the last twelve months in our data centers market report.

Training will remain enormous. Frontier runs could eventually consume several gigawatts each, and Meta alone expects $135 billion to $145 billion of 2026 capital spending that supports training as well as AI products.

Inference has the larger long-term multiplier, though. Every successful model can be used by millions of people, thousands of companies, fleets of agents and eventually machines that run AI continuously. Reasoning makes each request heavier. Agents turn one request into many. Video and robotics can turn occasional inference into continuous inference.

Our base case is therefore fairly clear: by the end of the decade, inference should consume roughly 1.5 to 2 times as much AI infrastructure as training. If autonomous agents become genuinely useful at scale, even that could look conservative.

The first AI compute race was about how much hardware companies could throw at building smarter models. The much larger recurring business is now forming around how often everyone can afford to run them.