How large is the AI inference chips market today?

How large is the AI inference chips market today?

The AI inference chips market is already worth roughly $180 billion to $220 billion a year by our estimate, if we count the economic value of data-center accelerators actually being used for inference. A stricter view based mainly on merchant chip revenue comes out lower, at roughly $150 billion to $190 billion.

There is no official industry number we can simply quote. NVIDIA sells Blackwell GPUs for training, post-training and inference. AMD does the same with Instinct. Google, Amazon, Microsoft and Meta increasingly use their own chips, so some of the economic value never appears as normal semiconductor revenue.

We therefore have to rebuild the market from the pieces we can observe.

NVIDIA is the biggest one by far. Its latest quarterly Data Center revenue reached $89 billion, up 117% from a year earlier. In the previous quarter, when NVIDIA still disclosed Data Center compute separately, compute represented about four-fifths of Data Center revenue. Applying roughly that mix today gives us more than $70 billion of accelerator-related compute revenue in a single quarter.

The last time NVIDIA gave a clear workload split, it said inference represented more than 40% of Data Center revenue over the previous four quarters. Since then, Blackwell has moved further into reasoning, token generation and large-scale serving. Using a 40% to 50% inference range therefore gives us roughly $115 billion to $145 billion of annualized NVIDIA inference-related accelerator revenue.

Broadcom adds another large piece. Its latest quarterly AI semiconductor revenue reached $16.7 billion, up 221% year over year. During the earnings call, Broadcom said custom XPUs represented 73% of that revenue, or about $12.2 billion in one quarter. Google’s latest TPU generations are heavily focused on inference, OpenAI’s first Broadcom-supported custom accelerator targets inference, and Meta’s upcoming MTIA production is aimed at inference and recommendation.

AMD, Huawei, Amazon’s internally consumed Trainium chips and several smaller architectures then push the market higher.

That gets us to a current market of roughly $180 billion to $220 billion. Even allowing for a wide error margin, the important part is the order of magnitude: AI inference silicon is already around a $200 billion market.

Current component	Approximate annual inference-chip value
NVIDIA data-center accelerators	$115B–$145B
Broadcom-supported custom XPUs	$30B–$40B
AMD Instinct	$4B–$8B
Chinese accelerators	$8B–$15B
AWS captive AI silicon not otherwise counted	$5B–$10B
Other specialist accelerators	$1B–$4B
Estimated total	$180B–$220B

What should we actually count as an AI inference chip?

For this market, we should count data-center processors whose job is to run AI models after or between training cycles, including chips that can also be used for training.

Requiring a chip to be dedicated only to inference would make the market definition almost useless. NVIDIA Blackwell and AMD Instinct can perform both jobs, yet a huge share of their deployed capacity now serves production models.

Our definition therefore includes NVIDIA Blackwell, Hopper and Rubin GPUs when they are used for inference; AMD Instinct accelerators; Google TPUs; Amazon Trainium; Microsoft Maia; Meta MTIA; Huawei Ascend; Cambricon accelerators; and smaller specialist architectures. Different chip types sit under one definition here, and we break down which segments and regions lead in our report on AI chips.

Recommendation inference belongs here too. Meta built early MTIA chips largely for ranking and recommendation models used across products such as advertising. Google has spent years running enormous recommendation, ranking and search models. Generative AI attracts more attention today, but large-scale inference existed long before ChatGPT.

We exclude networking chips, optical components, HBM, storage and ordinary server CPUs from the core estimate. They are essential to an inference system, but counting them would turn a chip-market estimate into an entire AI infrastructure estimate.

We also keep smartphone, PC and automotive AI processors separate. They perform real inference, but including every Snapdragon, Apple Silicon, Ryzen AI or automotive SoC would create a much broader market with very different economics.

Why is the AI inference chip market so hard to measure?

The AI inference chip market is hard to measure because the same accelerator can train a model one day and serve users the next, while hyperscalers increasingly hide chip economics inside their own cloud businesses.

An NVIDIA Blackwell GPU bought by Microsoft could handle post-training, synthetic-data generation and production inference during the same year. NVIDIA records one hardware sale. It does not report what percentage of that GPU’s useful life eventually goes to each workload.

Custom silicon creates another problem. Google does not buy a complete TPU from Broadcom in the same way an enterprise buys an NVIDIA GPU. Broadcom helps design and manufacture the accelerator, Google deploys it internally, and the economic return eventually appears through Google Cloud, Search, Gemini and other products.

Amazon shows how large this accounting gap can become. Amazon recently said its custom-chip business exceeded a $25 billion annual revenue run rate, covering Trainium, Graviton and other silicon. Andy Jassy has also said that if Amazon’s internally consumed chips were treated like products made by an independent semiconductor company and sold back to AWS, their annualized value would be around $50 billion.

So two perfectly reasonable approaches can produce different market sizes. Merchant chip revenue is easier to observe, while economic chip value captures more of what hyperscalers are actually deploying.

That is why we use a range rather than pretending the market can be measured to the nearest billion dollars.

Is NVIDIA alone already making more than $100 billion from AI inference?

Yes. NVIDIA is almost certainly running above $100 billion a year in accelerator revenue tied to AI inference today.

The calculation starts with NVIDIA’s current scale. Data Center revenue reached $89 billion in the latest quarter, compared with $41 billion a year earlier. In the preceding quarter, Data Center compute generated $60.4 billion out of $75.2 billion of total Data Center revenue.

If compute still represents roughly four-fifths of Data Center sales, current quarterly compute revenue is somewhere above $70 billion.

NVIDIA previously said inference represented more than 40% of Data Center revenue. Applying only 40% to the current compute base already produces more than $110 billion of annualized inference-related revenue. At 50%, the figure moves above $140 billion.

A 40% assumption does not look aggressive anymore. NVIDIA has spent much of the Blackwell and Rubin product cycle talking about reasoning, test-time compute, token throughput and inference economics. Dynamo was built specifically to improve distributed inference serving, while Rubin has been designed around workloads where tokens per watt and tokens per dollar increasingly matter.

We cannot know the exact split without internal NVIDIA workload data. We can be much more confident about the scale: inference has already become a nine-figure-billion-dollar business for NVIDIA alone.

Are GPUs still winning the AI inference chip market?

Yes. GPUs still dominate commercial AI inference today, although custom accelerators are taking a meaningful share at the largest cloud companies.

GPUs have one big advantage: customers do not need to know exactly what their AI workload will look like three years from now.

The same NVIDIA cluster can handle pre-training, reinforcement learning, synthetic-data generation, fine-tuning and production inference. When model architectures change, the hardware can usually be reused. CUDA and NVIDIA’s software stack also make it easier for developers to move between workloads. Flexibility across workloads is a real edge, and we weigh the likely future winners in our AI chips report.

That flexibility is worth a lot when companies are spending billions on infrastructure while models are still changing quickly.

Custom ASICs become more attractive when the workload is huge and predictable. Google can justify designing a TPU because it knows it will run enormous quantities of inference. The same logic increasingly applies to Amazon, Meta, Microsoft and OpenAI.

So GPUs currently win most of the broad commercial market, while custom accelerators are carving out some of the very largest and most repetitive inference workloads.

Are custom AI chips getting big enough to threaten NVIDIA?

Yes. Custom AI chips are now a real tens-of-billions-of-dollars market, and their growth has become the clearest competitive threat to NVIDIA in inference.

Broadcom gives us the best financial window into this shift. Its AI semiconductor revenue reached $16.7 billion in the latest quarter, up 221% year over year, and 73% came from custom XPUs. XPU shipments were more than 3.5 times higher than a year earlier.

The customer list makes the inference angle even clearer. Broadcom is shipping Google’s Ironwood TPU and has started production of TPU v8i, which it describes as optimized for inference. It has also begun shipping OpenAI’s first custom accelerator, Jalapeño, for inference workloads. Meta’s upcoming MTIA production is focused on inference and recommendation.

Microsoft is pushing in the same direction with Maia 200. Microsoft calls Maia 200 an inference accelerator and says it delivers around 30% better performance per dollar than the latest-generation hardware previously in its fleet. By the middle of the year, Maia 200 was already supporting OpenAI and Microsoft’s own models.

Amazon has gone even further. Trainium was originally associated mostly with training, but Amazon now says Bedrock runs most of its inference on Trainium. The custom-chip business has exceeded a $25 billion annual revenue run rate, and Trainium demand has become large enough for both Anthropic and OpenAI to make multi-gigawatt commitments.

These companies are approaching the problem differently, but the pattern is now hard to miss. The biggest buyers of NVIDIA GPUs are also spending heavily to reduce how dependent they are on NVIDIA for inference.

Custom accelerator	Main company or customer	Current inference role
Google Ironwood / TPU v8i	Google	Large-scale inference
Amazon Trainium	AWS	Training + major Bedrock inference
Microsoft Maia 200	Microsoft	Purpose-built inference
Meta MTIA	Meta	Recommendation + AI inference
OpenAI Jalapeño	OpenAI	Inference
Huawei Ascend	Chinese clouds and enterprises	Training + inference
NVIDIA Blackwell / Rubin	Broad market	Training + inference
AMD Instinct	Broad market	Training + inference

How big has Google’s TPU inference operation become?

Google’s TPU operation is already one of the world’s biggest custom AI-chip deployments, even though Alphabet never tells investors exactly how much TPU capacity is used for inference.

The architecture itself gives us a strong clue. Google introduced Ironwood with inference as a central use case. Broadcom has since started shipping TPU v8i, saying the new generation has more memory and bandwidth and is also optimized for inference workloads.

The scale is large enough to change Broadcom’s financial results. Broadcom delivered Ironwood in high volume to Google and Anthropic during its latest reported quarter, while TPU v8i entered production at the same time.

Google is therefore well past the experimental custom-chip stage. TPUs now support Gemini, Google Cloud customers and internal AI workloads across one of the world’s largest computing estates.

We should still avoid pretending that every TPU is an inference chip. Google also trains models on TPUs. The important point for our market estimate is simpler: Google now runs enough inference on custom silicon for it to represent a multibillion-dollar annual pool of accelerator value outside NVIDIA’s reported revenue.

Is Amazon Trainium really becoming an inference chip?

Yes. Trainium has become a major inference platform inside AWS despite a name that still makes it sound like a training-only product.

The clearest evidence comes from Bedrock. Amazon has said that most inference on Bedrock runs on Trainium, and Bedrock is now used by hundreds of thousands of customers. Earlier in the year, Amazon disclosed 1.4 million Trainium2 chips deployed, while more recent disclosures show Trainium3 moving into production and most near-term supply already committed.

Amazon’s economics explain why the company is pushing so hard. Andy Jassy says Trainium2 offered around 30% better price-performance than comparable GPUs, with Trainium3 improving another 30% to 40%. Amazon expects its own chips to save tens of billions of dollars in annual capital expenditure at scale and improve AWS operating margins compared with relying entirely on third-party accelerators for inference.

Trainium still handles plenty of training. Anthropic’s enormous Project Rainier cluster uses Trainium, and future commitments from Anthropic and OpenAI cover a mix of advanced AI workloads.

But Amazon has already proved something important for the inference market: one of the world’s largest public clouds can move a majority of a major model-serving platform onto internally designed accelerators.

How important is AMD in AI inference today?

AMD has become a credible inference supplier, but its current revenue still sits far below NVIDIA.

AMD’s latest Data Center revenue reached $6.7 billion for the quarter, up 107% year over year. That segment includes both EPYC server CPUs and Instinct accelerators, so we cannot simply treat the whole number as AI-chip revenue. Data Center revenue up 107% in a year is one reason we look at how fast the market grows, and why, in our report covering AI chips.

Instinct is nevertheless showing up in more production AI deployments. AMD has announced relationships or deployments involving Microsoft, Meta, OpenAI, Oracle and other large AI customers, while its Helios rack-scale platform increasingly emphasizes inference throughput and tokens per dollar.

The scale gap remains huge. NVIDIA generated $89 billion of Data Center revenue in one quarter, more than thirteen times AMD’s entire Data Center segment.

Our current estimate puts AMD inference accelerator revenue in the mid-single-digit billions annually. That is still small beside NVIDIA, but large enough for AMD to matter as the main merchant alternative to CUDA-based GPUs.

AMD’s bigger opportunity comes later. If customers become comfortable running the same production models across NVIDIA and AMD hardware, the inference market becomes meaningfully less captive to one supplier.

Is China already a big AI inference chip market?

Yes. China is already a multibillion-dollar inference accelerator market, although the available data is much less clean than what we get from NVIDIA, AMD or Broadcom.

Export restrictions have pushed Chinese cloud companies and AI developers toward Huawei Ascend, Cambricon and internally designed accelerators.

Huawei appears to dominate the domestic high-end market. Industry estimates put Chinese high-end cloud AI accelerator shipments above two million units this year, with Huawei responsible for more than half of domestic shipments.

Cambricon gives us a useful public-company reference point. Its first-half revenue reached roughly $890 million and more than doubled year over year. Alibaba is also deploying its own T-Head chips, while several Chinese cloud providers are building systems around domestic accelerators.

Inference demand is large enough for Huawei to discuss Chinese token consumption in the hundreds of trillions per day. We should treat company estimates of industry-wide token usage cautiously, but they fit the broader picture coming from accelerator shipments and cloud spending.

We currently assign roughly $8 billion to $15 billion of annual inference-chip value to Chinese accelerators. The confidence interval is wider than for the U.S. market, but excluding China altogether would now produce a clearly incomplete global estimate.

Is generative AI now the whole inference-chip market?

No. Generative AI is growing fastest, but recommendation, advertising, search, ranking and computer-vision inference still consume a huge amount of compute today.

Meta is the easiest example. Its early MTIA accelerators were designed largely for ranking and recommendation workloads behind products such as advertising. Those workloads run constantly across applications serving billions of users.

Google has spent years running machine-learning inference inside Search, advertising and recommendation systems. Amazon uses inference throughout commerce as well as AWS. ByteDance, Alibaba and other internet platforms do the same.

Generative AI changes the economics because one user action can now consume far more computation. A recommendation model may return a relatively compact prediction. A reasoning model can generate thousands of tokens, examine intermediate results, call tools and keep working.

Agents push that further. One human request can trigger several model calls, searches and follow-up steps.

So older inference workloads still matter, but generative AI is making the amount of compute attached to each user interaction much less predictable and potentially much larger.

Is AI inference already bigger than AI training?

Probably not by accelerator revenue yet, but inference is now close enough that we should expect the balance to keep moving toward inference.

The boundary has become messy. Frontier-model training still consumes enormous clusters, while reinforcement learning, synthetic-data generation and long post-training runs sit somewhere between the old categories of training and inference. With training and inference blurring together, sizing the market takes judgment, and we estimate how big it really is today in our report covering AI chips.

Inference grows differently because successful models keep consuming compute after development is finished. Every new user, query, reasoning step and agent action creates another serving workload.

Training is more episodic. A company trains or updates a model, spends heavily for a period, then moves into serving it.

Amazon has openly said it expects inference to represent the overwhelming majority of future AI cost because models are trained periodically but queried continuously. Microsoft is designing Maia specifically around large-scale inference economics. Google’s recent TPU generations and Meta’s MTIA program point in the same direction.

We would therefore describe the market as being in the middle of the shift. Training still matters enormously today, while inference increasingly looks like the larger long-term source of recurring chip demand.

Demand driver	Effect on training	Effect on inference
Larger frontier models	Very high	High
More AI users	Limited direct effect	Very high
Longer reasoning	Medium	Very high
AI agents	High in post-training	Very high
Video generation	High	Very high
Enterprise AI adoption	Medium	High
More model releases	Periodic compute demand	Recurring compute demand

Why is AI inference chip demand growing so fast right now?

AI inference chip demand is growing so fast because more people are using AI, users are making more model calls, and each call can require much more computation than it did a few years ago.

The first generative-AI boom mostly increased the number of people using large models.

Usage has since spread into coding, search, office software, customer support, advertising, design and enterprise applications. That raises the number of queries per user.

Reasoning adds another multiplier. Models can spend far more compute working through a problem before producing an answer.

Agents then combine several inference steps into one task. An agent may interpret a request, search for information, call another model, use a tool, inspect the result and try again.

The multiplication can become very large. If a service keeps the same number of users but doubles the number of AI calls per user and doubles the compute used per call, inference work roughly quadruples.

That helps explain why chip companies now talk so much about tokens per dollar, tokens per watt and token throughput. Raw model adoption no longer tells us enough about infrastructure demand.

If inference keeps getting cheaper, could the chip market actually shrink?

Not anytime soon, in our view. Cheaper inference is currently encouraging people to use much more AI, so efficiency gains are being absorbed by rising demand.

The improvement in cost is real. Microsoft says Maia 200 gives it around 30% better performance per dollar than the latest-generation hardware previously in its fleet. Amazon says Trainium2 offered about 30% better price-performance than comparable GPUs and Trainium3 improves substantially again. NVIDIA makes the same argument around every new GPU generation.

If AI usage were fixed, those efficiency gains would reduce the number of accelerators required.

AI usage is nowhere near fixed.

Cheaper inference lets developers run larger models, increase context windows, add reasoning, process more video, make agents work for longer and put AI into products where the economics previously looked unattractive.

The hyperscaler spending pattern supports that view. Hardware efficiency has improved dramatically while AI infrastructure investment has continued to rise.

Cheaper inference becomes a real threat to chip revenue only if efficiency improves faster than demand expands for a sustained period. We do not see evidence of that today.

Could custom chips eventually take most AI inference away from NVIDIA?

Custom chips could eventually take a large share of hyperscale inference from NVIDIA, but NVIDIA should remain extremely difficult to displace across the broader market.

Inference is well suited to ASICs when a company runs huge, repetitive workloads. Saving even 20% or 30% per token becomes enormous when the chip runs continuously across hundreds of thousands of servers. Saving 20% per token on huge workloads pays for custom chips; we compare which business models are winning in our report on AI chips.

That explains Google TPU, Amazon Trainium, Microsoft Maia and Meta MTIA.

The economics break down for smaller buyers. Designing an accelerator requires advanced semiconductor teams, packaging expertise, compilers, networking, software support and enough volume to spread the development cost across millions of workloads.

Most enterprises, AI startups, sovereign clouds and regional infrastructure companies will never reach that scale.

They still need accelerators they can buy from somebody else, install quickly and use across different models.

NVIDIA can therefore lose share among its biggest hyperscaler customers while continuing to grow across the market as a whole. The more realistic risk is gradual share erosion, especially in predictable inference workloads, rather than a sudden collapse in NVIDIA demand.

How concentrated is the AI inference chip market today?

AI inference chips remain extremely concentrated today: NVIDIA probably captures somewhere around 55% to 75% of the market’s economic value, while a small group of custom-chip programs accounts for most of the rest.

Broadcom-supported XPUs form the clearest second block. Google, Meta, OpenAI and other hyperscale customers are now pushing those programs into high-volume production.

Amazon occupies a slightly different category because much of Trainium’s value stays inside AWS. Huawei leads an increasingly separate Chinese ecosystem. AMD is currently the biggest merchant GPU alternative to NVIDIA but remains much smaller.

Dedicated inference companies such as Cerebras and other specialist architectures add interesting technology but little market share so far. Even hundreds of millions of dollars in specialist-chip revenue would barely move a market already measured in the hundreds of billions.

Their influence can still exceed their revenue. Specialist architectures have pushed the industry toward lower-latency serving, deterministic execution and better tokens-per-dollar economics. Larger chip companies are responding to those priorities even when the startups themselves remain small.

Supplier or architecture group	Approximate inference position today
NVIDIA	~55–75%
Broadcom-supported custom XPUs	~15–20%
Chinese accelerators	~4–8%
AMD	~2–4%
AWS captive silicon not otherwise captured	~2–5%
Other architectures	<3%

Should AI PCs and smartphone chips count in the inference-chip market?

No. We should keep AI PCs, smartphones and automotive processors outside the core inference-chip estimate because including them would blur the market more than it would clarify it.

Apple, Qualcomm, MediaTek, AMD and Intel now ship processors with increasingly capable neural-processing units. Those chips genuinely run AI inference on the device.

But a smartphone SoC would still exist without generative AI. Only part of its die, cost and customer value comes from the NPU.

Counting the entire processor as AI inference revenue would therefore inflate the market.

That is why some published “AI semiconductor” estimates reach much higher figures than ours. They often include CPUs, edge processors, smartphone chips and other silicon with AI capabilities.

For the question we are answering here, data-center accelerators give us the cleaner picture of how much money is actually being spent on dedicated AI inference compute.

How fast is the AI inference chip market growing today?

The AI inference chip market is growing extremely fast right now, with several major suppliers still posting triple-digit increases at the same time.

NVIDIA’s Data Center revenue rose 117% year over year in its latest quarter. Data Center revenue up 117% in a year shows how quickly things shift, and we look back over the past twelve months in our report covering AI chips.

Broadcom’s AI semiconductor revenue rose 221%, while custom XPU shipments increased more than 3.5 times.

AMD’s Data Center revenue grew 107%.

Amazon says its chip business is growing at triple-digit rates and has already crossed a $25 billion annual revenue run rate.

Cambricon’s revenue has also more than doubled, while domestic Chinese accelerator shipments continue to ramp quickly.

Those numbers cover very different businesses: merchant GPUs, custom ASICs, captive cloud silicon and Chinese domestic accelerators. The fact that all of them are growing strongly at the same time tells us demand is still expanding fast enough for new suppliers to gain business without forcing NVIDIA to shrink.

We should not extrapolate triple-digit growth indefinitely. A market approaching $200 billion cannot double every year for long.

But the current evidence still looks much closer to an industry scaling rapidly than one approaching saturation.

How much bigger could the AI inference chip market get?

AI inference chips can realistically become a several-hundred-billion-dollar annual market before the end of the decade, even if current growth slows sharply.

The reason is the recurring nature of inference.

Training demand depends heavily on how many models companies decide to build and update. Inference demand follows how much those models are actually used.

If AI keeps spreading through search, coding, office software, advertising, customer support, autonomous systems, video creation and agents, the same models can generate compute demand every minute of every day.

Efficiency will slow the revenue growth. Rubin, Maia, Trainium, TPUs and future AMD accelerators should all produce far more inference for each dollar and watt. Better caching, quantization, speculative decoding and serving software will help too.

That means ten times more tokens will not require ten times more semiconductor spending.

Starting from roughly $200 billion still leaves a huge path upward. At 20% annual growth, a $200 billion market becomes about $500 billion after five years. At 30%, it moves well above $700 billion.

We do not need today’s triple-digit vendor growth to continue for inference silicon to become one of the semiconductor industry’s largest categories.

So how large is the AI inference chip market really?

The AI inference chip market is roughly $200 billion a year today, with $180 billion to $220 billion as our best range for data-center accelerator silicon economically tied to inference.

A narrower merchant-revenue view comes out around $150 billion to $190 billion because it misses part of the value created by chips that Google, Amazon, Microsoft and other hyperscalers consume internally.

NVIDIA alone probably accounts for more than $100 billion of annual inference-related accelerator revenue. Custom accelerators designed with companies such as Broadcom now add several tens of billions. AMD, Huawei, Amazon Trainium and the rest of the market fill in the remaining gap.

That makes some older market estimates of only a few tens of billions increasingly difficult to reconcile with what chip companies and cloud providers are reporting now.

The bigger shift is what inference has become economically. AI models keep consuming compute every time somebody uses them, and reasoning systems or agents can make each request much heavier than before.

Our estimate therefore lands around $200 billion today, with inference already large enough to influence which chips hyperscalers design, where data-center capital goes and how the entire AI semiconductor market competes.