Do you still need Nvidia GPUs for AI inference?
In our AI chip market deck, you will find everything you need to understand the market
SUMMARY
No, you do not still need Nvidia GPUs for every AI inference workload. Nvidia remains the strongest default for large, fast-changing systems, but it is no longer a technical requirement across the market.
The real dividing line is not Nvidia versus non-Nvidia. It is flexible workloads versus predictable ones. GPUs keep their advantage when models, traffic patterns and software change often; custom chips become more attractive once the job is stable and repeated at huge volume.
The biggest AI companies are diversifying rather than abandoning Nvidia. OpenAI and Anthropic are adding Cerebras, AWS Trainium, Google TPUs and custom silicon while continuing to reserve enormous amounts of Nvidia capacity.
AMD has removed the old assumption that a non-Nvidia accelerator must be dramatically slower. Its MI355X now competes closely with Nvidia on several large-model inference tests, although CUDA and Nvidia’s wider deployment ecosystem still reduce operating risk.
Memory is becoming as important as raw compute. Large-model inference often spends more time moving weights and conversation state than performing arithmetic, which gives AMD, Microsoft and other chip designers a clearer path to compete.
Cloud accelerators can already replace Nvidia when the customer accepts deeper commitment to one provider. Google TPUs and AWS Trainium or Inferentia may lower cost for suitable workloads, but some of that saving is exchanged for less portability.
Specialist systems such as Groq and Cerebras do not need to support every model to matter. They can build large businesses by owning the latency-sensitive part of inference where faster token generation directly changes the user experience.
CUDA lock-in is becoming less about whether code can run elsewhere and more about whether it runs well elsewhere. Framework portability is improving; predictable performance, tuning knowledge and operational reliability are still harder to move.
Smaller models, quantization, CPUs and device NPUs are quietly taking Nvidia out of many everyday requests. This probably expands the total amount of inference while reducing the share that must reach a data-center GPU.
The cheapest chip is not always the cheapest deployment. Hardware price can be overwhelmed by migration work, low utilization, delayed launches, missing kernels or the need to rebuild monitoring and scheduling around a new platform.
The likely end state is a mixed fleet. Nvidia can keep growing while losing share, because total inference demand is rising quickly and different stages of one request may run on GPUs, cloud accelerators, specialist processors, CPUs and local NPUs.
Why can companies now run AI inference without Nvidia?
Today, many AI inference workloads run perfectly well without Nvidia, although the most demanding frontier systems still rely heavily on its GPUs.
The clearest change is visible among the companies spending the most money on AI infrastructure. OpenAI currently plans to deploy at least 10 gigawatts of Nvidia systems, yet it is also adding 750 megawatts of Cerebras capacity and building its own inference processor with Broadcom. That custom chip, called Jalapeño, is already running test workloads and is meant to be deployed at gigawatt scale.
Anthropic has made a similar choice. Claude runs across AWS Trainium, Google TPUs and Nvidia GPUs. Anthropic says this lets it place each workload on the hardware that suits it best, rather than forcing every model and request through one platform.
These companies have enough demand to justify several hardware platforms at once. A smaller business faces a different calculation. It may be technically possible to run its model on AMD, AWS chips or a CPU, while Nvidia still offers the quickest route to reliable production.
So the useful distinction is simple. Can the model run without Nvidia? In most cases, yes. Can the company hit its speed, scale and reliability targets without spending months rebuilding its infrastructure? That answer still varies widely.
If you want more recent data on this point, please see our latest AI chip market report.
Is AI inference really one market?
AI inference covers jobs so different that a single answer about Nvidia quickly becomes misleading.
A frontier reasoning model serving millions of users needs large clusters, fast networking and enormous memory capacity. An advertising model may repeat the same ranking calculation billions of times. A private document assistant might serve ten employees from an existing server. A phone can translate speech or summarize messages locally.
Nvidia performs especially well when the model changes frequently or when one system must handle many types of work. Specialized chips become more attractive once the operator knows exactly which calculations will run, how often they will run and what level of performance users expect.
Hyperscalers are therefore building their own accelerators for predictable workloads while continuing to buy Nvidia GPUs. Nvidia can lose share in parts of inference without losing its position across the whole market.
| AI inference workload | Hardware that can make sense today | How necessary is Nvidia? |
|---|---|---|
| Frontier reasoning models | Nvidia, AMD or hyperscaler clusters | Often the safest option |
| Large open-model APIs | Nvidia, AMD or specialized inference clouds | Useful, but replaceable |
| Advertising and recommendation models | Internal ASICs, CPUs or GPUs | Frequently unnecessary |
| Stable workloads inside AWS or Google Cloud | Trainium, Inferentia or TPUs | Usually unnecessary |
| Small private enterprise models | CPUs, local GPUs or cloud accelerators | Optional |
| Phones, laptops and embedded devices | NPUs and system-on-chip accelerators | Generally irrelevant |
This market map, featured in our AI chip market deck, highlights top companies and startups in the AI chip market
Is Nvidia still the default choice for large-scale AI inference?
Nvidia still holds the default position for large-scale AI inference, and current spending suggests that position remains extremely strong.
Nvidia’s latest reported quarterly data-center revenue reached $75.2 billion, up 92% from the same period a year earlier. Data-center revenue includes training, inference and networking, so we cannot treat the entire figure as inference sales. Even with that limitation, demand almost doubling while every major cloud provider develops alternative chips is hard to dismiss.
A buyer choosing Nvidia gains access to the same broad platform across AWS, Azure, Google Cloud, Oracle, CoreWeave and many smaller providers. Engineers are more likely to have used CUDA before, and new model architectures usually receive Nvidia optimizations quickly.
Nvidia also sells much more than an individual GPU. Its package includes high-speed networking, rack-scale systems, serving libraries, monitoring tools and support from a large group of server manufacturers. Competitors may match one part of that package while asking customers to assemble more of the system themselves.
For a hyperscaler, that extra work can be worth millions or billions of dollars in future savings. For an ordinary AI company, a slightly more expensive Nvidia deployment may still be the sensible decision if it launches months sooner.
Does Nvidia still have the fastest AI inference hardware?
Nvidia still has the strongest all-round inference platform, although AMD can now match or beat it on several important workloads.
The latest MLPerf Inference benchmark added and updated tests for large language models, image generation, video generation and edge systems. The results show a much broader competitive field than a few years ago, with performance depending heavily on the model, latency target, server design and software implementation.
AMD’s MI355X matched Nvidia’s B200 in the offline Llama 2 70B test, reached 97% of its server throughput and delivered 119% of its interactive result. Against Nvidia’s newer B300, AMD reached between 92% and 104% across those same scenarios.
On the newer GPT-OSS-120B workload, AMD reported 111% of B200 offline performance and 115% of B200 server performance. The MI355X remained behind B300, reaching 91% in offline serving and 82% in the server test.
That ends the old assumption that non-Nvidia hardware must be dramatically slower. Nvidia still supports more configurations, has a deeper software ecosystem and can scale from individual GPUs to tightly connected racks without changing the basic platform.
A benchmark also measures a specific configuration rather than every production concern. A chip can win on throughput while losing on response time, power use, availability or engineering effort. Nvidia remains the best all-rounder, but the fastest option for a particular model may now come from somewhere else.
If you want more recent data on this point, please see our latest AI chip market report.
As this chart shows, and as featured in our AI chip market deck, search interest in AI chips has grown significantly
Is memory now deciding AI inference performance?
Memory capacity and bandwidth now decide many large-model inference results as much as raw compute does.
When an AI system first reads a prompt, it can process many calculations in parallel. Once it starts producing an answer, the system repeatedly moves model weights and conversation data through memory. Faster arithmetic provides limited help when the processor spends its time waiting for data.
Recent chips reveal how important that bottleneck has become. Nvidia’s Blackwell Ultra B300 carries 288 GB of high-bandwidth memory, 50% more than the original Blackwell generation. AMD’s MI355X also has 288 GB, along with 8 TB per second of memory bandwidth. Microsoft’s Maia 200 combines 216 GB at 7 TB per second with 272 MB of faster on-chip memory.
More memory can let one accelerator hold a larger share of the model. It can also support longer conversations and more simultaneous users before the system has to spread the work across additional chips.
Every extra chip adds communication, power consumption and operational complexity. A processor with better memory economics may therefore deliver cheaper inference even when its headline compute figure looks less impressive.
Nvidia competes very well on memory, but the importance of data movement gives AMD and custom-chip designers a clearer way to challenge it.
Has AMD become a serious Nvidia alternative for AI inference?
AMD is now a serious alternative for large-model inference, especially for buyers willing to test their own workloads instead of choosing hardware by reputation.
Performance has improved sharply. In one six-month comparison published with the latest MLPerf results, the MI355X produced 3.1 times the Llama 2 70B server throughput of AMD’s previous MI325X result. AMD also crossed one million tokens per second on multi-node Llama 2 70B and GPT-OSS-120B deployments.
The more convincing development is reproducibility. Nine hardware partners submitted results using AMD Instinct accelerators. Their MI355X results came within 4% of AMD’s own numbers, and some were within 1%. That reduces the risk that AMD’s performance exists only in a carefully tuned internal laboratory.
ROCm remains the weaker part of the offer. Support for PyTorch, vLLM and popular models has improved quickly, but new kernels, quantization formats and unusual architectures can still arrive first or work more smoothly on CUDA.
An established team running a well-known model can now treat AMD as a genuine production option. A small team trying to deploy an unusual model immediately may still save time with Nvidia.
| AI inference consideration | Nvidia | AMD |
|---|---|---|
| Large-model performance | Consistently strong | Now competitive on major tests |
| Memory on current flagship GPU | 288 GB on B300 | 288 GB on MI355X |
| Serving software | Most mature ecosystem | Improving rapidly through ROCm |
| Cloud and server availability | Very broad | Expanding, but still narrower |
| Support for new model architectures | Usually arrives early | Increasingly fast, less predictable |
| Migration and operating risk | Lower for most teams | Depends heavily on the workload |
This chart, featured in our AI chip market deck, shows annual VC investment in AI chip startups
Can Google TPUs and AWS chips replace Nvidia for cloud inference?
Google TPUs and AWS chips can already replace Nvidia for many cloud inference jobs, provided the workload fits the cloud’s software and operating model.
Google’s current Ironwood TPU was designed for large-scale training and inference, including model sampling and workloads dominated by token generation. A full Ironwood pod can contain 9,216 chips, giving Google a platform built for the same scale of dense and mixture-of-experts models that usually run on GPU clusters.
These TPUs already support Google’s own models, while Anthropic uses Google TPUs alongside Trainium and Nvidia GPUs for Claude. TPU inference is a live production platform, not an experimental alternative.
AWS has followed a similar route with Inferentia and Trainium. Amazon used its own chips for parts of Rufus, its shopping assistant, and reported twice the response speed with 50% lower inference costs after combining the hardware with parallel decoding. AWS also says Inferentia and Trainium now work with PyTorch, Hugging Face and vLLM.
The catch is portability. Google and AWS chips are most compelling when the customer already plans to stay inside that cloud. Moving away later may require new compilation, testing and optimization work.
For a company deeply committed to AWS or Google Cloud, those chips deserve a real benchmark. A company that expects to move across clouds may value Nvidia’s wider availability more than a possible saving on one platform.
Are Microsoft and Meta actually replacing Nvidia with their own AI chips?
Microsoft and Meta are moving meaningful inference traffic onto their own chips, while keeping Nvidia GPUs for workloads that change fastest.
Microsoft built Maia 200 specifically for large-scale inference. The company says it delivers 30% better performance per dollar than the latest systems already in its fleet. Microsoft plans to use Maia for its own models and services, including Microsoft 365 Copilot and models offered through Azure.
Meta’s MTIA chip has already moved beyond pilots. Meta says it is deployed at scale in its data centers, mainly for advertising, ranking and recommendation inference, where it provides substantial efficiency gains over external silicon.
Both cases follow the same economic logic. Microsoft and Meta own huge, recurring workloads. They can study those workloads, design hardware around the most common operations and spread the engineering cost across billions of requests.
Their internal chips also reduce dependence on an outside supplier and give them more control over future capacity. Yet neither company has stopped buying Nvidia systems. General-purpose GPUs remain useful when model architectures evolve too quickly for a narrowly optimized chip.
Custom silicon removes specific workloads from Nvidia rather than removing Nvidia from the data center.
This chart, featured in our AI chip market deck, shows how Nvidia is leading in AI chips
Can Groq and Cerebras beat Nvidia for AI inference?
Groq and Cerebras can beat conventional GPU services on response speed for selected models, but their model coverage and deployment footprint remain narrower.
Groq built its processor around predictable execution and rapid token generation. The company currently operates 13 data centers, serves more than five million developers and says its systems process trillions of tokens each week. These are company-reported figures, but the scale shows that specialized inference has moved well beyond small demonstrations.
Cerebras attacks the problem with a wafer-scale processor that keeps far more computation and memory movement inside one large piece of silicon. OpenAI has agreed to deploy 750 megawatts of Cerebras capacity for latency-sensitive inference, with the rollout continuing through 2028.
The value of these systems is easiest to see in interactive coding, research agents and long reasoning answers. A model producing 1,000 tokens per second can change how a person works with it, even when the underlying answer quality stays the same.
GPU systems usually offer a wider model catalogue and greater flexibility. Groq or Cerebras may need time to enable a new architecture, while Nvidia users can often deploy through an existing framework.
Specialized providers do not need to take over the whole market to build a large business. Capturing the workloads where every second of latency affects productivity may be enough.
If you want more recent data on this point, please see our latest AI chip market report.
Why have custom AI inference chips not replaced Nvidia faster?
Custom AI inference chips spread slowly because a cheaper processor can still create an expensive engineering project.
The chip must work with the model, compiler, memory system, networking, scheduler and monitoring tools. Engineers also have to reproduce the required output quality, latency and reliability. Any weakness in that chain can erase the hardware saving.
Stable demand makes the investment easier to justify. Meta knows that advertising and recommendation models will run continuously at enormous volume. Microsoft can place Maia underneath Copilot and Azure services. Google can design TPUs around its own model roadmap.
Most companies have less certainty. A model selected today may be replaced in six months. A new attention system, quantization method or mixture-of-experts design can change which calculations dominate the workload.
Meta already runs MTIA at scale, yet the company says custom-silicon expansion has created its own scaling challenges. Even Meta needs deep hardware and software teams to keep those accelerators efficient.
Nvidia remains attractive during periods of uncertainty because its GPUs can be reassigned. The same cluster might serve language models, generate video, fine-tune a new model or run tomorrow’s architecture. Custom chips become strongest after the workload stops moving so quickly.
This chart, featured in our AI chip market deck, shows annual funding in AI chip startups
Does CUDA still lock AI inference teams into Nvidia?
CUDA still keeps many inference teams on Nvidia, although the lock-in now comes more from performance tuning and operating experience than from basic code compatibility.
Modern serving software can run across several platforms. vLLM supports Nvidia CUDA, AMD ROCm, Intel GPUs, x86 and Arm CPUs, Apple silicon and external hardware plugins. The llama.cpp project supports Nvidia, AMD, Apple Metal, Vulkan, CPUs and hybrid CPU-GPU inference.
A company can therefore keep a similar application interface while changing the processor underneath it. That is a major improvement over the earlier CUDA era, when moving platforms could require a large rewrite.
Performance still depends on the lower layers. Attention kernels, quantization methods and distributed communication libraries behave differently across hardware. Some quantization formats work on Nvidia while remaining unavailable or less mature on other devices.
The application may start successfully on both Nvidia and AMD while delivering very different throughput, memory use or stability. Engineers then have to tune the second platform until the saving becomes real.
Software portability is increasingly common. Predictable performance portability remains harder, and that still protects Nvidia.
Can smaller AI models, CPUs and NPUs avoid Nvidia GPUs?
Smaller models, quantization, CPUs and device NPUs now remove Nvidia from a large and growing share of everyday inference.
Quantization reduces the number of bits used to store a model. The llama.cpp project supports formats from 1.5 bits through 8 bits, along with hybrid execution that keeps part of a model on the CPU when it does not fit entirely in graphics memory.
That can turn a model that once required a data-center accelerator into something that runs on a desktop, laptop or inexpensive server. The trade-off may include lower quality or slower responses, but plenty of tasks do not need frontier-model performance.
On-device inference is also becoming a normal product feature. Apple gives developers access to the local model behind Apple Intelligence through its Foundation Models framework. The model works offline and avoids a separate cloud inference charge for each request.
Qualcomm has demonstrated multi-step enterprise agents running locally on Snapdragon NPUs. These systems can read company data, generate an analysis and trigger actions without sending the entire workflow to a cloud accelerator.
A laptop NPU cannot replace a large Nvidia cluster serving millions of people. It can still prevent many small requests from reaching that cluster in the first place.
The likely result is more inference overall, spread across cloud GPUs, custom accelerators, CPUs and devices. Nvidia may keep growing while handling a smaller percentage of all AI requests.
This chart, featured in our AI chip market deck, compares the main business model options for AI accelerator chip companies
Are Nvidia GPUs still the cheapest way to serve AI models?
Nvidia GPUs are rarely the cheapest on purchase price, yet they can still deliver the lowest total cost once engineering work, utilization and deployment delays are counted.
Inference cost depends on how many acceptable outputs a system produces for each dollar. Chip price forms only one part of that calculation. Response time, batch size, power use, networking, model support and idle capacity can matter just as much.
AWS says one version of its Rufus shopping assistant achieved twice the response speed and 50% lower inference cost with Trainium and Inferentia. Microsoft reports a 30% performance-per-dollar improvement from Maia 200 over existing fleet hardware. These results come from the companies selling or operating the systems, and the workloads differ, so they are evidence of possible savings rather than a universal ranking.
Nvidia is also improving its own economics. The company says its upcoming Rubin platform can reduce inference cost per token by as much as ten times compared with Blackwell. That is a forward-looking Nvidia claim, but it shows how rapidly the cost baseline is moving even within the GPU market.
The only reliable comparison uses the buyer’s actual prompts, model, context lengths and traffic pattern. A chip that looks cheaper in a presentation can lose once engineers account for migration time or poor utilization.
Nvidia’s price premium is easiest to justify when a team values flexibility and a quick launch. Custom silicon wins more often when the workload is enormous, predictable and likely to remain stable for years.
If you want more recent data on this point, please see our latest AI chip market report.
Will reasoning models make Nvidia more necessary for AI inference?
Reasoning models will create more demand for Nvidia while also giving specialized inference chips their best opening yet.
A conventional chatbot may produce a few hundred tokens. A reasoning model can work through thousands of tokens before and during its final answer. An agent may call several models, search documents, run code and keep a long history across many steps.
That raises demand for compute, memory and networking, all areas where Nvidia performs strongly. Nvidia designed Blackwell Ultra and Rubin around long-context reasoning, low-precision computation and large connected systems. Its Rubin roadmap focuses heavily on lowering the cost of producing each token.
Reasoning also makes it more useful to split inference into separate jobs. One system can process the prompt and prepare the model state, while another generates the output tokens. Each phase has different hardware needs.
Cerebras has already gained OpenAI capacity for high-speed generation. More recently, AMD and Cerebras announced a system that combines AMD hardware for high-throughput processing with the Cerebras wafer-scale engine for low-latency output.
This kind of separation could weaken Nvidia’s control over the complete request. A company may use a GPU or TPU for one stage, a specialized processor for another and a CPU for retrieval or tool execution.
Inference will probably diversify faster than training. Training frontier models remains experimental and changes constantly, which rewards programmable GPUs. Once a model enters production, repeated patterns become easier to measure and optimize with custom hardware.
This chart, featured in our AI chip market deck, shows how revenue is split across customer segments in the AI chip market
Do you still need Nvidia GPUs for AI inference?
No. You no longer need Nvidia GPUs for every AI inference workload, but Nvidia remains the best default when flexibility, model support and deployment speed matter.
AMD now competes closely on major large-model benchmarks. Google and AWS operate mature cloud accelerators. Microsoft, Meta and OpenAI are building their own chips. Groq and Cerebras serve latency-sensitive models, while CPUs and NPUs increasingly handle smaller workloads locally.
The strongest AI companies are diversifying rather than walking away from Nvidia. OpenAI is building Jalapeño and adding Cerebras while planning at least 10 gigawatts of Nvidia infrastructure. Anthropic runs Claude on Trainium, TPUs and Nvidia GPUs. These companies expect several chip families to remain useful at the same time.
Nvidia’s position has changed in an important way. It has lost its status as a technical requirement, yet it still offers the easiest broad platform for companies that need to support changing models at scale.
A stable, high-volume workload should now be benchmarked on AMD, TPUs, AWS chips or specialized inference systems. A small local model may need no data-center GPU at all. A team deploying a new frontier model under time pressure will often find that Nvidia remains the least risky choice.
Our judgment is clear: Nvidia no longer owns AI inference, although it still leads it. Its future advantage will come from offering the most useful general platform inside increasingly mixed hardware fleets.
| Situation | Do you need Nvidia? | Current judgment |
|---|---|---|
| Small or quantized local model | No | Use CPUs, local GPUs or NPUs |
| Stable model with enormous traffic | No | Benchmark custom and cloud silicon |
| Workloads already committed to AWS or Google Cloud | No | Test Trainium, Inferentia or TPUs |
| Large open-model serving | Usually no | AMD is now a credible alternative |
| Unusual new model requiring rapid deployment | Often | Nvidia remains the safest option |
| Frontier reasoning at rack scale | Often, but rarely exclusively | Mix Nvidia with other capacity |
| Team with limited infrastructure engineers | Often | Nvidia reduces integration risk |
| Hyperscaler with predictable demand | No | A mixed fleet increasingly makes sense |
If you want more recent data on this point, please see our latest AI chip market report.
OUR METHODOLOGY
This analysis tests whether companies still need Nvidia GPUs for AI inference. We compare Nvidia with AMD accelerators, Google TPUs, AWS Trainium and Inferentia, Microsoft Maia, Meta MTIA, specialist systems from Cerebras and Groq, CPUs, local GPUs and device NPUs.
We broke the question into the factors that determine whether Nvidia is genuinely necessary: workload type, model scale, throughput, latency, memory capacity and bandwidth, software support, hardware availability, portability, engineering effort and total deployment cost.
We gave the most weight to evidence showing hardware operating under meaningful conditions. That included standardized benchmark submissions, disclosed system specifications, production deployments, customer commitments, cloud availability, framework compatibility and results tied to named workloads.
Benchmarks were used to answer narrow performance questions, not to declare a universal winner. A result for one model, server configuration or latency target does not establish which platform is best for every deployment, so benchmark results were read alongside memory, networking, supported configurations, software maturity and the work required to reproduce the result in production.
Company-reported figures were included when they described a clearly identified system, workload, configuration or deployment. Forward-looking cost and efficiency claims were treated as directional evidence rather than direct rankings, particularly when companies measured different models or operating conditions.
We paid particular attention to infrastructure choices made by OpenAI, Anthropic, Amazon, Google, Microsoft and Meta. These companies operate workloads large enough to expose differences in cost, latency, capacity and reliability that may remain invisible in a small trial.
No single data point determined the conclusion. We separated alternatives that can replace Nvidia for a defined workload from platforms capable of supporting a wider and more changeable inference environment, then combined those findings into the final judgment.
Key sources used for this analysis include: Nvidia’s latest quarterly results, OpenAI’s Cerebras partnership announcement, Anthropic on its use of Nvidia GPUs, AWS Trainium and Google TPUs, AMD’s MLPerf Inference 6.0 results, AMD’s MI355X specifications, Google on the Ironwood TPU, AWS on Rufus, Trainium and Inferentia, Microsoft’s Maia 200 announcement, Meta on MTIA production deployments, Nvidia’s inference platform materials, Nvidia’s Rubin architecture overview, Apple’s Foundation Models framework, and the llama.cpp project.
This chart, featured in our AI chip market deck, shows how AI accelerator chip technology has evolved over time
Related blog posts
- AI inference chips: which startup is ahead?
- AI inference: which startup is ahead?
