AI Chips: what are the biggest unsolved problems?

Why are AI chips getting faster while AI computing is still so expensive?

AI chips are improving incredibly fast, but the cost of building and running large AI systems remains enormous because demand for computing power keeps growing alongside hardware performance.

Stanford's 2025 AI Index estimated that machine-learning hardware performance improved roughly 43% annually, while hardware price-performance improved around 30% per year. Those are remarkable gains, especially for an industry already operating near the limits of semiconductor manufacturing.

Yet NVIDIA's data-center revenue reached $89 billion in the quarter ending July 2026, up 117% from a year earlier. That single quarter generated more revenue than NVIDIA's entire fiscal 2024, when the company reported $60.9 billion across all its businesses.

Meanwhile, Stanford found that the cost of running a model with GPT-3.5-level performance fell from around $20 per million tokens in November 2022 to $0.07 in October 2024. That's a reduction of more than 280 times in less than two years.

So why hasn't cheap computing become the norm?

Because companies are putting those efficiency gains to work on bigger models, longer conversations, video generation, and AI systems that perform multiple reasoning steps before producing an answer.

We can already see this in the shift toward reasoning models and AI agents. A task that previously required one short response might now involve dozens of model calls, searches, and intermediate calculations.

Cheaper individual operations are enabling much more computation per task. The unresolved challenge is whether hardware and software improvements can bring down the cost of those more demanding tasks as quickly as their complexity increases.

If you want more recent data on this point, see the latest figures in our AI Chips report ↓.

Is raw processing power still the biggest problem for AI chips?

For many AI inference workloads, raw processing power has stopped being the main limitation. Memory bandwidth is now just as important, and sometimes more important, than how many calculations a chip can perform.

Modern accelerators are exceptionally good at performing mathematical operations in parallel. The difficulty comes when those computing units need information that hasn't arrived from memory yet.

A May 2026 research report from UC Berkeley identified memory bandwidth as a major limitation for large-language-model inference. A July 2026 survey in Artificial Intelligence Review reached similar conclusions, highlighting the cost of moving model weights, attention data, and intermediate results.

The problem changes depending on what the chip is doing.

Training a large AI model involves intensive mathematical calculations, often across thousands of GPUs. Generating text token by token can instead require repeatedly reading enormous amounts of model data for relatively little computation.

Longer conversations introduce another problem: the chip must keep track of more previous information, increasing the amount of memory needed for attention caches.

This explains why advertised floating-point performance can be misleading. A chip with twice the theoretical computing power won't necessarily generate text twice as fast.

For AI inference, memory bandwidth, latency, and the number of useful tokens generated per dollar often tell us more than peak processing power.

Why is memory becoming such a big problem for AI chips?

Memory is currently one of the hardest problems in AI chip design because large models need enormous amounts of fast storage, and memory technology hasn't kept pace with the growth in processing power.

Take a dense AI model with 70 billion parameters. Its weights alone require approximately 140 GB of memory at 16-bit precision, 70 GB at 8-bit precision, or 35 GB at 4-bit precision.

A 405-billion-parameter model needs about 810 GB at 16-bit precision, before accounting for attention caches and other operating requirements.

That's why large AI models often need several accelerators working together, even when an individual chip has enough processing power to handle a substantial share of the calculations.

High-bandwidth memory, or HBM, has become essential. NVIDIA, AMD, and Google's advanced accelerators use memory systems designed to move information much faster than conventional server memory.

But HBM creates its own problems. Memory stacks are expensive to manufacture, difficult to integrate into advanced packages, and constrained by heat, power consumption, and production capacity.

The situation is getting more complicated as AI models support longer conversations and more simultaneous users. Even when model weights fit inside an accelerator's memory, attention caches can consume substantial additional capacity.

We can break down the main memory problems and the technologies being developed to address them.

Memory problem	Why it's difficult	Current approaches
Large model weights	Models can exceed individual-chip memory	Quantization, model partitioning
Limited bandwidth	Computing units wait for data	HBM, improved data reuse
Attention caches	Memory grows with conversation length	Cache compression, PagedAttention
Memory latency	Small transfers can delay processing	Better caching and scheduling
HBM manufacturing	Complex memory stacks limit supply	Capacity expansion, yield improvements
Memory power use	Moving data consumes electricity	Lower precision, local memory
Multi-chip memory	Sharing data between chips adds delays	Faster interconnects
Memory fragmentation	Unused memory reduces serving capacity	Smarter allocation systems

Can AI chips keep improving without making transistors smaller?

AI chips can keep getting better without relying entirely on smaller transistors, but semiconductor scaling remains important and increasingly expensive.

For decades, shrinking transistors allowed manufacturers to pack more computing power into the same area while improving energy efficiency.

That process hasn't stopped. TSMC's third-quarter 2026 results showed that 2-nanometer technology already represented 8% of its wafer revenue, while 3-nanometer technology accounted for 33% and 5-nanometer technology for 28%.

Together, those three generations generated 69% of TSMC's wafer revenue. The figure covers all customer applications, but it demonstrates how much semiconductor production has moved toward advanced manufacturing.

At the same time, pushing beyond current processes requires increasingly sophisticated equipment.

In March 2026, Belgian research institute imec received an ASML EXE:5200 high-numerical-aperture EUV lithography system to support research into future semiconductor technologies.

Chip designers are also finding improvements elsewhere.

NVIDIA's Blackwell architecture combines advanced computing units, high-bandwidth memory, and fast chip-to-chip connections. AMD uses chiplet-based designs to combine different silicon components, while Google optimizes its TPUs around the mathematical operations most common in AI workloads.

These approaches are becoming increasingly important because shrinking transistors alone can no longer deliver the improvements customers expect from every new generation.

The hard part now is combining advances in manufacturing, architecture, memory, and software without making the finished processor prohibitively expensive.

Are AI chips becoming too difficult to manufacture?

Advanced AI chips are becoming harder to manufacture because building a complete accelerator now requires several demanding production processes to work together almost perfectly.

Producing the main processor is only one part of the job. Manufacturers also need high-bandwidth memory, advanced packaging, specialized substrates, interconnects, power components, and extensive testing.

Each component has its own manufacturing capacity and yield constraints.

TSMC's third-quarter 2026 revenue reached $46.72 billion, up 41.2% year over year in US-dollar terms. Demand for advanced manufacturing remains extremely strong, but those facilities also serve smartphone manufacturers and other chip designers.

Packaging has become particularly difficult.

TSMC's CoWoS technology allows advanced processors to be integrated with stacks of HBM through high-density connections. NVIDIA's Blackwell processors and AMD's Instinct accelerators both depend on sophisticated packaging techniques.

These designs make it possible to build computing systems that would be impractical as conventional single-piece chips. However, assembling several advanced components creates additional opportunities for manufacturing defects, delays, and lower yields.

NVIDIA acknowledged these risks in its fiscal 2027 second-quarter regulatory filing, specifically identifying manufacturing complexity, component availability, quality issues, and lower yields among the challenges affecting production.

There's also an important difference between manufacturing wafers and delivering finished accelerators.

A processor can be successfully fabricated while its final assembly remains delayed by unavailable memory, packaging capacity, or another component.

That's why expanding semiconductor factories alone won't automatically eliminate AI chip shortages. Manufacturers need the entire production chain to grow together, especially the advanced packaging and memory technologies used in the most powerful accelerators.

If you want more recent data on this point, see the latest figures in our AI Chips report ↓.

Can AI chips keep getting faster without consuming much more electricity?

AI chips are becoming more energy-efficient, but the industry's electricity consumption continues to rise because companies are installing more accelerators and running increasingly demanding workloads.

Stanford's 2025 AI Index estimated that machine-learning hardware energy efficiency improved approximately 40% annually over the period studied.

That should have created substantial savings. Instead, the International Energy Agency's updated Energy and AI outlook estimates that global data-center electricity consumption will grow from approximately 485 terawatt-hours in 2025 to around 950 TWh in 2030.

That's an increase of about 96% in five years. The IEA also expects electricity consumption from AI-focused data centers to triple over that period.

These figures include the full electricity requirements of data centers, not exclusively AI processors.

The engineering challenge becomes clearer when we look at individual systems.

Supermicro specifies up to 132 kW of power-supply capacity for its NVIDIA GB200 NVL72 rack configuration, which contains 72 Blackwell GPUs. Actual consumption depends on the workload and configuration, but that capacity illustrates how much electrical power can be concentrated in one rack.

Nearly all electricity consumed by computing equipment eventually becomes heat, creating a second problem: cooling.

NVIDIA's GB200 NVL72 uses liquid cooling, and Supermicro's corresponding infrastructure includes coolant distribution equipment designed for extremely dense systems.

Traditional air cooling is increasingly difficult to use efficiently at these power densities. Liquid cooling helps, but it adds pumps, pipes, maintenance requirements, and potential failure points.

The larger challenge is connecting these systems to reliable electricity supplies.

The IEA has noted that data centers can often be constructed within two or three years, while major grid connections and transmission upgrades can take considerably longer.

A company might have the money and the GPUs to build a new AI cluster but still lack enough electricity to operate it.

Power availability and cooling are becoming central constraints on AI infrastructure growth, particularly for operators planning hundreds of megawatts of computing capacity.

Why do AI chips become less efficient when thousands work together?

AI chips lose some of their efficiency in large clusters because processors spend more time communicating and coordinating with one another as the system grows.

A single accelerator can process information stored in its local memory. Large AI models frequently need thousands of accelerators to exchange intermediate results, synchronize calculations, and share different portions of a model.

Every exchange takes time.

NVIDIA's GB200 NVL72 addresses this problem by connecting 72 Blackwell GPUs through an NVLink system providing 130 TB/s of aggregate communication bandwidth under NVIDIA's specifications.

Google's Ironwood TPU architecture takes another approach, supporting configurations with up to 9,216 chips in a superpod.

Those are substantial engineering achievements, but increasing the number of accelerators doesn't guarantee a proportional increase in useful performance.

Communication delays, uneven workload distribution, and synchronization overhead become harder to manage as clusters grow.

Evidence of this shift appeared in MLCommons' April 2026 MLPerf Inference results. Multi-node submissions increased 30% compared with the preceding round, while configurations containing more than ten nodes represented 10% of submissions, up from 2%.

That fivefold increase in the share of larger configurations over one benchmark cycle shows growing interest in measuring more complex inference systems. It doesn't establish how efficiently every large production cluster scales.

An accelerator may finish its assigned task quickly but still have to wait for other processors before the system can continue.

Making larger clusters efficient requires improvements in networking, memory distribution, scheduling, and model architecture. Simply adding more GPUs can become an expensive way to gain relatively little additional performance.

Are AI chip benchmarks making performance look better than it really is?

AI chip benchmarks can make hardware look more impressive than it feels in everyday use, especially when results emphasize total throughput rather than response time or cost per completed task.

A system might generate an enormous number of tokens per second by processing many requests together. That doesn't necessarily mean an individual user receives a fast response.

The reverse is also possible. A system optimized for extremely low latency might deliver excellent responsiveness while serving fewer users overall.

This distinction matters much more now that AI applications increasingly involve reasoning, tool use, and multiple model calls.

MLCommons has been adjusting its benchmarks to reflect these changes.

Its September 2026 MLPerf Inference v6.1 release included a record 30 participating organizations and 120 submitted systems. It also introduced agentic and end-to-end evaluations, alongside updated tests for large language models.

More than half of the participating organizations used the newer API-focused testing infrastructure, which brings some benchmark measurements closer to real serving environments.

That is encouraging, but comparing AI chips remains difficult because production workloads vary considerably.

Generating short answers, processing video, running a reasoning model, and serving thousands of simultaneous users place different demands on the hardware.

A benchmark result can therefore be completely valid while being poorly suited to a particular customer's needs.

Performance metric	What it measures	Main limitation
Peak FLOPS	Theoretical arithmetic performance	Doesn't show actual utilization
Tokens per second	Text generation throughput	Can hide individual response delays
Time to first token	Initial response speed	Doesn't measure the full request
Output-token latency	Speed of ongoing generation	Doesn't show total serving capacity
Throughput per watt	Energy efficiency	Excludes many infrastructure costs
Throughput per dollar	Hardware cost efficiency	Depends on prices and workload
Memory capacity	Available model and cache storage	Doesn't measure memory speed
Multi-node scaling	Performance across accelerators	May hide operational complexity
Cost per completed task	Practical AI service cost	Difficult to standardize across tasks

Can specialized AI chips actually replace NVIDIA GPUs?

Specialized AI chips are already competitive for certain workloads, especially inside large cloud companies, but replacing NVIDIA GPUs across the broader market remains much harder.

Google provides one of the strongest examples.

Its Ironwood TPU generation, introduced in April 2025 and made commercially available later that year, was designed with particular attention to inference and large-scale AI workloads.

Amazon has developed Trainium and Inferentia processors, Microsoft has introduced Maia, and Meta has invested in its MTIA accelerator family.

These companies have a major advantage: they control large amounts of their own computing demand.

That allows them to optimize hardware for workloads they understand, rather than supporting every possible model and software configuration.

The trade-off is flexibility.

AI architectures change quickly. New attention mechanisms, numerical formats, model structures, and inference techniques can alter which hardware designs are most efficient.

A specialized chip may perform extremely well on the workloads it was designed for while requiring additional engineering to support newer models.

General-purpose GPUs remain attractive because researchers and developers can use them across a wide range of applications.

We should therefore expect more specialized accelerators inside cloud platforms, particularly where the operator can control both hardware and software. A wholesale replacement of GPUs across AI research, training, and commercial inference looks far less likely.

If you want more recent data on this point, see the latest figures in our AI Chips report ↓.

Why is NVIDIA still so difficult to challenge in AI chips?

NVIDIA remains the strongest company in AI computing because competitors have to match its software, networking, and deployment ecosystem as well as its GPU performance.

CUDA is central to that position. Developers have spent years building software around NVIDIA hardware, supported by optimized libraries, inference frameworks, and development tools.

But NVIDIA's advantage extends beyond software compatibility.

Its platforms combine GPUs with high-speed interconnects, networking equipment, rack-scale systems, and increasingly integrated deployment tools.

The financial results show how powerful this position remains.

In August 2026, NVIDIA reported $89 billion in quarterly data-center revenue, up 117% year over year. Data-center products represented approximately 92.5% of its $96.2 billion quarterly revenue.

The company's gross margin also reached 75%, showing that its growth has not required aggressive margin sacrifices.

This happened while AMD expanded its Instinct accelerator business, Google continued developing TPUs, and Amazon and Microsoft invested in their own processors.

Competitors therefore face an unusually difficult challenge: offering lower total computing costs without requiring customers to spend too much time rewriting software, optimizing models, or changing infrastructure.

AMD's ROCm ecosystem has progressed, while Google's TPUs have demonstrated that alternative hardware can work at significant scale.

Still, NVIDIA's continued revenue growth suggests that competition has yet to seriously weaken its commercial position.

The most promising challengers are likely to win specific workloads or customer groups first. Matching NVIDIA across the full AI computing market requires a much broader ecosystem than a faster individual chip.

Are AI chips becoming obsolete too quickly?

AI chips are becoming economically outdated faster than many operators would like, even though older accelerators can remain technically useful for years.

NVIDIA has moved through Hopper, Blackwell, and Rubin generations, while Google has developed successive TPU architectures and AMD continues expanding its Instinct product family.

Each generation can offer improvements in memory capacity, throughput, energy efficiency, or networking.

That's attractive for customers buying new equipment, but it creates a financial problem for companies that recently invested billions in existing systems.

Imagine an accelerator purchased for $30,000 and depreciated over four years. Ignoring financing and residual value, that represents $7,500 of annual depreciation.

If competitive pressure forces the operator to recover the same investment over two years instead, the annual depreciation burden rises to $15,000.

Those figures are illustrative, but the underlying problem is real.

An accelerator can continue working perfectly while a newer generation delivers substantially more useful computing for the same electricity consumption.

However, older hardware doesn't automatically become worthless.

Smaller models, batch processing, development environments, and workloads with flexible latency requirements can continue using previous-generation accelerators efficiently.

The missing evidence is a sufficiently transparent record of actual GPU utilization, resale values, and commercially useful lifetimes across large AI fleets.

Without those figures, it's difficult to determine how quickly hardware upgrades are genuinely paying for themselves.

For AI infrastructure operators, depreciation and utilization may become as important to profitability as the performance of the latest chip generation.

If you want more recent data on this point, see the latest figures in our AI Chips report ↓.

Can AI inference become cheap enough for almost everyone?

AI inference is becoming much cheaper, and there's good evidence that this trend can continue, but advanced reasoning and autonomous agents are creating new demands that could absorb much of the savings.

Stanford's 2025 AI Index recorded a more than 280-fold reduction in the price of GPT-3.5-level inference between November 2022 and October 2024.

That improvement came from several sources working together: better hardware, smaller and more efficient models, quantization, improved serving software, and stronger competition.

Techniques such as FlashAttention reduce unnecessary memory transfers. PagedAttention helps manage attention-cache memory, while speculative decoding can accelerate text generation.

These advances help operators extract more useful work from existing accelerators.

But the cost of an AI application depends on how much computation it performs.

A conventional chatbot might generate a few hundred tokens in one response. A more advanced agent may search documents, call external tools, run several reasoning steps, and repeatedly evaluate its own output.

The price of each token can fall sharply while the number of tokens used per task increases.

There's also evidence that software improvements can rapidly change what hardware is necessary.

Stanford's 2025 AI Index showed that a performance threshold on the MMLU benchmark, previously reached by models with hundreds of billions of parameters, could later be achieved by a model containing only 3.8 billion parameters.

That's a benchmark-specific comparison, but it illustrates the scale of improvement possible through model design.

For relatively standardized tasks, cheap AI inference looks increasingly achievable. The economics of long-running, complex AI agents are much less settled.

The metric worth watching is the cost of successfully completing a task, including all the intermediate computation it requires.

Can powerful AI chips run large models directly on phones and laptops?

Consumer AI chips can already run useful language models locally, but matching the capabilities of the largest cloud AI systems remains unrealistic for most phones and laptops.

Apple, Qualcomm, AMD, and Intel have all invested in processors designed to accelerate AI workloads on personal devices.

Local processing offers some clear advantages. Applications can work without a constant internet connection, certain private data can remain on the device, and users may avoid repeated cloud computing charges.

Memory remains the biggest practical obstacle.

A dense 7-billion-parameter model requires approximately 3.5 GB just to store its weights at 4-bit precision. A 70-billion-parameter model needs roughly 35 GB at the same precision.

Those estimates exclude additional memory for attention caches, runtime operations, and other applications.

Most smartphones cannot dedicate that much memory to one AI model.

Battery life and heat also impose limits. A phone has a much smaller power budget than a data-center accelerator and cannot sustain the same level of computation.

However, smaller AI models are improving quickly.

The progress recorded in Stanford's 2025 AI Index suggests that capabilities once requiring enormous models can increasingly be delivered by much smaller ones, at least on certain tasks.

That makes local AI particularly promising for transcription, image processing, translation, summarization, and other relatively focused applications.

The next challenge is making those applications accurate and responsive without draining batteries or consuming too much memory.

Cloud AI will remain important for demanding workloads, while more everyday tasks should gradually move onto personal devices.

Is the AI chip supply chain too dependent on Taiwan?

The AI chip supply chain remains heavily dependent on Taiwan, particularly for the most advanced processors, and replacing that manufacturing capacity would be extremely difficult.

TSMC manufactures leading-edge chips for companies including NVIDIA, AMD, and Apple. Its manufacturing capabilities are also closely connected to advanced packaging technologies used in high-performance AI accelerators.

The company's third-quarter 2026 financial results show its scale: $46.72 billion in revenue, with 79% of wafer revenue coming from 7-nanometer or more advanced manufacturing technologies.

But the supply chain extends well beyond Taiwan.

ASML in the Netherlands supplies critical advanced lithography systems. South Korea's SK hynix and Samsung, alongside US-based Micron, are major high-bandwidth memory suppliers.

These companies operate in different parts of the production process, and their technologies must work together.

A disruption involving advanced memory, packaging, or chip fabrication can delay an entire system even if the other components remain available.

Geopolitical restrictions add another layer of uncertainty.

In January 2026, the US Bureau of Industry and Security revised its approach to certain advanced semiconductor exports to China, allowing case-by-case licensing reviews for products including NVIDIA's H200 and AMD's MI325X under specified conditions.

This illustrates how chip availability can change through government decisions independently of technical performance.

Manufacturing investments in the United States and other regions are gradually expanding geographical options.

However, reproducing an advanced semiconductor ecosystem requires far more than constructing a factory. Companies need specialized equipment, packaging capacity, suppliers, skilled workers, and years of process development.

Taiwan's concentration of advanced production therefore remains one of the industry's most consequential vulnerabilities.

What are the biggest unsolved problems in AI chips today?

The biggest unsolved problems in AI chips are memory bandwidth, electricity availability, advanced manufacturing, and the difficulty of using thousands of processors efficiently. These constraints currently matter more than simply making individual chips faster.

Memory is arguably the most immediate technical challenge for large-scale inference. Modern accelerators can perform extraordinary numbers of calculations, but moving model weights and attention data remains expensive in both time and energy.

Electricity is becoming the larger physical constraint. The IEA expects global data-center electricity consumption to approach 950 TWh by 2030, with AI-focused facilities growing especially quickly. Some data-center projects already face longer waits for grid infrastructure than for the computing equipment itself.

Manufacturing adds another bottleneck. Advanced processors increasingly rely on high-bandwidth memory, complex packaging, and tightly integrated components. TSMC's leading-edge capabilities and NVIDIA's supply-chain disclosures show how difficult it is to increase production of complete systems.

Communication between accelerators is another serious problem. NVIDIA and Google have developed systems connecting dozens or thousands of chips, but adding processors also increases networking requirements, synchronization overhead, and operational complexity.

Software compatibility makes the competitive landscape even harder to change. Specialized accelerators from Google, Amazon, AMD, and others are increasingly credible, but NVIDIA continues to benefit from a deeply established software and hardware ecosystem.

Some of these problems are improving quickly. Quantization, better inference software, smaller models, and more efficient memory management are already lowering computing costs.

Others will take much longer. Building electrical grids, expanding advanced packaging capacity, developing new semiconductor processes, and diversifying manufacturing supply chains require substantial capital and years of work.

Our conclusion is that the AI chip industry's biggest challenge today is scaling complete computing systems without allowing memory, energy, manufacturing complexity, and infrastructure costs to erase the gains from faster processors.

The next breakthrough may come from a better GPU, but the companies that solve the surrounding bottlenecks will probably have just as much influence over how cheap and widely available advanced AI eventually becomes.

If you want more recent data on this point, see the latest figures in our AI Chips report ↓.