What's worth building in AI infrastructure?

What actually counts as AI infrastructure today?

AI infrastructure today includes much more than GPUs and cloud servers: we need to include the software and physical systems that let companies train, run, connect, secure and pay for AI workloads.

Five years ago, the category was easier to picture. Nvidia accelerators, cloud compute, training frameworks and data pipelines covered most of what people meant by AI infrastructure. The stack has become much wider.

A production AI application can now touch an Nvidia GPU, an AWS Trainium chip or Google TPU, an inference engine such as vLLM, a model gateway, object storage, a KV cache, an observability system, an agent sandbox and an identity layer. Underneath all of that sits networking, power and cooling infrastructure that increasingly determines whether another GPU can even be installed.

Agents are widening the definition again. A coding agent needs somewhere to execute code. An enterprise agent needs credentials and permissions. A long-running agent may generate enough context that moving and storing its KV cache becomes an infrastructure problem in its own right.

So when we ask what is worth building in AI infrastructure now, we are looking well beyond “who can provide more GPUs?” The interesting openings increasingly sit around inference, agent runtimes, security, memory, networking, power and the software that decides how all of those resources get used.

Why is AI infrastructure booming but getting harder for startups?

AI infrastructure is currently growing at an extraordinary rate, but the easiest parts of the market are becoming harder for a startup to enter.

The scale of hyperscaler spending explains the contradiction. Alphabet now expects roughly $195 billion to $205 billion of capital expenditure this year. Microsoft expects about $190 billion for the calendar year. Meta has guided to $130 billion to $145 billion. Amazon has talked about spending roughly $200 billion.

Taken together, those four companies are heading toward around $715 billion to $740 billion of capital spending in a single year. Even allowing for the fact that not every dollar is AI-related, the order of magnitude has changed completely.

Demand has kept up with the spending. AWS recently reached a $169 billion annualized revenue run rate and grew 37% year over year, its fastest growth in 18 quarters. Google Cloud revenue jumped 82% year over year to $24.8 billion in its latest quarter. Microsoft still expects AI capacity constraints despite spending more than $40 billion of capex in its latest quarter.

For a startup, that creates a strange market. Customers desperately want more AI infrastructure, yet the largest suppliers can throw tens of billions of dollars at the same shortage. Building something capital-intensive simply because demand is strong is therefore a much weaker idea than it sounds.

The better question is where all this spending creates secondary bottlenecks that the hyperscalers cannot remove just by ordering more servers.

Company	Current annual capex outlook	What we learn
Alphabet	$195B–$205B	AI capacity is still worth raising an already huge budget for
Microsoft	About $190B	Capacity remains tight despite enormous spending
Meta	$130B–$145B	AI infrastructure has become one of Meta's biggest uses of capital
Amazon	Roughly $200B	AWS is pre-building capacity against very large customer commitments
Combined	About $715B–$740B	Competing with hyperscalers on capital alone makes little sense

Is there still room for another GPU cloud?

Another generic GPU cloud looks like a bad startup to start today, even though GPU clouds themselves are producing enormous revenue.

CoreWeave shows both sides of the story unusually well. Its latest quarterly revenue reached roughly $2.6 billion, up 112% year over year. Revenue backlog reached about $104 billion, before another $25 billion-plus of customer commitments signed shortly afterward. Active power climbed to 1.5 gigawatts and contracted power to approximately 3.7 gigawatts. Backlogs this large raise the issue of whether the sector is overbuilt or still full, which we weigh in our report on AI infrastructure.

Demand clearly exists. The financing burden does too.

The problem is what it takes to serve that demand. CoreWeave now has tens of billions of dollars of assets and liabilities, large financing costs, enormous data-center commitments and contracts stretching across power, land and equipment. Its latest quarter included roughly $640 million of interest expense alone.

This is what the mature GPU-cloud model looks like these days. A serious competitor needs access to chips, cheap debt, data centers, power agreements, networking and enough contracted demand to finance the entire machine.

There can still be room for sovereign clouds, regional providers and highly specialized clusters. A company may also find temporary arbitrage when a particular accelerator is scarce.

But “we rent Nvidia GPUs” is no longer an attractive general startup thesis. The companies that entered early have already turned this business into infrastructure finance at industrial scale.

Is AI inference still worth building around?

AI inference is currently one of the strongest parts of AI infrastructure, but the opportunity has moved from simply hosting models to making inference dramatically more efficient.

The growth numbers are hard to ignore. Fireworks AI says it has passed a $1 billion annualized revenue run rate and now serves more than 40 trillion tokens a day. More than 95% of those tokens come from customer-specialized models rather than generic public models.

Baseten gives us another view of the same market. The company says revenue grew 20 times in a year while inference volume grew 40 times. Together AI recently reported annual bookings above $1.15 billion as companies used its infrastructure to run open models.

Those metrics are different, so we should not pretend they are directly comparable. Together they show something more useful: independent inference infrastructure has gone from an experimental developer product to a billion-dollar-scale business surprisingly quickly.

The interesting part now is cost compression. Baseten recently showed a customized video-inference stack running an open video model more than 50 times faster than its baseline implementation, bringing the reported cost of a generated clip from around five cents to well below one cent. It achieved that through a combination of distillation, quantization and custom kernels.

That is much closer to the kind of inference company we would build today. If a startup can make a real production workload two times, five times or twenty times cheaper, customers have a reason to care even as cloud providers improve.

Simple access to inference is getting crowded. Removing large chunks of inference cost is still extremely valuable.

Has simple AI model routing already become too crowded?

Simple AI model routing is already crowded enough that we would avoid building another “one API for every model” company today.

Stripe's agreement to buy OpenRouter for more than $8 billion makes the change particularly visible. OpenRouter built a neutral marketplace where developers could reach many AI models through a common interface and switch providers without rebuilding their application.

That was a powerful wedge while model choice was exploding. Now almost every adjacent infrastructure company can see the same opportunity. Inference providers offer multiple models. Cloud platforms expose model catalogs. AI gateways add fallback and routing. Application frameworks can switch providers directly.

The remaining opportunity sits deeper in the decision itself.

Imagine a system that observes the actual quality, latency and cost of every request and decides whether it should run on Claude, Gemini, GPT, an open model or a smaller specialized model. It could also choose region, provider and accelerator based on current capacity.

That solves a harder problem than API normalization. The customer is buying measurable optimization rather than convenience.

OpenRouter's acquisition makes generic routing look more like a layer being absorbed into larger platforms. Intelligent workload placement still feels open.

Are AI agent sandboxes becoming real infrastructure?

AI agent sandboxes are becoming a real infrastructure market because useful agents increasingly need computers they can control, rather than only model APIs.

E2B says its platform has launched hundreds of millions of sandboxes, with 88% of the Fortune 100 signed up. Daytona reached a $1 million forward revenue run rate in less than three months and doubled that figure another six weeks later before raising a $24 million Series A.

The demand is easy to understand once agents move beyond chat. A coding agent needs a terminal, file system and package manager. A research agent may need a browser. Reinforcement-learning systems need thousands of reproducible environments that can be created, destroyed and cloned automatically.

Cloud infrastructure already provides most of these primitives individually. Agents need them packaged differently. An environment may have to start almost instantly, execute unknown code safely, preserve state, fork into several possible paths and disappear when the job finishes.

That makes the basic sandbox useful, but it probably will not remain enough on its own. E2B, Daytona, Modal, Vercel and other infrastructure companies can all compete on startup speed and isolation.

We see more room around the sandbox: persistent state, secrets, network policy, audit trails, identity, approvals, resource scheduling and secure access to company systems. That is where an agent's temporary computer starts becoming production infrastructure.

Is AI agent security the best open market in AI infrastructure?

AI agent security is one of the best AI infrastructure markets to enter right now because companies are giving software permission to act before they have a clean way to control those actions.

The recent startup activity here is unusually concentrated. NewCore emerged with $66 million to build identity infrastructure for humans and AI agents. Arcade raised a $60 million Series A for agent authorization and secure actions. Geordie raised $30 million for agent security and governance. Money is piling into agent security, and we trace who is getting funded, and by whom, across the sector in our report covering AI infrastructure.

Those three companies alone have raised $156 million around slightly different versions of the same new problem. More importantly, Geordie says its ARR grew 1,300% during the first five months of the year, while Arcade says its tool-call volume increased 25 times over six months. Those are company-reported figures, but usage is moving alongside the funding.

The technical problem is also different enough from traditional identity to create room for new architecture. A human signs into Salesforce and performs actions deliberately. An agent may use a person's delegated permission, call several tools, create sub-agents and encounter untrusted instructions along the way.

Companies therefore need to know which agent is acting, who authorized it, which resource it can touch, what action it can perform and when that permission should disappear.

Okta, Microsoft and the large security platforms will clearly move into this market. That does not kill the opportunity. Security categories have repeatedly produced large independent companies even when platform vendors offered adjacent products.

For now, agent identity, delegated authorization and runtime policy would sit near the top of our build list.

Is AI observability already too crowded?

Generic AI observability is currently moving from an open startup category into a consolidation phase.

The latest evidence is unusually direct. Dynatrace recently agreed to buy Arize for $915 million, bringing AI evaluation and observability into a much larger monitoring platform. ClickHouse has integrated Langfuse into its product stack. CoreWeave previously bought Weights & Biases for roughly $1 billion.

When generative AI first appeared, developers suddenly needed new tools for traces, prompts, evaluations and model behavior. AI-native companies could build those products faster than Datadog, Dynatrace or the data platforms. The incumbents have now caught up enough to start buying the category.

There are still hard problems inside observability. Long-running agents can make dozens or hundreds of decisions before something goes wrong. A developer may need to replay the trajectory, identify the step that caused the failure, inspect which context the model saw and calculate how much the unsuccessful run cost.

Those narrower problems can still support companies. Agent replay, causal debugging, policy observability and per-task economics all feel less mature than basic LLM traces and dashboards.

We would therefore avoid another horizontal “Datadog for LLMs.” A tool that explains why an autonomous workflow failed may still have plenty of room.

Will context memory become its own AI infrastructure layer?

Context memory is starting to become a real AI infrastructure category as models and agents keep more state for longer periods.

The clearest evidence lately comes from Nvidia. Its BlueField-4-based Inference Context Memory platform is designed specifically to move KV cache away from scarce GPU memory and share that context across large AI clusters. Nvidia says the architecture can improve token throughput and power efficiency by as much as five times in supported workloads.

KV cache sounds obscure until we look at what long-running inference actually does. Every conversation, reasoning chain and agent trajectory creates intermediate data that the model may need again. Keeping all of it in expensive accelerator memory wastes capacity. Recomputing it wastes compute.

That turns context placement into an economic question.

Nvidia has already brought storage companies including Dell, HPE, IBM, Pure Storage, VAST and WEKA into the ecosystem. The involvement of so many established storage vendors shows that context memory is moving out of model-runtime internals and into physical infrastructure.

A startup probably does not want to challenge that ecosystem by inventing another storage appliance. Software looks more open: deciding which context should stay hot, where it should live, when it should be evicted, how several agents can share it and how persistent context moves securely across machines or regions.

If agents become longer-lived, this problem gets larger automatically. That makes context infrastructure one of the more interesting early markets we found.

Are networking and data movement better startup bets than GPUs?

Networking and data movement now look more attractive than building another general-purpose AI accelerator because increasingly expensive chips spend more of their time depending on everything around them.

Broadcom's recent numbers show how quickly this part of the stack is growing. AI semiconductor revenue reached $10.8 billion in its latest reported quarter, up 143% year over year, driven by custom accelerators and AI networking. Broadcom expects that figure to reach roughly $16 billion in the following quarter.

Arista gives us a second view. Quarterly revenue recently crossed $3 billion for the first time, up 37.7% year over year, while the company introduced 1.6-terabit networking platforms designed for AI fabrics.

These companies are already enormous, so building another Ethernet switch vendor would hardly count as an easy opening.

The startup opportunities are narrower and more software-heavy. AI clusters need better congestion control, collective-communication optimization, topology-aware workload scheduling, network telemetry and smarter decisions about where data should move before a job starts.

The economics can be compelling. Once a company has spent billions on accelerators, leaving those accelerators idle because tensors or model state are moving too slowly becomes extremely expensive.

A startup that can prove an existing cluster completes materially more work has a much cleaner sales pitch than a startup trying to convince customers to replace the whole cluster.

Is power becoming the real AI infrastructure bottleneck?

Power is currently one of the hardest limits on AI infrastructure growth, which makes the software connecting electricity supply to compute scheduling much more interesting than it looked a few years ago.

Lawrence Berkeley National Laboratory now estimates that data centers could consume about 11.8% of U.S. electricity by 2030 in its reference case, with scenarios ranging from 9.5% to 15.3%. The International Energy Agency expects data centers to account for around half of U.S. electricity-demand growth through 2030. Grid connections that take years are the kind of open problem we study in our AI infrastructure report.

The problem is concentrated geographically. A country may have enough electricity overall while a particular data-center region has no easy way to connect another 500 megawatts. Grid upgrades also take much longer than installing servers.

AI workloads have some flexibility that traditional industrial loads do not. A training job can sometimes move to another region or another hour. Batch inference can be delayed. Batteries can cover peaks. Different accelerators can consume different amounts of power for the same job.

Yet compute schedulers and energy systems are still largely managed separately.

A power-aware AI scheduler could decide where and when jobs run based on GPU availability, electricity prices, local power limits, batteries and demand-response commitments.

As of now, we would rather build that control software than another dashboard telling data-center operators how much electricity they consumed yesterday.

Is AI data-center cooling still worth building around?

AI data-center cooling is growing fast, but a startup needs a real technical advantage because large incumbents are already capturing much of the demand.

Vertiv's latest quarterly sales reached about $3.27 billion, up 24% year over year, while operating profit grew 44%. The company raised its full-year organic-growth outlook to roughly 30% to 32%.

Those numbers show that power and thermal equipment have become direct beneficiaries of AI infrastructure spending. New rack-scale systems are also pushing densities higher, making liquid cooling much more common in new AI deployments.

The market is real. The startup opening is narrower.

Standard cooling distribution units, pumps and manifolds can quickly turn into procurement markets where Vertiv, Schneider Electric, server vendors and specialist suppliers have obvious advantages.

We find the edges more interesting: retrofitting older data centers for much higher rack densities, detecting thermal problems before equipment throttles, coordinating cooling with workload scheduling, improving heat exchangers or fluids, and recovering useful heat economically.

Cooling can produce excellent AI infrastructure companies. It just requires more than putting “AI data center” in front of an existing industrial product.

Does it still make sense to build a new AI chip?

Building another general-purpose AI accelerator today is an extremely difficult startup bet unless the architecture changes the economics of one specific workload by a very large amount.

Competition now comes from several directions at once. Nvidia keeps improving complete rack-scale systems rather than selling isolated GPUs. Amazon says its chips business has passed a $25 billion annual revenue run rate, and Trainium has attracted multi-gigawatt commitments from Anthropic and OpenAI. Google has spent years building TPUs. Microsoft has Maia. With Amazon, Google and Microsoft all building chips, we look at who is ahead and who is falling behind in our AI infrastructure reality check.

OpenAI has also moved directly into silicon with Broadcom. Their new inference processor is meant to deploy at gigawatt scale across multiple generations.

A chip startup now has to survive silicon-development costs, advanced packaging, memory supply, compiler work and customer migration before it even reaches the question of whether its hardware is faster.

There is still room lower down the problem stack. Optical interconnects, memory architecture, chiplets, packaging, power delivery and highly specialized accelerators can benefit from the same growth without trying to reproduce Nvidia's entire software ecosystem.

We would also pay attention to chips designed around new workload shapes. Agentic systems are increasing CPU use. Long-context inference stresses memory. Some edge applications care far more about watts than peak throughput.

A startup with a tenfold improvement on one of those constraints has a plausible wedge. A startup with a GPU that benchmarks 20% better has a much harder path.

Can a startup build the control plane for Nvidia, Trainium, TPU and Maia?

A hardware-neutral AI compute control plane looks increasingly worth building because companies are gaining more accelerator choices while the cost of choosing badly keeps rising.

A few years ago, “AI compute” mostly meant Nvidia. Today a company may have access to Nvidia GPUs, AMD GPUs, AWS Trainium, Google TPUs, Microsoft Maia and specialized accelerators. CPUs are also becoming more important for agentic workloads.

Each option has different prices, availability, memory limits, compilers and performance characteristics. The cheapest chip on paper may be terrible for one model and excellent for another.

Modular has already gone after part of this problem. The company raised $250 million at a $1.6 billion valuation to build what it calls a unified compute layer, with support expanding across CPUs, GPUs and custom accelerators.

Open-source software is helping too. vLLM is becoming more hardware-neutral, while AMD, Google and other vendors are making their accelerators easier to use with common frameworks.

Simple portability alone may eventually become a commodity.

The more interesting control plane would benchmark a customer's real workloads continuously, know current hardware prices and availability, optimize or compile the workload for several targets and then decide where every job should run.

A video model might go to Nvidia. Another inference workload might be cheaper on Trainium. A batch job could move to spare AMD capacity. The application would not need to care.

This type of company also has a useful strategic property: every new proprietary chip makes the problem larger. Hyperscaler fragmentation becomes fuel for the neutral layer sitting above it.

What’s actually worth building in AI infrastructure today?

The best AI infrastructure startups to build today sit around agent security, inference optimization, context memory, heterogeneous compute and power-aware scheduling.

Raw demand is still enormous. AWS is growing at its fastest rate in years, Google Cloud has been growing above 80%, independent inference companies have reached billion-dollar-scale run rates or bookings, and data-center electricity demand keeps being revised upward.

But we would not chase the largest pools of spending blindly. GPU clouds, generic model gateways and broad AI observability have already attracted strong companies, huge amounts of capital and increasingly aggressive incumbents.

The more interesting openings come from problems that are getting worse as AI scales.

Agents need identities, permissions and computers. Inference needs to become cheaper every year. Longer contexts need somewhere to live. More accelerators create more hardware fragmentation. Bigger clusters put pressure on networks. More data centers collide with electricity and cooling constraints.

Those problems also reinforce each other. A future agent workload may need a sandbox, delegated credentials, persistent context and thousands of inference requests. The infrastructure could choose among several accelerators while deciding whether local power capacity allows the job to run immediately.

We think the next important infrastructure companies will increasingly make those decisions.

That is why our favorite ideas currently look more like control systems than resource providers. The scarce resource may be GPUs, memory, network bandwidth, electricity or permission to access a database. The valuable software decides how that scarce resource gets used.

Area	What we would build	Why it still looks open	Verdict
AI agent security	Agent identity, delegated authorization and runtime policy	Enterprises are deploying autonomous software faster than identity systems can govern it	Best opportunity
Inference optimization	Workload-specific runtimes, compilation and cost optimization	Usage is exploding while cost per request keeps falling	Best opportunity
Heterogeneous compute	Automatic placement across Nvidia, AMD, Trainium, TPU and other accelerators	Hardware fragmentation is increasing	Best opportunity
Context memory	KV-cache placement, sharing and persistent agent context	Long-context and agent workloads create a new storage problem	Best opportunity
Power-aware compute	Scheduling tied to electricity, batteries and capacity limits	Power is becoming a direct constraint on new compute	Best opportunity
AI networking	Workload-aware congestion control, communication and scheduling	Expensive accelerators increasingly depend on data movement	Strong
Agent sandboxes	Stateful, secure execution environments with policy and identity	Agents increasingly need their own computers	Strong
Specialized cooling	Retrofit, thermal control and differentiated components	Rack densities keep rising	Selective
AI observability	Agent replay, causal debugging and specialized monitoring	Horizontal observability is consolidating	Selective
GPU cloud	More undifferentiated GPU rental capacity	Capital requirements and incumbents are already extreme	Avoid
Generic model routing	Common API and simple provider switching	OpenRouter validated the category and consolidation has begun	Avoid
General-purpose AI chip	Another broad Nvidia alternative	Custom silicon from hyperscalers keeps raising the entry barrier	Avoid