GPU efficiency: which startup is ahead?

Why are AI companies suddenly so focused on GPU efficiency?

GPU efficiency has become a much bigger priority because companies are spending heavily on AI computing power while leaving surprisingly large amounts of it unused.

Cast AI's 2026 State of Kubernetes Optimization Report found average GPU utilization of just 5% across the enterprise Kubernetes environments it analyzed. The study covered tens of thousands of clusters running on AWS, Google Cloud, and Microsoft Azure. These measurements describe enterprise environments before optimization, rather than the highly utilized training clusters operated by major AI laboratories.

The financial pressure is increasing, too. AWS raised prices for H200 Capacity Blocks by 15% in January 2026, according to Cast AI's analysis of cloud pricing. Companies that reserved GPUs expecting computing costs to keep falling suddenly faced a less forgiving market. Pricier compute, after years of expecting cheaper, is a shift we cover over the past year in our report on AI infrastructure.

Meanwhile, AI applications are creating unpredictable demand. A customer-support chatbot might receive thousands of requests during working hours and very few overnight. Reserving dedicated GPUs for peak traffic means paying for hardware even when nobody is using it.

That's why startups such as Cast AI, ScaleOps, Fireworks AI, and Baseten are attracting attention. Some help companies share their existing GPUs more efficiently, while others improve how AI models run on those GPUs.

What does being the most GPU-efficient startup actually mean?

The most efficient GPU startup is the one that helps customers complete the most useful AI work for the least money, without sacrificing the speed or reliability they need.

That sounds straightforward, but the companies competing in this market solve different problems. Cast AI and ScaleOps focus on resource allocation, making sure computing capacity isn't unnecessarily reserved. Fireworks AI and Baseten concentrate on inference, the process of running trained AI models to generate answers.

A GPU operating at 95% utilization might look impressive, but that doesn't necessarily make it economical. It could be processing unnecessary requests, running a model that is too large for the task, or costing twice as much as another suitable accelerator.

We therefore need to consider GPU utilization, cost per successful inference, and the amount of capacity required to support actual customer demand. These measures can produce different winners, especially when comparing infrastructure software with inference providers.

Which startup is leading GPU efficiency right now?

Cast AI currently has the strongest overall case among independent startups specializing in GPU resource optimization, while Fireworks AI leads the broader competition in commercial AI inference.

Cast AI combines GPU sharing, automatic scaling, resource allocation, and multi-cloud capacity sourcing. ScaleOps is its closest direct challenger, with exceptionally fast reported growth and a growing list of enterprise customers.

Fireworks AI operates in a larger commercial category. It surpassed $1 billion in annualized revenue run rate in July 2026, compared with Cast AI's disclosed valuation of more than $1 billion rather than a comparable revenue figure. Modal has also crossed $1 billion in annualized run rate, although a substantial part of its business now comes from products outside GPU inference.

Run:ai deserves special consideration. Its GPU-sharing benchmarks are among the strongest available, but Nvidia acquired the company in December 2024, removing it from the independent startup category.

We give Cast AI the lead in independent GPU optimization software. Fireworks and Modal have much stronger publicly disclosed evidence of commercial scale, although their businesses aren't directly comparable.

Company	GPU efficiency approach	Key evidence	Our assessment
Cast AI	GPU sharing, scaling, multi-cloud optimization	Valuation above $1B; enterprise utilization dataset	Leading independent optimization specialist
ScaleOps	Autonomous infrastructure allocation	$130M Series C; over $800M valuation	Strongest direct challenger
Fireworks AI	Optimized model inference	Over $1B annualized revenue	Commercial inference leader
Baseten	Model deployment and inference optimization	Reported $600M annualized revenue in Q1 2026	Major inference competitor
Modal	Serverless AI compute and inference	Over $1B annualized run rate	Fast-growing compute platform
Together AI	Training and optimized inference infrastructure	Large-scale commercial AI infrastructure platform	Broad inference competitor
Run:ai	GPU scheduling and fractional allocation	Up to roughly 3x mixed-workload user capacity in tests	Technical leader owned by Nvidia

Are companies really using only 5% of their GPUs?

Cast AI's reported 5% GPU utilization is supported by measurements from its enterprise Kubernetes dataset, but applying that figure to the entire AI industry would be misleading.

The company examined workloads across AWS, Azure, and Google Cloud through April 2026, collecting measurements before customers activated its optimization features. Average CPU utilization was 8%, down from 10% the preceding year, while memory utilization fell from 23% to 20%.

GPU usage was unusually low across all three cloud platforms: approximately 2% on Azure Kubernetes Service, 5% on Amazon EKS, and 6% on Google GKE. These differences were relatively small compared with the overall amount of unused capacity.

One organization in the dataset operated 136 H200 accelerators at around 49% utilization. That's almost ten times the reported average, showing how widely resource usage can vary between customers.

Fewer than 2% of the GPUs examined were running on Spot Instances, which can offer lower prices in exchange for the possibility of interruption. Companies were therefore leaving capacity unused while making relatively little use of cheaper, interruptible computing.

These figures establish a substantial optimization opportunity in enterprise Kubernetes deployments. They don't mean every company could eliminate 95% of its GPU spending. That would be a pretty wild interpretation of the data.

Can GPU sharing really reduce hardware requirements by half?

GPU sharing can cut hardware requirements substantially when several AI applications use their assigned GPUs only intermittently.

In February 2026, Nvidia and Nebius published benchmarks showing that a half-GPU allocation delivered 77% of the throughput and 86% of the concurrent-user capacity of a full allocation. The tests used Nvidia's Run:ai orchestration technology and Nvidia NIM inference microservices. Half a GPU delivering 77% of the throughput shows efficiency tech maturing; we look at which technologies gain traction in our AI infrastructure market report.

We can express that result another way: achieving 77% of full-GPU throughput with half the allocated capacity produces approximately 1.54 times the throughput per allocated GPU share. That comparison measures allocation efficiency rather than any improvement in the physical processor itself.

The effect became even stronger for certain combinations of applications. Quarter-GPU allocations supported up to twice as many concurrent users for smaller models, while sharing GPUs across chat, reasoning, and embedding workloads supported nearly three times as many users in the tested configurations.

Cast AI has reported a similar pattern at ALLEN Digital, where automated sharing allowed workloads previously associated with roughly 100 provisioned GPUs to run on approximately 25 to 75 physical GPUs, depending on demand.

The results are impressive, although their applicability depends heavily on workload characteristics. Large training jobs that already keep accelerators busy have much less spare capacity available for sharing.

Has Nvidia already taken the lead with Run:ai?

Nvidia is now the strongest established competitor in GPU orchestration thanks to Run:ai, and its decision to open-source the platform makes life harder for independent startups.

Nvidia completed its acquisition of Run:ai in December 2024 in a deal widely reported at approximately $700 million. The company subsequently integrated Run:ai into its AI infrastructure offering.

In October 2026, Nvidia announced the open-source release of the full platform. Developers gained access to the KAI Scheduler, GPU Fractioning Operator, resource-management components, and workload-management interfaces.

Eight ecosystem partners were identified as adopting the technology, including Crusoe, Nebius, Rafay, Spectro Cloud, and Mirantis. Nvidia therefore gains an additional distribution channel through infrastructure providers that already serve AI customers.

For Cast AI and ScaleOps, this changes the competitive environment. GPU scheduling and resource fractioning are becoming easier to obtain without buying a proprietary platform.

The independent companies still have room to compete through automatic optimization, cloud cost management, and integrations that extend beyond scheduling. But Nvidia has raised the standard considerably, and basic GPU sharing is becoming a weaker selling point on its own.

Is ScaleOps catching up with Cast AI?

ScaleOps is currently the strongest direct threat to Cast AI, with reported growth that suggests customers are adopting its autonomous infrastructure software quickly.

In March 2026, ScaleOps raised $130 million in Series C funding at a valuation exceeding $800 million. The round brought total funding above $210 million, following a $58 million Series B announced in November 2024.

The company also reported more than 450% year-over-year growth. That means the measured business activity increased by more than 5.5 times over the period, although ScaleOps did not disclose enough detail to compare absolute revenue directly with Cast AI.

Its enterprise customer references include Adobe, Salesforce, DocuSign, Wiz, and Coupa. These relationships suggest that the platform is being adopted by organizations with demanding production infrastructure.

The difference between the two startups is becoming clearer. ScaleOps emphasizes continuous, application-aware allocation of resources, including GPUs, CPUs, and memory. Cast AI combines resource optimization with an effort to source computing capacity across different providers.

ScaleOps has the stronger disclosed percentage-growth figure. Cast AI currently has broader public evidence around GPU utilization and a more explicit multi-cloud GPU sourcing strategy.

What can Cast AI do that ScaleOps can't easily copy?

Cast AI's most interesting advantage is its OMNI Compute platform, which allows Kubernetes clusters to access GPU capacity beyond their original cloud environment.

In January 2026, Cast AI announced that Oracle Cloud Infrastructure would supply GPU capacity through OMNI Compute. The platform is designed to connect computing resources across cloud providers and regions without forcing developers to rebuild their applications around each provider.

This solves a problem that scheduling alone cannot. If a company cannot obtain enough affordable H100 or H200 capacity in its current environment, a more sophisticated scheduler won't suddenly create additional GPUs.

Cast AI also supports GPU time-slicing, multi-instance GPU partitioning, and workload placement. Its product documentation describes configurations allowing up to 48 time-sharing replicas per GPU, depending on the workload.

ScaleOps could develop or integrate similar capabilities. There's nothing inherently impossible to copy here. But Cast AI currently offers a particularly broad combination of GPU allocation and external capacity sourcing.

The missing evidence is how much OMNI Compute saves customers after accounting for network costs, cloud pricing differences, and operational complexity. The architecture is promising; its advantage over competing systems hasn't yet been established through independent head-to-head testing.

Is Fireworks AI actually bigger than the GPU optimization startups?

Fireworks AI is already operating at a much larger disclosed revenue scale than the standalone GPU optimization companies.

In July 2026, Fireworks announced that it had exceeded $1 billion in annualized revenue run rate and was processing more than 40 trillion tokens daily. It also raised $1.505 billion at a $17.5 billion valuation. Fireworks passing $1 billion in annualized revenue shows what real customers look like; we separate them from promises in our AI infrastructure report.

The usage figure is particularly revealing. At 40 trillion tokens a day, Fireworks was processing more than 14 quadrillion tokens at an annualized pace, although actual yearly volume will depend on how demand changes.

More than 95% of its tokens reportedly came from models specialized for customers' proprietary data and tasks. Customers identified by the company include Cursor, which develops AI coding tools, and Harvey, which builds legal AI software.

That customer mix suggests Fireworks is increasingly handling specialized production workloads rather than relying entirely on generic chatbot traffic.

Its commercial scale is impressive, but the comparison has an important limitation. Fireworks sells inference services and must support substantial computing infrastructure, whereas Cast AI sells software designed to optimize infrastructure customers already operate.

A billion dollars of inference revenue doesn't automatically produce better margins or greater GPU efficiency than a smaller software business.

Is Fireworks AI really the fastest inference provider?

Fireworks AI is not currently the fastest provider in every independent inference test, despite its commercial success.

In September 2026, Openbenchmarks compared ten companies serving the same GLM 5.3 Flash model. Each provider received 600 requests, covering lookup, classification, and extraction tasks under a common test configuration.

Baseten recorded median end-to-end latency of 331 milliseconds, compared with 574 milliseconds for Modal and 1,954 milliseconds for Fireworks. In that test, Baseten completed the median request nearly six times faster than Fireworks.

The accuracy results told a different story. Fireworks and Telnyx achieved 99.5% task success, while Modal reached 99.3% and Baseten 95.8%. Across 600 requests, that represents approximately 597 successful tasks for Fireworks versus 575 for Baseten.

These measurements show that the fastest provider in the test was not the most accurate. The benchmark used a single model at concurrency one, so it cannot establish performance for large batches, different models, or heavily loaded GPU clusters.

We also lack equivalent GPU-hour and infrastructure-cost data. Fireworks may achieve attractive economics on other workloads, but its commercial leadership should not be confused with universal benchmark superiority.

Provider	Median latency	Median generation speed	Task success
Baseten	331 ms	240.5 tokens/s	95.8%
Modal	574 ms	222.2 tokens/s	99.3%
Together AI	609 ms	151.6 tokens/s	98.5%
Telnyx	669 ms	184.3 tokens/s	99.5%
Nebius	1,451 ms	269.3 tokens/s	84.8%
Parasail	1,590 ms	52.4 tokens/s	98.3%
Novita AI	1,738 ms	60.6 tokens/s	94.2%
Z.AI	1,749 ms	56.3 tokens/s	97.5%
DeepInfra	1,827 ms	69.4 tokens/s	95.5%
Fireworks AI	1,954 ms	58.2 tokens/s	99.5%

Could Baseten overtake Fireworks AI?

Baseten could become Fireworks AI's biggest inference competitor because its reported revenue growth is exceptional and its performance in independent benchmarks is already competitive.

According to reporting by The Information in May 2026, Baseten's annualized revenue rose from approximately $200 million to $600 million during the first quarter. That would represent a threefold increase in about three months.

The company subsequently announced a $1.5 billion financing round at a $13 billion valuation. Its disclosed operations included approximately one billion inference calls daily across 87 clusters and 18 cloud environments.

That level of activity shows Baseten has moved well beyond the early stage of testing whether companies want managed AI inference. It is already operating infrastructure supporting substantial production traffic.

Baseten also has Nvidia's backing, including a reported $150 million investment in a January 2026 financing round. That relationship provides access to an important partner in the accelerator ecosystem, although it doesn't guarantee preferential performance or pricing.

Fireworks remains ahead on the latest disclosed annualized revenue figures available in our comparison. Baseten, however, has enough scale and benchmark evidence to challenge the assumption that Fireworks will dominate specialized inference.

Has Modal become a bigger GPU efficiency player than expected?

Modal has become one of the fastest-growing AI computing startups, although a growing share of its success comes from products beyond GPU inference.

In October 2026, Modal announced that it had passed $1 billion in annualized run rate. Its revenue had grown more than fifteenfold year over year, while overall usage tripled during the preceding five months.

The company also raised $750 million at a $15.75 billion valuation. That was more than three times its $4.65 billion valuation from a financing round announced only months earlier. A valuation that more than tripled in months stands out; we look at what these companies are worth in our AI infrastructure report.

The most surprising detail is the size of Modal's Sandboxes product. Introduced in 2023, Sandboxes has reached approximately $300 million in annualized run rate, according to the company. That accounts for roughly 30% of Modal's disclosed company-wide figure.

Modal says it has launched nearly three billion sandbox instances and can scale to one million concurrent sandboxes. These environments support AI agents, generated code, reinforcement learning, and other workloads.

The company also offers serverless GPU computing and inference infrastructure, but its numbers increasingly describe a broader AI execution platform.

Modal is now a major commercial infrastructure competitor. Its growth is too large to ignore, even though its financial disclosures don't isolate the value created by GPU optimization.

Are specialized AI chips going to make GPU optimization software irrelevant?

Specialized AI accelerators could reduce demand for conventional GPUs in some inference workloads, but they won't eliminate the need to manage computing resources efficiently.

Cerebras and Groq illustrate the challenge. Cerebras uses wafer-scale processors designed for high-performance AI computing, while Groq developed an architecture focused on predictable, low-latency inference.

Both have demonstrated very high generation speeds on selected supported models. However, speed alone doesn't reveal whether a system is cheaper to operate, particularly when hardware prices, memory requirements, power consumption, and model compatibility differ.

Existing enterprise infrastructure also creates practical barriers to switching. Large parts of the AI software ecosystem rely on Nvidia's CUDA platform, and many organizations already own or lease substantial GPU capacity.

Even if specialized accelerators capture a growing share of inference, companies will still need scheduling, capacity allocation, and workload management. The underlying optimization problem remains.

The more immediate threat to Cast AI and ScaleOps comes from increasingly capable open-source scheduling tools, including Nvidia's recently released Run:ai components. Those tools could make basic GPU sharing easier for customers to implement themselves.

Can GPU efficiency startups make money when open-source tools are improving?

GPU efficiency software can support an attractive business when it saves customers substantially more than it costs, but open-source competition is making that value harder to prove.

Consider a company spending $10 million annually on GPU infrastructure. Reducing that bill by 20% would free $2 million per year. A vendor charging several hundred thousand dollars for the software could still offer a compelling return. Charging a fraction of the savings is one way to earn here; we compare which business models are winning in our AI infrastructure report.

The actual financial result depends on more than the advertised savings percentage. Customers need to account for performance changes, engineering costs, migration work, software fees, and the risk of interrupting important applications.

Open-source tools create a lower-cost alternative. Kubernetes already supports several workload-management frameworks, and Nvidia's release of Run:ai adds GPU scheduling and fractioning capabilities that engineering teams can adopt directly.

Proprietary vendors therefore need to do substantially more than redistribute existing open-source features. Cast AI's multi-cloud sourcing and ScaleOps' automatic resource management are examples of capabilities that could justify ongoing software spending.

There is also a business-model difference worth keeping in mind. Fireworks and Baseten earn revenue from processing customer workloads, while Cast AI and ScaleOps primarily sell software designed to reduce infrastructure waste. Without comparable gross-margin disclosures, we cannot determine which approach produces better profits.

What would prove that Cast AI is really the most efficient?

Cast AI would establish a convincing technical lead by showing that customers consistently use fewer paid GPU-hours than they would with ScaleOps, Run:ai, or a well-configured open-source alternative.

The comparison would need to use identical workloads, accelerator types, latency requirements, and cloud prices. Otherwise, a vendor could appear more efficient simply because it ran smaller models or accepted slower responses.

For example, reducing a cluster from 100 GPUs to 60 while maintaining the same successful request volume and response times would represent a 40% reduction in provisioned capacity. That would be stronger evidence than reporting a rise in utilization alone.

Production results should also cover several weeks rather than a short demonstration. Customer demand varies throughout the day, and savings achieved during quiet periods might disappear during peak traffic.

Cast AI has disclosed useful utilization baselines and customer examples. Run:ai has published detailed GPU-sharing benchmarks with Nebius. ScaleOps has reported rapid growth and enterprise adoption.

What we still don't have is an independent comparison combining all three under the same conditions. That limits how confidently we can rank their actual technical efficiency, even if Cast AI currently offers the strongest overall case among independent optimization specialists.

GPU efficiency: which startup is actually ahead?

Cast AI is currently our pick for the leading independent GPU efficiency startup, with ScaleOps close behind. Fireworks AI is ahead in the commercial scale of optimized inference, while Nvidia-owned Run:ai remains the technical benchmark for GPU orchestration.

Cast AI earns the top independent software position because it combines GPU sharing, automatic resource allocation, and multi-cloud compute sourcing in a single offering. Its enterprise customer base and detailed utilization research strengthen that case, although independent comparisons of customer savings remain limited.

ScaleOps has a credible opportunity to overtake it. The company's rapid reported growth and strong enterprise adoption suggest that autonomous resource management is attracting serious demand.

Fireworks is already operating a much larger disclosed revenue business, and Baseten and Modal have established themselves as major competitors in AI computing. Their success shows that much of the commercial opportunity lies in running models efficiently, beyond simply managing idle GPUs.

For now, the evidence supports Cast AI as the independent GPU optimization leader. We would change that judgment if ScaleOps demonstrated stronger production savings across comparable workloads, or if independent benchmarks showed that another platform consistently delivered more successful AI work per dollar.