How do companies in the AI safety market make money?
In our AI safety market deck, you will find everything you need to understand the market
SUMMARY
Companies in the AI safety market make money by selling repeatable protection, testing, monitoring, governance and assurance around AI systems, usually through enterprise software, usage-based runtime fees, paid evaluations, advisory work and government or research contracts.
The strongest businesses are attached to budgets that already exist. CISOs can pay for AI security, risk teams can pay for governance, engineering teams can pay for agent controls, and frontier labs can pay for external evaluations; very few customers need a standalone budget called “AI safety.”
Runtime protection currently has the cleanest expansion model because the product is used whenever an AI system acts. More AI traffic, tool calls or agent activity can translate directly into more billable usage without another full sales cycle.
Evaluation becomes more attractive when it stops being a one-off report. Vendors such as Patronus AI and Gray Swan are pushing testing into continuous simulation, automated adversarial testing and repeated release workflows, which gives them a path toward recurring revenue.
Governance software is strongest when it speeds up AI deployment as well as producing compliance evidence. If the platform only creates paperwork, it is easier to cut; if it controls inventories, approvals, policies and audit history, it becomes part of how the company operates AI.
AI agents are raising the value of safety products because mistakes now have operational consequences. A model that can execute code, call APIs, move data or change infrastructure gives security teams a clearer reason to buy controls before and during execution.
The market is commercially real but still small next to mainstream cybersecurity. Specialist vendors have reached eight-figure annualized revenue and seven-figure contracts, yet some strategically important companies were acquired while still producing less than $10 million a year.
That gap explains the acquisition wave. Palo Alto Networks, Cisco, Check Point, F5 and Fortinet are buying model scanning, red teaming, runtime guardrails and agent-security capabilities before customers standardize on independent vendors.
Open source does not kill the category, but it does compress the value of narrow features. Pricing power sits higher up the stack in deployment, monitoring, private infrastructure, enterprise controls, integrations, support and accumulated failure data.
Pure frontier alignment research still has a different economic model. Philanthropy, governments and AI labs can finance work that has no obvious software buyer, while commercial businesses appear when that research becomes a test, monitor, control or production requirement.
The durable companies will own something customers cannot casually remove: unique adversarial data, a deep governance record, a trusted evaluation workflow or a runtime control point sitting between AI agents and the systems they can affect. The money follows safety work that has to happen again tomorrow.
This market map, featured in our AI safety market deck, highlights top companies and startups in the AI safety market
What do AI safety companies actually sell today?
AI safety companies currently make money by selling security, evaluation, governance and assurance around AI systems; “AI safety” itself is rarely the line item on the invoice.
The category covers several businesses that look very different once we follow the money. HiddenLayer protects AI models, applications and agents from attacks. Gray Swan tries to break models and agents before attackers do, then sells automated testing and runtime protection. Credo AI helps large companies keep track of AI systems, policies, approvals and risks. Apollo Research works much closer to frontier safety research but is also turning that research into monitoring software for coding agents.
Those companies all reduce AI risk, but customers are buying different outcomes. A security team wants to stop prompt injection or data leakage. A bank wants to know which AI systems are running and who approved them. An AI lab wants an external team to find dangerous behavior before a model launch. An engineering organization wants to catch a coding agent before it deletes data or runs a dangerous command.
For the rest of this analysis, we therefore use “AI safety market” to mean companies that get paid to find, measure, prevent, monitor or document dangerous and unwanted AI behavior. That keeps the commercial market separate from safety research that still depends mainly on philanthropy or internal lab budgets.
Who actually pays AI safety companies?
The people paying AI safety companies today are usually CISOs, AI platform teams, risk teams, regulated business units and frontier AI labs.
The easiest sales happen when an existing executive already owns the problem. A CISO understands an AI system leaking confidential information. A risk team understands an unapproved model making decisions in a regulated workflow. An AI platform team understands why an agent with access to code, databases and external tools needs controls.
Large AI developers form a smaller but particularly valuable customer group. Forbes reported recently that frontier labs still generate most of Gray Swan's revenue, even though the company now has around 20 enterprise customers. OpenAI was Gray Swan's first customer shortly after the company launched, and Gray Swan has since worked with Anthropic, Google DeepMind, Meta, xAI and ByteDance.
Government can pay too. AI assurance companies win public-sector contracts, while safety researchers work with national AI institutes and defense organizations. HiddenLayer, for example, has expanded into US government work alongside its commercial business.
What we do not see yet is a large, standardized corporate budget literally called “AI safety.” Money usually comes from cybersecurity, AI infrastructure, compliance, risk or research budgets. The vendor gets paid when it can attach safety to a problem somebody already has to solve.
As this chart shows, and as featured in our AI safety market deck, search interest in AI safety has been growing steadily
Are companies spending real money on AI safety yet?
Enterprise AI safety spending is already real, but revenue is still concentrated in a small group of vendors and a handful of large contracts.
The clearest public numbers came from The Information's investigation of specialist AI security vendors. HiddenLayer eventually passed $10 million in annualized revenue after signing several seven-figure contracts. Robust Intelligence was doing around $9 million in annualized revenue when Cisco agreed to acquire it. At the same time, Lakera and CalypsoAI had each generated less than $5 million in revenue.
Those numbers are useful because they stop us from confusing high startup valuations with a mature software category. A company can be strategically important enough to sell for hundreds of millions of dollars while still producing less than $10 million a year.
There are signs that the revenue ceiling is moving quickly. Credo AI said its software revenue roughly tripled in 2024 while its customer base doubled and platform retention remained at 100%. The company then reported another 2× increase in total revenue in 2025 and 150% growth in enterprise customers.
Still, we should keep the scale in perspective. Today, most specialist AI safety companies remain tiny next to established cybersecurity vendors. The interesting part is the direction: we are moving from experimental purchases worth tens of thousands of dollars toward recurring contracts and occasional seven-figure deals.
How do AI safety companies actually charge customers?
AI safety companies mainly charge through annual software contracts, usage-linked fees, paid evaluations, professional services and government or research contracts.
Traditional enterprise software is the easiest model to understand. A company signs an annual contract for an AI governance, security or monitoring platform. Pricing usually depends on factors such as the number of AI systems, applications, users or environments being covered. Public marketplace listings from vendors including Credo AI show annual packages with usage allowances and additional charges above those limits.
Runtime protection introduces a more interesting model because revenue can move with AI usage. Guardrail vendors can charge by request, characters processed, events inspected or another measure of model activity. If the customer's AI traffic doubles, the safety bill can grow without the vendor having to win the customer again.
Evaluations and red teaming look more like specialist services. A frontier lab might pay a company to attack a new model before release. A bank might commission a test of an agent handling financial workflows. These projects can become recurring when every major model or agent update needs another round of testing.
Consulting remains part of the market too. Credo AI said its advisory engagements grew fivefold in 2025. That makes sense at this stage: many large companies still need somebody to help decide what should be governed before software can automate the process. The better business eventually moves as much of that repeated work as possible into the platform.
| How the company charges | What the customer is buying | Revenue behavior | Examples |
|---|---|---|---|
| Annual software contract | Governance, security, monitoring | Recurring | Credo AI, HiddenLayer |
| Usage-based fees | Runtime checks and guardrails | Grows with AI activity | Guardrail and monitoring vendors |
| Paid evaluations | Model or agent testing | Repeated around releases | Gray Swan, Patronus AI |
| Advisory and assurance | Human expertise and evidence | Higher labor component | Governance and assurance firms |
| Government or research contracts | Testing, research, technical work | Often large but less predictable | Specialist safety organizations |
This chart, featured in our AI safety market deck, shows annual venture capital investment in AI safety startups
Why are AI agents making AI safety more valuable right now?
AI agents are making AI safety more valuable because agents can now take actions inside real systems, so a bad output can become a bad transaction, code change or data transfer.
A chatbot saying something stupid is mostly an output problem. A coding agent can read private repositories, execute commands, install packages and edit infrastructure. A business agent may call APIs, access customer records or trigger workflows. Once AI moves from answering to acting, companies need controls while the system is running.
The spending forecasts are starting to reflect that change. Gartner currently expects products for securing AI ecosystems and AI agents to grow from $2.8 billion in 2026 to $16.4 billion by 2030. That works out to roughly 56% annual growth. The estimate covers a broader category than pure AI safety startups, but that is exactly the point: AI safety is beginning to enter normal cybersecurity spending.
The latest acquisitions also point toward agents. Fortinet recently bought Virtue AI, whose technology tests autonomous agents across more than 50 sandboxed environments and 14 high-stakes domains, scans MCP tools and source code, monitors agent behavior and can block malicious tool calls before execution. Fortinet explicitly described the deal as part of its push into continuous protection for agentic AI.
AI safety becomes much easier to monetize when a system can actually do damage. These days, that boundary is moving quickly.
Are runtime AI guardrails the best AI safety business model right now?
Runtime AI guardrails currently have the cleanest recurring-revenue model in AI safety because the product gets used every time an AI system acts.
A red-team report can be valuable, but the engagement eventually finishes. Runtime protection stays in the traffic path. Every prompt, tool call or agent action gives the safety vendor another event to inspect and potentially another unit to bill.
Apollo Research shows how quickly frontier-safety work can move in this direction. Its Watcher product monitors coding agents such as Claude Code and Codex, reviews what they are doing and can block dangerous actions before execution. Apollo currently says Watcher is monitoring billions of agent tokens every month across production engineering teams.
CalypsoAI followed a similar route before F5 acquired it. The product combined automated red teaming with inference-time protection. Lakera built Lakera Red for testing and Lakera Guard for runtime protection before Check Point acquired the company.
There is a catch. Runtime safety products sit directly between the AI system and the task the customer wants completed. Too many false alarms, noticeable latency or an outage can turn the safety product into the problem. That raises the technical bar considerably.
The winners here will probably be the products customers stop thinking about. Once a guardrail quietly becomes part of every AI request, switching it off feels more dangerous than continuing to pay for it.
If you want more recent data on this point, please see our latest AI safety market report.
This chart, featured in our AI safety market deck, shows how HiddenLayer is positioned in AI safety
Can AI evaluation companies turn testing into recurring revenue?
AI evaluation companies can build recurring revenue when testing becomes part of every model release, agent update and production workflow.
Patronus AI provides unusually fresh evidence. TechCrunch recently reported that Patronus revenue had grown 15× over the previous year. The company then raised a $50 million Series B, bringing total funding to $70 million, as it expanded from conventional model evaluation into simulated environments where AI agents can be tested on long, messy tasks.
That shift is commercially important. Static benchmarks can be run occasionally. A realistic simulation platform can be used during training, after training, before deployment and whenever the agent changes. Patronus says virtually every frontier AI lab and many newer AI companies are already customers.
Gray Swan is moving in the same direction from a security angle. Its Arena now includes around 15,000 security professionals who attack AI systems. The company uses what those humans discover to improve Shade, its automated adversarial-testing agent, while Cygnal handles runtime protection. Forbes recently reported that Gray Swan had reached roughly 20 enterprise customers alongside its frontier-lab business and raised $40 million at a reported $200 million valuation.
Testing becomes a much better business once it stops producing only a report. The valuable asset is a system that keeps finding new ways for models and agents to fail.
Do OpenAI, Anthropic and other frontier AI labs really pay outside safety companies?
OpenAI, Anthropic and other frontier AI labs are already paying outside safety companies for red teaming and evaluations, although only a small number of vendors currently have enough credibility to win that work.
Gray Swan gives us the clearest commercial example. OpenAI became its first customer shortly after the startup launched. The company's work has since appeared in 11 frontier-model system cards, and Forbes reports that frontier labs currently account for most of Gray Swan's revenue.
External evaluators can bring attacks and methods that an internal safety team has not tried. More importantly, the relationship repeats. OpenAI, Anthropic, Google DeepMind and other labs keep releasing new models, changing post-training methods and adding new capabilities. Each major change can create another evaluation cycle.
Independent organizations also evaluate frontier models without using a normal vendor relationship. METR, for example, has conducted predeployment evaluations of major frontier systems, including a recent evaluation of GPT-5.6 Sol, while deliberately avoiding funding from frontier AI companies.
That distinction is useful. Frontier labs clearly create demand for outside safety work, but several funding models can serve it. Commercial firms take customer revenue. Independent nonprofits can use philanthropy. Governments can pay for evaluations they want performed independently of the developer.
This chart, featured in our AI safety market deck, shows annual funding in AI safety startups
Is AI governance software basically a compliance product?
AI governance software is becoming an operating tool for large companies, with regulation acting as an accelerant rather than the whole reason customers buy.
Credo AI's own growth gives us a useful test. The company said platform revenue grew roughly 3× in 2024, followed by 2× overall revenue growth in 2025. More importantly, Credo said regulatory uncertainty was not the main reason customers were buying. Companies were struggling to keep track of third-party AI, approve new use cases and produce evidence about AI systems already running inside the business.
Its reported customer results fit that explanation. Credo says customers reduced AI use-case review time by 70% and manual compliance work by 60%. Those are company-reported figures, so we should treat them accordingly, but they describe a much more concrete purchase than “help us be responsible.”
Regulation still brings money into the category. Gartner currently expects dedicated AI governance platform spending to reach $492 million in 2026 and exceed $1 billion by 2030. As more rules require inventories, risk classifications, testing and documented controls, spreadsheets become harder to defend.
The better governance products therefore make two departments happy at once. Compliance gets the evidence it needs, while AI teams get approvals through faster. A product that only produces regulatory paperwork will have a much harder time defending its budget.
If you want more recent data on this point, please see our latest AI safety market report.
How big is the AI assurance market today?
The AI assurance market is already large enough to support hundreds of suppliers, although much of the money still comes from consulting and technical services.
The UK offers one of the best bottom-up measurements we have. A government-backed study counted 524 companies supplying AI assurance products and services and estimated £1.01 billion in gross value added. Of those, 84 were specialist AI assurance businesses with about £360 million in GVA.
We can extract something more interesting from those numbers. Specialists represented around 16% of the suppliers but generated roughly 36% of the market's GVA. Average GVA per specialist was therefore about £4.3 million, more than twice the roughly £1.9 million average across all suppliers.
The same research found that demand was strongest in sectors that already live with formal risk controls, including financial services, life sciences and pharmaceuticals. That makes intuitive sense. An industry that already pays people to test, audit and sign off important systems has a much easier time adding AI to the process.
The UK government now believes the domestic AI assurance sector could eventually reach £18.8 billion in GVA by 2035 if adoption barriers fall. We would treat that forecast cautiously, but today's £1 billion base is measured economic activity rather than a hypothetical future market.
| UK AI assurance market | All suppliers | Specialist firms | Specialist share |
|---|---|---|---|
| Companies | 524 | 84 | ~16% |
| Gross value added | £1.01B | £0.36B | ~36% |
| Approx. GVA per company | £1.93M | £4.29M | ~2.2× average |
This chart, featured in our AI safety market deck, compares the main business model options for AI alignment research labs
Does open-source AI safety software make it hard to charge customers?
Open-source AI safety tools make narrow features harder to charge for, but they do not kill the business model when the vendor owns deployment, monitoring and enterprise controls.
Protect AI is a good example. The company released and supported open tools around model scanning and AI security while selling a broader enterprise platform. Palo Alto Networks later paid $634.5 million in total consideration for Protect AI, according to its SEC filing.
The logic is similar to other parts of cybersecurity. A developer may get a scanner or evaluator for free, but a large company still needs central policies, access controls, dashboards, private deployment, integrations, audit history, support and somebody accountable when the system fails.
Open source can actually help early in a new category. Engineers can test the product before procurement gets involved. The tool gets integrated into workflows. The vendor sees how customers really use it.
Pricing power becomes weak when the paid product is barely more than the open-source component. Charging gets much easier once the product controls how hundreds of AI systems are tested, approved and monitored.
If you want more recent data on this point, please see our latest AI safety market report.
Why are cybersecurity giants buying AI safety startups?
Cybersecurity giants are paying large strategic prices for AI safety startups because they want these capabilities inside existing security platforms before customers standardize elsewhere.
Palo Alto Networks' Protect AI deal is the biggest disclosed example we found: $634.5 million in total purchase consideration. Cisco's acquisition of Robust Intelligence was estimated by 451 Research at roughly $350 million. Check Point disclosed about $190 million of net cash consideration for Lakera. F5 paid $145.2 million in cash when it closed the CalypsoAI acquisition.
The revenue comparison makes those prices more revealing. Robust Intelligence reportedly had around $9 million in annualized revenue near its sale. CalypsoAI's revenue was small enough for F5 to describe it as immaterial to the company's overall results. These acquirers were clearly buying more than current cash flow.
Fortinet has now joined the same race with its acquisition of Virtue AI, adding automated red teaming, agent monitoring, MCP security and runtime guardrails. Financial terms were undisclosed, and Fortinet said the consideration was immaterial to its business.
Taken together, these deals tell us where the large security companies expect customer spending to move. Model scanning, agent security, red teaming and AI guardrails are being pulled into the same platforms that already sell firewalls, application security and cloud protection.
| AI safety/security startup | Buyer | Disclosed or reported deal value | Main capability acquired |
|---|---|---|---|
| Protect AI | Palo Alto Networks | $634.5M total consideration | AI model and application security |
| Robust Intelligence | Cisco | ~$350M 451 Research estimate | AI validation and protection |
| Lakera | Check Point | ~$190M net cash consideration | Red teaming and runtime guardrails |
| CalypsoAI | F5 | $145.2M cash at closing | Red teaming and inference protection |
| Virtue AI | Fortinet | Undisclosed | Agent validation and runtime protection |
This chart, featured in our AI safety market deck, shows revenue breakdown by customer segment in the AI safety market
Will standalone AI safety startups survive the cybersecurity giants?
Many AI safety startups will probably disappear into broader cybersecurity platforms, especially vendors whose entire product is one guardrail, scanner or red-team feature.
The customer has a simple reason to accept consolidation. A large company may already use Palo Alto Networks, Cisco, Check Point, Fortinet or F5. Buying another specialist means another vendor review, another contract, another integration and another security console.
The products are already being absorbed. Cisco has turned Robust Intelligence technology into part of its broader AI Defense offering. F5 integrated CalypsoAI into its application and security platform. IDC went as far as describing generative-AI red teaming as becoming a platform feature after the CalypsoAI transaction.
There will still be room for independent companies, but they need more than a good detector. A specialist can survive if it owns unusual data, has deep credibility with frontier labs, controls a workflow that security suites do not handle well or keeps moving technically faster than the incumbents.
This makes AI safety an unusual startup market. The category can grow very fast while the number of independent companies shrinks.
If you want more recent data on this point, please see our latest AI safety market report.
Can pure frontier AI alignment research become a real business?
Pure frontier AI alignment research still struggles to behave like a normal software business, but the line between research and product is getting thinner.
METR shows why commercial revenue is not always the natural answer. The nonprofit recently raised around $71 million in funding commitments over six months for work on autonomous capabilities, recursive self-improvement, monitoring systems, risk assessments and AI incidents. That is serious funding, yet it comes from philanthropic supporters rather than customers buying software subscriptions.
Apollo Research chose another route. The organization moved into a Public Benefit Corporation after saying it was seeing quickly growing market demand for frontier AI safety research and evaluations and believed that demand would be difficult to serve at scale as a nonprofit.
Apollo then created a separate product effort around Watcher, its coding-agent monitoring system. As we saw earlier, Watcher is already processing billions of agent tokens each month. The research into scheming, deception and monitoring now feeds something engineering teams can actually deploy.
That gives us a fairly clean dividing line. Research into a possible future failure can be extremely valuable without having a natural customer. Once that research becomes a test, a monitor, a deployment requirement or a control that runs inside production, a conventional business starts to appear.
This chart, featured in our AI safety market deck, shows how prompt injection defense platform technology has evolved over time
What makes an AI safety company hard to replace?
The hardest AI safety companies to replace will own unique failure data, deep workflow integration or a control point that customers cannot easily remove.
Gray Swan is building around data. As we saw previously, its Arena has roughly 15,000 people trying to break AI systems. Those attacks can feed the company's automated testing tools. A competitor can build another scanner, but reproducing years of unusual adversarial examples is harder.
Governance platforms can build a different kind of lock-in. If a company stores years of AI inventories, approvals, risk decisions, exceptions, policies and audit evidence in one system, moving to a competitor becomes painful even when the underlying software is technically replaceable.
Runtime security creates perhaps the strongest position. Once a safety product sits between an agent and its tools, the product becomes part of the production architecture. Removing it means changing integrations and accepting a period with less protection.
We therefore see much stronger businesses around accumulated data and embedded control than around isolated safety algorithms. Individual classifiers, jailbreak detectors and benchmarks will improve quickly and many will become free. The surrounding system is where durable revenue can remain.
So how do companies in the AI safety market make money?
AI safety companies make money today by turning AI risk into something a customer can buy repeatedly: protection, testing, monitoring, governance or proof.
The strongest commercial models are already clear. Enterprise security and governance platforms sell annual contracts. Runtime guardrails can expand with AI usage. Evaluation companies get paid each time models and agents need to be tested. Assurance firms sell independent evidence. Specialist teams still make money from advisory work when customers need humans to solve problems the software cannot handle yet.
Frontier safety research follows a different path when the person benefiting from the work cannot easily be turned into a customer. Philanthropy, governments and AI labs can finance that research until part of it becomes a product. Apollo's move from scheming research into coding-agent monitoring is a good example of that transition.
The market is also being pulled rapidly toward mainstream cybersecurity. Large security companies have spent hundreds of millions of dollars buying specialist AI safety technology, and the latest deals are increasingly focused on agents, continuous testing and runtime control. That makes it harder for narrow startups to stay independent, but it strengthens the case that companies will keep spending on the underlying problem.
Our final judgment is fairly sharp. AI safety already works as a business when a company can connect a specific AI failure to somebody's budget and then protect against that failure repeatedly. Runtime security, continuous evaluation and enterprise governance currently fit that test best. Pure alignment research still depends much more heavily on research funding.
The money is following the places where AI safety has to happen again tomorrow.
If you want more recent data on this point, please see our latest AI safety market report.
In our AI safety market deck, we identify pain points entrepreneurs should prioritize
OUR METHODOLOGY
There is no single metric that tells us whether AI safety is becoming a real commercial market, or which business models are actually working. Revenue disclosures are scarce, companies define the category differently, and fast-moving areas such as AI agents can change the economics of the market within months.
We therefore broke the question into a set of commercial dimensions: who pays, what customers buy, how vendors charge, whether spending repeats, how revenues are developing, where large contracts are appearing, what established cybersecurity companies are acquiring, and which products are becoming embedded in production.
For each dimension, we prioritized the freshest evidence we could verify and gave the most weight to real commercial behavior: disclosed revenue, customer growth, recurring contracts, usage in production, acquisition prices, buyer behavior and measured market activity. Funding and valuations were useful context, but we did not treat them as evidence of customer demand on their own.
We assessed those signals individually and then compared them across companies, business models and buyer types. A revenue figure can show that customers are paying without proving that the model scales. An acquisition can show strategic value without proving current market size. A production-usage metric can show adoption without proving pricing power. The conclusions above are based on where those different forms of evidence converge.
We also separate commercial AI safety from frontier safety research that still relies mainly on philanthropy, government funding or internal AI-lab budgets. That distinction is important because research can be valuable without having a natural software customer; it enters the commercial market once it becomes a repeatable test, monitor, control, assurance process or production requirement.
Key sources for the commercial and governance evidence include Credo AI's 2024 business update, Credo AI's 2025 year in review, and Gartner's AI governance platform spending forecast.
For agent monitoring, evaluation and frontier-lab demand, we used Apollo Research on Watcher, Apollo Research's production-usage disclosures, Gray Swan on its Arena network, Forbes on Gray Swan's customers and revenue mix, Patronus AI's Series B announcement, and TechCrunch on Patronus AI's reported revenue growth.
For assurance and non-commercial frontier research, we relied on the UK government's bottom-up study of the AI assurance market and METR's August 2026 funding update.
For acquisition values and the move of AI safety into mainstream cybersecurity platforms, key sources include Palo Alto Networks on Protect AI, Palo Alto Networks' annual reporting, Cisco and 451 Research on Robust Intelligence, Check Point's filing on Lakera, F5's SEC filing on CalypsoAI, and Fortinet on its acquisition of Virtue AI.
This chart, featured in our AI safety market deck, shows revenue breakdown by geography across Europe, Asia, North America, Africa, and South America in the AI safety market
