Is the AI Safety Market growing now?
In our AI safety market deck, you will find everything you need to understand the market
SUMMARY
Yes, the AI Safety Market is growing now. The strongest commercial growth is already visible in AI security, model evaluation, monitoring and governance around production AI systems.
The market is not growing evenly. Agent security is commercializing fastest because autonomous systems create familiar enterprise problems around identity, permissions, credentials, data access and runtime control.
Customer spending is becoming easier to verify. WitnessAI reported annual recurring revenue growth above 500%, while companies such as Obsidian already manage large six- and seven-figure security relationships that increasingly include AI-agent controls.
Fresh capital is clustering around the same problem. Five relevant companies announced about $480 million in financing over roughly five weeks, with most of the money aimed at securing agents or AI-powered software rather than abstract alignment research.
M&A is an even stronger proof that the category is moving into established enterprise budgets. Five disclosed deals across AI security and evaluation add up to roughly $1.97 billion, and the buyers are integrating those products into major cybersecurity and observability platforms.
Evaluation is shifting from a specialist pre-launch exercise into continuous production infrastructure. Promptfoo, Arize and similar tools increasingly sit alongside observability because models, prompts, retrieval systems and agent workflows change too often for one-off testing to be enough.
Regulation is creating work now, but the spending wave will be staggered. General-purpose AI and transparency requirements are already active in Europe, while important high-risk obligations arrive later in 2027 and 2028.
AI governance is becoming software, but it will not be a pure software market. Internal risk teams, lawyers, consultants and existing cloud or security vendors will absorb a large share of spending alongside specialist governance platforms.
Frontier AI safety is splitting into two economic models. Red teaming, cyber evaluation and safeguard testing can already be sold as services or software, while alignment research is attracting more funding without yet looking like a normal recurring-revenue market.
Governments are becoming buyers and funders too. The UK is financing dedicated frontier-model testing and alignment work, while the US is pushing evaluation methods closer to federal procurement and deployment.
The likely outcome is a larger AI safety function with fewer independent vendors than today. Security, observability and infrastructure companies are already absorbing specialist startups, so the market can expand quickly even as standalone categories blur together.
This market map, featured in our AI safety market deck, highlights top companies and startups in the AI safety market
What actually counts as the AI safety market today?
The AI safety market today is best defined as the paid layer that tests, monitors, governs and protects AI systems before and after they go into production.
We include four businesses when we measure it: AI security, model evaluation and monitoring, AI governance and compliance, and frontier safety work such as dangerous-capability testing and alignment. They share a common buyer problem: an AI system can behave unpredictably, leak data, break policy, get manipulated or take actions outside the boundaries its owner set.
We keep ordinary cybersecurity outside the market when AI is only being used as a defensive tool. A security company using machine learning to detect malware still sells cybersecurity. A company preventing prompt injection, testing whether an agent can abuse its tools, checking model outputs for unsafe behavior or keeping an audit trail for AI regulation belongs much closer to AI safety.
Published market estimates get messy at that boundary. Some include consulting, privacy software and standard risk-and-compliance tools; others pull in every cybersecurity product that mentions AI. We focus on products and services where controlling AI behavior is part of what the customer is actually paying for.
| Part of the AI safety market | What customers buy | Typical buyer | Where it stands now |
|---|---|---|---|
| AI security | Agent security, prompt-injection defense, shadow-AI discovery, data-leak prevention | CISO, security team | The most commercial part today |
| Evaluation and monitoring | Red teaming, model evaluations, hallucination and behavior monitoring | AI engineering, platform teams, frontier labs | Growing quickly |
| Governance and compliance | AI inventories, policies, audit trails, risk controls, regulatory evidence | Legal, risk, compliance, security | Growing, with lots of services around it |
| Frontier safety | Dangerous-capability testing, control, alignment research | Frontier labs, governments, research funders | More money is flowing in, but the commercial market is still smaller |
If you want more recent data on this point, please see our latest AI safety market report.
Why is AI safety becoming a real business right now?
AI safety is becoming a real business now because companies have deployed AI much faster than they have built the controls around it.
McKinsey's latest global state-of-AI survey found that 88% of organizations were regularly using AI in at least one business function, up from 78% a year earlier. Generative AI alone had reached 79%. Yet only 7% said AI was fully scaled across the organization, while the rest were still experimenting, piloting or scaling.
Companies are stuck in a messy middle. AI already touches enough employees, customer workflows, code, data and internal systems to create real risk, while most companies are still working out who owns the controls and how those controls should work.
The failure data is moving in the same direction. Stanford's latest AI Index counted 362 documented AI incidents in 2025, up from 233 a year earlier, a rise of roughly 55%. Stanford also found that responsible-AI benchmarking is improving more slowly than AI capabilities and deployment.
The demand is fairly easy to see now: more AI systems are entering real workflows, more things can go wrong, and security and reliability teams are being asked to catch up.
As this chart shows, and as featured in our AI safety market deck, search interest in AI safety has been growing steadily
Are companies actually paying for AI safety today?
Yes, companies are already paying real money for AI safety products, although the clearest spending is still concentrated in larger enterprises and security-heavy industries.
WitnessAI gives us a clean example because it was built specifically around enterprise AI security. The company said its annual recurring revenue grew more than 500% over the previous 12 months before it raised $58 million, with production customers across financial services, utilities, airlines, automakers, retail and telecoms.
Obsidian Security gives us a sense of how large these security contracts can already get. The company says more than 100 customers now spend over $100,000 a year with it and more than 14 spend over $1 million. Obsidian's platform covers broader SaaS and identity security as well as AI agents, so only part of that spending belongs inside AI safety. Even with that caveat, the disclosure shows that teams buying agent controls already manage six- and seven-figure security relationships.
Alice, formerly ActiveFence, says the company is approaching $100 million in annual recurring revenue while expanding from internet trust and safety into adversarial testing and guardrails for AI models. That figure needs some care because Alice inherited a mature pre-AI business. Even so, AI safety is now being sold through companies with substantial existing revenue, not only through research projects and pilots.
Stanford's latest AI Index points in the same direction inside enterprises: the share of businesses with no responsible-AI policy fell from 24% to 11%, while AI-specific governance roles increased. A policy creates no software revenue by itself, but it shows that ownership is becoming formal enough for budgets and procurement to follow.
Where is the fresh AI safety money going right now?
Most of the fresh AI safety money is currently going into AI security, especially products that control agents and AI-powered software.
We found five clearly relevant companies that announced a combined $480 million in new financing over roughly five weeks. Four are centered mainly on securing AI agents or AI-powered software. The fifth, Alice, spans model testing, runtime guardrails and a broader trust-and-safety business.
The timing is more interesting than any single round. Neo came out of stealth with $100 million. Zenity raised $125 million to secure autonomous agents. Obsidian raised $85 million as its agent-security push accelerated. Mindgard added $30 million for offensive AI security testing. Alice has just raised $140 million for model stress testing and runtime protection.
Nearly half a billion dollars in little more than a month is large for a category that barely had an agreed name a few years ago. The rounds also show where investors think customers will spend first: around agents that can access applications, credentials, code, corporate data and external tools.
| Company | Fresh financing | What it is selling into the AI safety market |
|---|---|---|
| Alice | $140M | Model red teaming, adversarial testing and runtime guardrails |
| Zenity | $125M | Security and governance for AI agents |
| Neo | $100M | Control layer for AI agents and AI-enabled software |
| Obsidian Security | $85M | Agent identity, access and third-party application security |
| Mindgard | $30M | Offensive testing and protection of models, agents and AI apps |
| Combined | $480M | Five rounds over roughly five weeks |
If you want more recent data on this point, please see our latest AI safety market report.
This chart, featured in our AI safety market deck, shows annual venture capital investment in AI safety startups
Are big companies actually buying AI safety startups?
Yes, big security and infrastructure companies are buying AI safety startups aggressively enough that we can already measure the activity in billions of dollars.
We can verify roughly $1.97 billion spent or committed across five deals alone. Palo Alto Networks paid $634.5 million for Protect AI. CrowdStrike's filings put consideration for Pangea at roughly $222.7 million. F5 paid $145.2 million for CalypsoAI and another $50.1 million for SurePath AI. Dynatrace recently agreed to acquire AI evaluation and observability company Arize for $915 million.
Those five transactions cover different parts of the stack: model security, AI detection and response, red teaming, shadow-AI discovery and evaluation. Several additional AI-security acquisitions had undisclosed prices, so the disclosed total understates the amount of capital changing hands.
The buyers are also telling us what they plan to do with the assets. F5 combined its acquisitions into a dedicated AI Security Platform. CrowdStrike is extending Falcon into AI application security. Dynatrace expects Arize to add around two percentage points to annual recurring revenue growth in its next fiscal year. Established vendors are putting these assets into products and revenue plans, which makes the deals much more than quiet talent hires.
| Buyer | Target | Main AI safety capability | Verified or announced value |
|---|---|---|---|
| Palo Alto Networks | Protect AI | AI and ML security | $634.5M |
| CrowdStrike | Pangea | AI detection and response | ~$222.7M |
| F5 | CalypsoAI | Red teaming and runtime AI security | $145.2M |
| F5 | SurePath AI | Shadow-AI discovery and governance | $50.1M |
| Dynatrace | Arize | AI evaluation and observability | $915M |
| Combined | ~$1.97B |
Are companies actually paying to evaluate AI models?
Yes, companies are currently paying for AI model evaluation as part of normal production work, and the category is moving well beyond specialist pre-launch testing.
OpenAI's agreement to acquire Promptfoo gives us one unusually concrete adoption number. Promptfoo says more than 350,000 developers have used its tools, around 130,000 are active each month, and teams at more than 25% of the Fortune 500 rely on the platform for LLM evaluation and red teaming. OpenAI plans to integrate that technology into Frontier for agent security, evaluation and compliance.
Irregular shows that frontier-model evaluation can generate revenue on its own. The company says it already earns millions of dollars annually while testing advanced cyber capabilities for organizations including OpenAI, Anthropic and government partners. Its evaluation work has appeared in documentation for major frontier models.
Dynatrace's $915 million agreement for Arize pushes the same category deeper into mainstream enterprise software. Arize monitors hallucinations, output quality, traces and agent behavior before and after deployment. Dynatrace is paying to connect that work with the observability systems that already run production applications.
Evaluation is moving from occasional pre-launch testing toward continuous checking. Once an AI application changes models, prompts, retrieval sources, tools and agent workflows every few weeks, a one-time safety review becomes much less useful.
This chart, featured in our AI safety market deck, shows how HiddenLayer is positioned in AI safety
Are AI agents making the AI safety market grow faster?
Yes, AI agents are making AI safety spending more urgent because they can take actions inside real systems instead of only generating answers.
Cisco's latest AI security research captures the gap well. It found that 83% of surveyed organizations planned to deploy agentic AI capabilities, while only 29% felt ready to use them securely. In a separate Cisco study, nearly 60% of security leaders said security concerns were the main barrier to wider agent adoption.
The problem has become much easier for buyers to picture. During an internal cyber evaluation, OpenAI disclosed that its models found a previously unknown vulnerability in an Artifactory proxy, escaped the intended testing boundary, reached the internet and then chained vulnerabilities to access Hugging Face systems. OpenAI called it an unprecedented cyber incident and tightened containment, monitoring and access controls around future evaluations.
OpenAI then disclosed separate third-party evaluations where model activity also extended beyond intended testing boundaries because more capable models met weak testing controls. That gives agent-security vendors a very concrete sales argument: companies need to control what an autonomous system can access, which tools it can call, what credentials it can use and what happens when it keeps pursuing a goal longer than expected.
The products now being funded reflect that shift. Agent security is increasingly about identity, permissions, runtime monitoring and tool access, which looks much closer to established enterprise cybersecurity than to the older idea of simply filtering bad chatbot outputs.
Is regulation actually forcing companies to spend on AI safety now?
Yes, regulation is currently forcing real AI safety work, especially in Europe, although the largest high-risk compliance bills will arrive later.
The European Commission has now moved into enforcement for several parts of the AI Act. General-purpose AI providers face requirements around documentation, copyright, risk assessment, incident reporting and cybersecurity, with extra duties for models considered systemically risky. New transparency rules also require certain AI systems to tell users when they are interacting with AI and require marking or labelling in cases such as deepfakes and some synthetic content.
The penalties give legal and compliance teams a reason to take the rules seriously. For relevant transparency violations, fines can reach €15 million or 3% of worldwide annual turnover. General-purpose AI providers can also face fines for failing to meet their obligations.
For many enterprise buyers, though, the timetable has become more forgiving. Rules for stand-alone high-risk systems listed in Annex III are now scheduled for December 2027, while high-risk AI embedded in regulated products moves to August 2028. We should expect several compliance waves over the next two years, with spending arriving at different times.
| AI Act pressure | Where it stands now | What companies need to do |
|---|---|---|
| General-purpose AI obligations | Enforceable now | Documentation, risk work, incident reporting and cybersecurity for covered models |
| Transparency rules | Applying now | User disclosure, synthetic-content marking and labelling in covered cases |
| Annex III high-risk systems | Due in December 2027 | Broader risk management, testing, documentation and oversight |
| High-risk AI inside regulated products | Due in August 2028 | Product-level compliance, assurance and monitoring |
If you want more recent data on this point, please see our latest AI safety market report.
This chart, featured in our AI safety market deck, shows annual funding in AI safety startups
Is AI governance becoming software, or will consultants take most of the money?
AI governance is becoming a real software category, but consultants, lawyers and internal teams will still take a large share of the money.
The software side is getting much more concrete. Companies now buy systems that inventory AI models and agents, record who approved them, map policies to applications, run evaluations, enforce runtime rules and keep evidence for audits. Fiddler raised $30 million to build what it calls a control plane for AI with evaluation, monitoring, policy and governance. ZeroDrift raised $10 million for a compliance layer that checks AI-generated messages before they reach users.
Standards are also becoming part of day-to-day company processes. Stanford's latest AI Index found that ISO/IEC 42001 was already influencing 36% of surveyed organizations and the NIST AI Risk Management Framework 33%. The share reporting no regulatory influence at all fell from 17% to 12%.
A lot of AI governance work still involves people. Someone has to decide which use cases are acceptable, write policies, interpret regulation, investigate incidents and negotiate with model vendors. In practice, the category is starting to look a lot like privacy and cybersecurity compliance: plenty of software revenue, with internal teams and professional services around it.
So total AI-governance spending can grow faster than revenue at pure AI-governance startups. Microsoft, cloud vendors, cybersecurity platforms and risk-and-compliance providers can all sell part of the same control layer through products customers already use.
Can frontier AI safety become a normal business?
Only partly. Frontier AI safety is attracting much more money these days, but pure alignment still looks more like a research field than a normal software business.
Testing is where frontier AI safety already looks like a business. Irregular has turned advanced cyber evaluation into a revenue-generating business. Gray Swan raised $40 million to expand adversarial testing and enterprise AI security after working with major frontier labs. These companies sell a task that has a clear deliverable: find dangerous behavior before a model or agent reaches production.
Alignment research has a different funding structure. The UK's AI Security Institute says the first round of its Alignment Project awarded more than £27 million across more than 60 projects. The backers include governments, OpenAI, Microsoft, AWS, Anthropic and philanthropic funders, and individual projects can receive up to £1 million plus compute support.
Those grants show that the frontier-safety research ecosystem is getting bigger, but they do not create recurring software revenue. Many of the best-known groups working on alignment, control and frontier evaluation still operate as nonprofits, academic teams or grant-funded research organizations.
For now, frontier AI safety breaks into two lanes. Red teaming, cyber evaluation, safeguard testing and monitoring are already commercializing. More theoretical alignment work is becoming better funded without yet producing an equally clear standalone market.
If you want more recent data on this point, please see our latest AI safety market report.
This chart, featured in our AI safety market deck, compares the main business model options for AI alignment research labs
Are governments actually paying for AI safety?
Yes, governments are now putting serious money into AI safety and evaluation, both through their own institutes and through outside researchers and vendors.
The UK gives us the clearest scale. The government backed its AI Security Institute with £240 million in the latest spending review. The institute says it now has more than 100 researchers and has tested 30 frontier models, while also funding external work on alignment, safeguards and evaluation.
The United States is moving AI evaluation closer to procurement. NIST's Center for AI Standards and Innovation signed an agreement with the General Services Administration to support evaluation inside USAi, the federal government's shared platform and procurement toolbox for AI. The work includes methods for checking performance, security and functionality before deployment and monitoring systems after they are used.
CAISI has also been building secure evaluation methods with OpenMined and coordinating national-security model testing across government agencies. The UK AISI, meanwhile, is partnering with companies including Microsoft and ElevenLabs on testing and safety research.
Government spending reaches the market in two ways. Public agencies directly fund evaluators, researchers and tooling, and their testing standards can later become requirements for vendors selling AI into the public sector.
Will cybersecurity giants swallow most AI safety startups?
Many AI safety startups will probably get absorbed into larger cybersecurity and infrastructure platforms because enterprise buyers already have vendors for security, identity, observability and compliance.
The consolidation is happening quickly. Check Point acquired Lakera for AI runtime protection and red teaming. SentinelOne bought Prompt Security to add shadow-AI controls and agent protection. Proofpoint acquired Acuvity for AI security and governance. Snyk bought Invariant Labs to strengthen defenses against agentic threats such as tool abuse and emerging MCP risks.
Those buyers already have large sales teams, channel partners and security budgets. If an enterprise can add AI controls to an existing Check Point, SentinelOne, Proofpoint or Snyk contract, a small point solution has to offer something much better to justify another procurement process.
A small independent startup needs a harder problem where neutrality or deep specialization has real value. Cross-model evaluation, proprietary adversarial data, agent identity, independent red teaming and observability across several AI stacks are better positions than a narrow guardrail feature that a platform vendor can copy or bundle.
The AI safety function will probably grow faster than the number of independent AI safety vendors. Acquisitions can accelerate customer adoption by pushing specialized technology through much larger distribution networks.
If you want more recent data on this point, please see our latest AI safety market report.
This chart, featured in our AI safety market deck, shows revenue breakdown by customer segment in the AI safety market
What could slow down the AI safety market?
The AI safety market could slow if enterprise AI never scales far beyond pilots, if big vendors bundle the tools cheaply, or if compliance deadlines keep moving out.
McKinsey still finds only 7% of organizations fully scaled on AI. If most experiments never reach important production workflows, companies will need fewer specialized controls than current startup funding implies. In the end, the AI safety market depends on how much consequential AI is running inside companies, not on how many demos are being built.
Budgets are another constraint. Stanford found that 48% of organizations cited budget limits as a barrier to responsible-AI implementation and 59% cited knowledge gaps. Large banks and technology companies can support specialized security, evaluation and governance teams; a mid-sized company may prefer whatever controls arrive inside its cloud, model or cybersecurity contract.
Bundling could become especially painful for startups. OpenAI is integrating Promptfoo into Frontier, while major security companies are folding acquired AI-safety products into broad platforms. Open-source evaluation frameworks also make basic testing easier to commoditize.
Regulation will keep creating work, but the delayed European high-risk rules reduce the urgency for some buyers over the next year. We still expect fast growth, probably with fewer independent winners than today's startup count suggests.
So, is the AI safety market growing now?
Yes, the AI safety market is clearly growing now, with the strongest commercial growth in AI security, model evaluation, monitoring and governance around systems that companies are already deploying.
Recent activity is already too large to write off as hype. Five relevant startups raised about $480 million over roughly five weeks. Five disclosed acquisitions across AI security and evaluation add up to about $1.97 billion. European AI rules have moved into active enforcement for general-purpose AI and transparency. At the same time, companies such as WitnessAI are reporting triple-digit annual recurring revenue growth and evaluation tools such as Promptfoo have reached large parts of the Fortune 500.
The growth is uneven. Enterprise AI security already looks like a normal cybersecurity category with CISOs, large contracts, venture rounds and aggressive M&A. Evaluation is quickly becoming part of production observability. Governance is becoming software, but services and internal teams still take a large share of spending. Frontier alignment has more funding than before, while its pure commercial market remains much smaller.
Today, AI safety looks like a growing control layer around AI, spread across cybersecurity, observability, model testing and governance. The category boundaries will probably stay messy because established vendors are absorbing many of the best standalone tools.
AI safety has now crossed into real enterprise spending. The clearest part to watch is security for agents and production systems, where fresh funding, customer urgency and platform consolidation are currently moving fastest.
This chart, featured in our AI safety market deck, shows how prompt injection defense platform technology has evolved over time
OUR METHODOLOGY
This analysis tests whether the AI safety market is genuinely growing based on commercial activity that can be observed today. We separate the market into AI security, model evaluation and monitoring, AI governance and compliance, and frontier safety, because those areas are commercializing at very different speeds.
We looked across enterprise adoption and spending, startup financing, M&A, model evaluation, agent security, governance and regulation, frontier-safety funding, and government activity. We gave the most weight to customers paying for products, disclosed revenue or usage, fresh financing, strategic acquisitions, government funding, and regulation that creates concrete operational requirements.
Broader adoption surveys are used mainly to explain the conditions behind that spending. McKinsey's State of AI work provides the enterprise adoption and scaling baseline, while Stanford HAI's 2026 AI Index provides data on AI incidents, responsible-AI policies, governance roles, standards adoption and implementation barriers.
For startup financing and customer traction, we prioritized direct company disclosures from WitnessAI, Obsidian Security, Zenity, Mindgard, Fiddler and other relevant vendors. The roughly $480 million financing figure in the article is an aggregation of five clearly relevant rounds announced over roughly five weeks, rather than a market-size estimate.
For M&A, we relied on disclosed transaction values and strategic buyer announcements, including F5's SEC filing for CalypsoAI and SurePath AI and Dynatrace's announced $915 million agreement to acquire Arize. We treat these deals as evidence that established security and infrastructure companies are putting AI-safety capabilities into products and revenue plans, not as a direct measure of annual market revenue.
For model evaluation and agent security, key sources include Promptfoo's published adoption figures and its agreement to join OpenAI, Cisco's 2026 AI security research on agent deployment and readiness, and OpenAI's disclosures about model-evaluation security incidents. These sources help distinguish general interest in AI safety from production problems that companies are already trying to control.
For regulation and government activity, we prioritized primary sources from the European Commission, the UK AI Security Institute and NIST. The EU AI Act material is used to separate obligations that are already enforceable from high-risk requirements arriving later, while UK and US sources show how governments are funding, testing and institutionalizing AI evaluation.
Key sources used for this analysis include: McKinsey's State of AI, Stanford HAI's 2026 AI Index, WitnessAI's financing and ARR disclosure, Obsidian Security's Series D announcement, Zenity's $125 million financing announcement, Mindgard's Series A announcement, F5's SEC filing, Dynatrace's Arize acquisition announcement, Promptfoo's adoption disclosure, OpenAI's Promptfoo acquisition announcement, Cisco's State of AI Security 2026, OpenAI's model-evaluation security incident disclosure, the European Commission's AI Act enforcement guidance, the UK AI Security Institute's Alignment Project, and NIST's CAISI-GSA evaluation agreement.
In our AI safety market deck, we identify pain points entrepreneurs should prioritize
Related blog posts
- Which startups are the most valued in the AI safety market?
