AI Coding: what is actually working now?

Why does AI coding feel different now?

AI coding has clearly crossed into real software production: developers are giving agents actual engineering work, and companies are now measuring the result in pull requests, cycle time and money.

The biggest change is what developers ask AI to do. GitHub Copilot began as an autocomplete tool that predicted the next few lines. Today, GitHub, Cursor, Claude Code and Codex can inspect repositories, edit several files, run commands, execute tests, investigate failures and create pull requests. The jump from “suggest some code” to “take this task and come back with a working change” has happened surprisingly fast.

Usage has followed. Microsoft recently said GitHub Copilot had reached 50 million users, up from more than 26 million earlier in the fiscal year. One in three pull requests on GitHub now involves an agent. Copilot revenue also accelerated by more than 60% quarter over quarter after GitHub added more usage-based billing.

The same thing is happening outside GitHub. OpenAI says more than 5 million people now use Codex every week. Anthropic says Claude Code has passed $2.5 billion in run-rate revenue, more than double its level at the start of the year, with enterprise customers generating over half of that revenue. Cursor has reportedly crossed $2 billion in annualized revenue, while the company says more than 70% of Fortune 500 companies use its product.

Those numbers stop short of proving that every developer becomes faster. They do show something simpler and already important: AI coding has become a large paid software category, and developers are increasingly trusting these tools with work that eventually reaches production.

Current evidence	What it tells us
GitHub Copilot	50M users
GitHub	1 in 3 pull requests now involves an agent
Codex	More than 5M weekly users
Claude Code	More than $2.5B run-rate revenue
Cursor	Reportedly more than $2B annualized revenue

Does AI coding actually make developers faster?

AI coding now appears to make many developers faster, but the size of the gain depends so heavily on the task that a universal productivity percentage would be misleading.

The early evidence looked much cleaner. GitHub ran a randomized experiment with 95 professional developers and asked them to build a small JavaScript HTTP server. Developers using Copilot finished in 1 hour 11 minutes on average, compared with 2 hours 41 minutes without it. GitHub calculated a 55% improvement.

Then METR tested something much harder. Sixteen experienced open-source developers worked on 246 real issues in repositories they knew extremely well. These developers had spent an average of about five years in the codebases. With the AI tools available in early 2025, mainly Cursor and Claude, they took 19% longer.

That experiment became even more interesting later. When METR tried to repeat it with newer agents, developers increasingly declined to participate because some assigned tasks had to be completed without AI. METR said roughly 30% to 50% of potential participants were withholding at least some tasks for this reason. Among developers from the original experiment who returned, the raw estimate moved from a 19% slowdown to an 18% speedup, although the sample and selection effects were too messy to make 18% a reliable headline number.

DORA provides a broader view. In its latest study of nearly 5,000 technology professionals, around 90% said they were using AI at work and more than 80% believed AI was improving their productivity.

Put those studies together and the apparent contradiction mostly disappears. AI can save a lot of time on a well-defined task, save much less on difficult work inside a familiar codebase, and occasionally create extra work when the developer has to inspect or repair what the model generated. Current agents also seem meaningfully better than the ones tested only a year earlier.

Study	Result	What was being tested
GitHub controlled experiment	55% faster	Small fixed coding task
METR experienced maintainers	19% slower	Real issues in mature familiar repositories
METR later returning developers	Raw estimate of 18% faster, highly uncertain	Newer agentic tools
DORA	80%+ report productivity gains	Everyday workplace use

What AI coding tasks work really well today?

AI coding works best today when the task is clear and the software itself can quickly tell the agent whether the answer is wrong.

Codebase navigation is already one of the easiest wins. A coding agent can search thousands of files, find implementations, follow call paths and explain unfamiliar components in minutes. OpenAI says its own engineers use Codex heavily for this kind of investigation when entering unfamiliar systems or debugging incidents.

Tests are another strong use case. An agent can generate a test, run it, inspect the failure and try again. Salesforce says one team cut the time required to improve test coverage in a legacy system by 85% after adopting Cursor.

Large mechanical changes can work surprisingly well too. Faire used agents to migrate a major application away from MobX. The migration could be divided into many independent changes, which made it ideal for parallel agents. Faire estimates that a project originally expected to occupy a team for around 18 months can now be supervised by one engineer coordinating the agents.

Bug investigation is moving in the same direction. 1Password reported that one complicated problem spanning several services required around 90% less investigation time with Codex. GitHub's coding agents can also take a failing CI run or a review comment, modify the relevant files, rerun the repository's checks and update the pull request.

There is a clear common thread. Compilers, unit tests, type checkers, linters, CI and logs give the agent immediate feedback. AI coding gets much harder when “correct” depends mostly on human taste, undocumented business logic or decisions about what the product should become.

If you want more recent data on this point, please see our latest AI code assistant market report.

Can AI coding agents actually fix real software issues by themselves?

AI coding agents can now solve a meaningful share of real software bugs on their own, although passing the automated test still falls well short of proving that a maintainer would merge the code.

Modern benchmarks are far tougher than the small coding tests that dominated the first generation of coding models. SWE-Bench Pro, for example, takes real GitHub issues and asks an agent to navigate a repository and produce a working patch. Frontier models now solve a large share of those tasks.

That looks impressive until a human maintainer inspects the result.

METR asked maintainers of scikit-learn, Sphinx and pytest to review AI-generated patches that had already passed SWE-bench's automated tests. Roughly half of those supposedly successful patches would still have been rejected instead of merged. Across the sample, the benchmark overstated real maintainer acceptance by about 24 percentage points.

The rejected patches reveal where current agents still struggle. Some added weak tests. Others introduced unnecessary complexity, made incorrect assumptions or technically fixed the benchmark while producing code that experienced maintainers did not want in the project.

Real development does give agents one advantage that this evaluation did not: iteration. A maintainer can leave comments, send the agent back to work and review the next version. That makes these systems more useful than a one-shot benchmark suggests.

Even so, “the coding agent solved the issue” currently needs a qualifier. A substantial amount of autonomous implementation already works. Human acceptance remains a separate hurdle.

How much work can an AI coding agent do before a human has to step in?

AI coding agents can now handle tasks representing hours of human work, and sometimes much more, as long as the job has good feedback and reasonably clear boundaries.

METR's time-horizon research gives us a better way to see the change than another benchmark percentage. METR measures how long a task would take a human and asks how often a frontier agent can finish comparable work successfully. The length of tasks agents can handle has been rising exponentially across successive model generations.

Some controlled experiments now stretch surprisingly far. METR's MirrorCode work found that agents could sometimes reimplement software projects representing weeks of human work, including one codebase of roughly 16,000 lines.

Real usage points in the same direction. OpenAI studied Codex activity and found that 70.2% of sampled individual users had asked Codex at least once to take on a task estimated to represent more than an hour of human effort. More than a quarter had submitted something estimated at over eight hours.

An eight-hour human task does not mean the model literally works unsupervised for eight flawless hours. The important change is that developers can now hand over chunks of work much larger than one function or one prompt.

That changes the rhythm of software development. A developer can start a task, move elsewhere and inspect the resulting implementation later instead of sitting next to the model accepting suggestions one line at a time.

Does running several AI coding agents at once really make developers more productive?

Parallel AI coding is already producing some of the biggest reported gains because one developer can supervise several independent jobs instead of completing them sequentially.

Faire says it launches more than 2,000 automated Cursor agent runs every week across bug triage, failed-CI investigation, review and migrations. More than 25 recurring automations launch without a developer manually starting every run. Faire says weekly pull-request throughput doubled after introducing cloud agents.

Coinbase has pushed this even further. According to a company case study published by Cursor, more than 2,400 Coinbase engineers use the product and engineers commonly run five to seven asynchronous agents. Coinbase says 75% of its pull requests now originate from agents, while merged pull requests per engineer increased 55%.

The productivity gain is easy to miss if we keep measuring AI like autocomplete. Suppose an agent spends two hours fixing a bug while the developer starts three other tasks. The developer may spend only 15 minutes reviewing the result. A stopwatch measuring how fast that developer could manually fix the same bug captures only part of what changed.

Parallelism can also create a new mess. Five agents can generate five questionable pull requests just as easily as five good ones. Teams reporting the best results have generally built strong tests, isolated environments and automated checks around agent output.

Today, the interesting unit of productivity is increasingly the amount of verified work one engineer can supervise.

If you want more recent data on this point, please see our latest AI code assistant market report.

Is AI-generated code actually good enough for production?

AI-generated code can absolutely reach production quality today, but companies get much better results when automated checks and human review grow alongside code generation.

DORA found that 59% of respondents believed AI was improving code quality. Salesforce says its internal engineering metrics showed quality improving alongside higher pull-request velocity after rolling out Cursor. GitHub's earlier controlled Copilot study also found better reviewer scores for readability and maintainability in several categories.

There is enough contrary evidence to keep that from becoming a blanket rule.

Stack Overflow found that 66% of developers had dealt with AI answers that were “almost right,” and 45% said debugging AI-generated code could take more time. An observational study by Uplevel covering nearly 800 developers found no significant throughput improvement after Copilot adoption and recorded 41% more bugs in the Copilot group. The study could not prove Copilot caused those bugs, but it certainly gives us no reason to assume generated code comes free of downstream costs.

Security review makes the problem even clearer. A peer-reviewed study testing GitHub Copilot's code-review feature against known vulnerable code found that it regularly missed serious issues such as SQL injection, cross-site scripting and insecure deserialization.

The stronger workflow combines AI with deterministic tooling. GitHub's agentic Autofix, for example, starts from alerts detected by tools such as CodeQL. AI then investigates the relevant code and proposes a repair, while the specialized scanner checks whether the vulnerability disappeared.

That combination is much more convincing today than asking a general coding model to read a pull request and find every important problem from scratch.

Can AI coding actually deal with ugly legacy software?

AI coding is becoming genuinely useful on legacy systems, especially for migrations, tests and repetitive changes that can be checked automatically.

Greenfield projects remain easier. OpenAI demonstrated the extreme case with an internal product containing roughly one million lines of application code, tests, CI configuration, documentation and tooling generated through Codex. The team estimated that development took about one tenth of the time it would have expected from a conventional workflow.

Legacy software gives agents a much uglier environment. There are old abstractions, undocumented behavior, strange build systems and years of hidden assumptions.

Still, several recent company cases show agents pushing into exactly that territory. Salesforce reports strong gains in a mature codebase accumulated over decades. Faire's large MobX migration involved changing an existing production application rather than generating a clean new one. Cisco says it uses Codex across large multi-repository systems including C and C++, and reports 10-to-15-times higher defect-resolution throughput in some workflows.

These are company case studies, so the exact gains deserve more caution than an independent experiment. The pattern is useful anyway: old code by itself is no longer enough to make agents useless.

The hardest legacy problems now sit higher up. An agent can transform 2,000 similar call sites surprisingly well. Deciding whether an undocumented behavior exists because of an old customer contract is a very different problem.

What makes an existing codebase easy for AI coding agents?

AI coding agents perform much better in repositories with clear instructions, reproducible environments and fast automated feedback.

Cursor discovered this while deploying its own cloud agents. The company says agents need access to the same tools and data developers use, documentation explaining how work is done, and reliable ways to test important workflows. Cursor says its cloud agents now author a majority of the code the company ships.

Faire reached a similar conclusion from a very different codebase. Its frontend and backend rely on separate dependencies, Bazel, Gradle, AWS credentials and internal systems. Simply giving an agent the source code did not recreate the environment a developer actually works in. Faire had to build isolated environments where agents could install dependencies, reach the necessary services and verify their changes.

DORA's broader research points the same way. Around 90% of organizations in its study already operated internal developer platforms and 76% had dedicated platform teams. DORA found that AI tends to amplify the engineering system around it: teams with good foundations get more out of the tools, while weak processes can turn faster code generation into extra instability.

This could end up being one of the biggest changes AI brings to software engineering. Repositories are gradually being redesigned so machines can understand how to work inside them. Clear setup scripts, strong tests, documented conventions and good observability help human developers too, which makes this one of the less speculative AI investments companies can make.

If you want more recent data on this point, please see our latest AI code assistant market report.

Is vibe coding actually working for professional software?

Vibe coding is working for prototypes and low-risk software, while most professional developers still stop well short of letting prompts replace engineering discipline.

Stack Overflow asked developers directly whether vibe coding was part of their professional development work. Seventy-two percent said no, with another 5% giving an even stronger rejection. In the same survey, 84% of developers said they already used or planned to use AI tools.

That gap tells us a lot. Developers have embraced AI coding much faster than they have embraced the idea of blindly shipping whatever an AI produces.

Look at the teams getting serious results. Coinbase writes technical and product specifications for agents and then checks the implementation against those requirements. Faire gives agents controlled environments and automated validation. OpenAI's agent-first engineering experiment spent a great deal of effort building the feedback loops through which agents could test themselves.

Vibe coding still has a large useful territory. Personal tools, internal dashboards, prototypes, experiments and simple applications can tolerate mistakes that would be unacceptable in payments, identity, healthcare, infrastructure or security software.

For professional engineering, the trend these days is closer to heavier specification plus cheaper implementation. Developers may type far less code themselves while becoming more explicit about what the software must do.

Are companies seeing real productivity gains from AI coding yet?

AI coding is now producing measurable gains inside some large engineering organizations, although the huge spread between companies tells us that implementation quality matters as much as buying the tool.

1Password provides a relatively grounded example. The company says its main Codex cohort showed nearly 21% higher engineering productivity and an 11% reduction in median pull-request cycle time. It estimated roughly $800,000 in extra annual engineering capacity and a 553% return on what it spent on Codex.

Salesforce reports more than a 30% increase in pull-request velocity after adopting Cursor broadly. Coinbase says merged pull requests per engineer increased 55% and that some teams cut the time from idea to production from around 20 days to 1.8 days.

As seen above, Faire says weekly pull-request throughput doubled after moving substantial work to cloud agents. Cisco reports a different type of gain: much faster defect resolution and more than 1,500 engineering hours saved every month.

Averaging all of this into “AI makes developers 40% faster” would be pretty meaningless. The companies use different definitions, different codebases and different workflows, and most of these numbers come from vendor-published customer stories.

The broader pattern is still hard to ignore. The strongest AI coding gains are now appearing in production metrics such as cycle time, merged pull requests, defect resolution and engineering capacity. A year or two ago, much of the evidence stopped at “developers say they feel faster.”

Company	Reported result
1Password	~21% higher productivity; ~11% lower median PR cycle time
Salesforce	30%+ higher PR velocity
Coinbase	55% more merged PRs per engineer
Faire	2x weekly PR throughput
Cisco	10–15x defect-resolution throughput in selected workflows

If AI writes code faster, do companies actually ship software faster?

AI coding now appears to increase software-delivery throughput, but companies can easily move the saved time into review, verification and cleanup instead.

DORA gives us a rare view of how this changed over time. Earlier research found that a 25% increase in AI adoption was associated with a 1.5% decrease in delivery throughput and a 7.2% decrease in delivery stability. Developers could generate changes faster, but larger batches and extra review work clogged the system downstream.

DORA's latest results have moved in a more positive direction. Higher AI adoption is now associated with higher delivery throughput. Stability remains the uncomfortable part: DORA still finds that heavier AI use can increase delivery instability when the surrounding engineering system is weak.

That fits what developers report. AI can produce a first implementation very quickly. Someone still has to understand the change, review it, test it, deploy it and deal with whatever breaks.

Some companies have managed to speed up the whole chain. Coinbase says selected teams reduced idea-to-production time from around 20 days to 1.8 days. 1Password reports an 11% fall in median pull-request cycle time. Those are more useful numbers than lines of code generated because they tell us whether software actually moved through the organization faster.

The bottleneck is shifting. Writing code used to consume a huge share of engineering effort. Today, some teams can generate implementation faster than they can confidently approve it.

Is AI coding already a real business?

AI coding is already a multibillion-dollar software business, so the commercial question has moved from “will developers pay?” to “how much agent work will companies ultimately buy?”

The commercial scale is already large. Microsoft says GitHub Copilot has 50 million users and recently reported more than 60% quarter-over-quarter revenue acceleration. Anthropic says Claude Code exceeds $2.5 billion in run-rate revenue, with enterprise customers generating more than half of it. Cursor has reportedly passed $2 billion in annualized revenue. OpenAI says Codex has more than 5 million weekly users.

Gartner has estimated annualized enterprise spending on AI coding agents at roughly $10 billion.

The business model is also moving away from simple subscriptions. Autocomplete fits a predictable monthly seat. An autonomous agent can run for much longer, invoke expensive models, use tools and operate several jobs at once. GitHub has therefore started mixing seat fees with usage-based billing, and other coding-agent products increasingly charge according to consumption.

That changes how companies will judge ROI. A $20 autocomplete product only needed to save a developer a small amount of time each month. A team running hundreds or thousands of agent jobs can spend far more, so management eventually has to connect that bill to completed work.

The encouraging part for AI coding companies is that large customers are already making that calculation and increasing their usage.

If you want more recent data on this point, please see our latest AI code assistant market report.

Are coding benchmarks still useful, or are they fooling us?

AI coding benchmarks are useful for showing how quickly models are improving, but current benchmark scores are poor substitutes for asking whether experienced developers would actually ship the result.

SWE-bench helped move the field forward because agents have to solve issues inside real repositories instead of completing tiny coding puzzles. Terminal benchmarks similarly test whether models can use shells, tools and multi-step workflows.

Those evaluations capture genuine progress. Frontier models today can navigate repositories, modify several files and recover from failed attempts in ways that earlier coding assistants simply could not.

The problem starts when a benchmark score gets translated into something like “AI can now do 60% of a software engineer's job.”

The maintainer study from METR gives us the clearest warning. Patches had already passed SWE-bench's automated tests, yet roughly half still failed the human merge test when maintainers from scikit-learn, Sphinx and pytest reviewed them.

Real engineering contains many things the benchmark cannot easily score: whether the change fits the architecture, whether the test is good enough, whether unnecessary complexity was introduced, whether a strange behavior is intentional, and whether another solution will be easier to maintain in two years.

So we should keep watching the benchmarks. They are excellent capability thermometers. They are much weaker measures of how much engineering judgment a company can safely hand over.

Are AI coding agents already replacing software engineers?

AI coding agents are already removing chunks of work that software engineers used to do manually, while full replacement remains well beyond what companies can safely rely on today.

The job itself is clearly starting to shift. Coinbase describes engineers writing specifications and supervising five to seven agents rather than manually implementing every change. Faire says one engineer can supervise a migration that was previously expected to require an entire team for around 18 months. OpenAI has even run an experimental project where humans wrote essentially none of the application code themselves and instead designed the environment, requirements and feedback loops around Codex.

Stack Overflow captures the strange middle ground well. Professional developers use AI heavily, yet only a small minority say they strongly trust its output. Daily dependence can rise long before unconditional trust does.

That changes headcount economics even without a robot software engineer appearing. If one developer can supervise several implementation jobs, a team may need fewer engineering hours to ship the same amount of software. It can also use the freed capacity to build more, so fewer hours per feature will not automatically mean fewer developers.

The parts of software engineering that remain hardest to delegate are also obvious from the failures we have already seen: deciding what should be built, turning vague product needs into precise requirements, choosing architectural trade-offs, understanding undocumented business constraints and deciding whether an apparently working change is safe enough to ship.

Current AI coding agents are eating implementation work quickly. Engineering judgment is moving much more slowly.

So, what is actually working in AI coding now?

AI coding is genuinely working now as a high-leverage implementation layer: agents can already take over meaningful pieces of coding, testing, debugging, migration and repository work, while humans still own the decisions and the final check.

The strongest evidence now comes from real repositories and production workflows. Agents routinely inspect codebases, edit multiple files, execute tests, fix CI failures and open pull requests. Some developers give them tasks representing hours of human work and run several agents at once. Large engineering teams are beginning to report measurable improvements in pull-request throughput, defect resolution and cycle time.

The pattern is also much clearer about where AI coding works best. A bounded task, good repository context and a fast way to check the result create the strongest setup. Tests, compilers, linters, type systems and logs give an agent something concrete to react to. Large migrations, test generation, bug investigation and localized feature work fit that pattern particularly well.

The weak spots are just as visible. As pointed out above, METR showed that a patch can pass automated benchmark tests and still fail a maintainer's review. Stack Overflow finds widespread use alongside persistent distrust of AI accuracy. DORA finds higher throughput today, but also warns that AI can increase instability when teams generate changes faster than their engineering system can absorb them.

The market data removes another source of doubt. GitHub Copilot now has 50 million users, Claude Code has become a multibillion-dollar run-rate business, Codex has millions of weekly users and companies are paying for increasing amounts of agent execution. This has moved well beyond a temporary developer experiment.

The conclusion is pretty clear. AI coding works today, and it works well enough to change how software gets made. The biggest gains come from letting agents execute clearly specified work inside an environment that can check them quickly. Developers can already spend much less time typing and manually implementing routine changes.

The frontier has moved to supervision. Humans still decide what the software should do, recognize when the agent has misunderstood the job and take responsibility for whatever reaches production. AI has already become good at doing a surprising amount of the work between those decisions.

If you want more recent data on this point, please see our latest AI code assistant market report.

Who is the author of this content?

NEW MARKET PITCH TEAM

We track new markets so founders and investors can move faster

We build living "market pitch" documents for emerging markets: AI, synthetic biology, new proteins, and more. Instead of outdated PDFs or hallucinated LLM answers, our clients get a clean, visual, always-updated view of what's really happening: key players, deals, regulations, and signals that matter. Learn more about us.