Is AI agent traffic klling the internet?

Is AI agent traffic killing the internet?

What do we actually mean by AI agent traffic?

Most of the traffic described as AI agents today still comes from bulk crawlers; personal assistants shopping or booking for users remain a small minority.

The label covers several very different machines. Training crawlers collect large datasets. Search crawlers build indexes for AI answers. Fetchers retrieve fresh pages after a user asks a question. Browser agents click through sites, fill forms, or try to complete a purchase.

Those categories create different costs and benefits. A training crawler may download thousands of pages without sending a single visitor. A fetcher serves a real user, although that user may read the answer inside ChatGPT or Claude and never open the source. A shopping agent can eventually produce a sale.

Fastly’s May 2026 network sample shows what websites are actually seeing. Broad crawlers produced about 85% of the identifiable AI requests it observed, while fetchers linked more closely to live user questions produced 15%. Fully delegated agents are an even smaller part of the web today. That naming shortcut muddies the debate. People say “agents,” while most of the pressure still comes from crawlers and answer engines.

Type of automated traffic	What it does	What the website gets
Training crawler	Copies large amounts of content for model development	Usually no direct visit or sale
AI search crawler	Builds an index used in future answers	Possible visibility, often few clicks
AI fetcher	Retrieves current information for a live user query	Some chance of a referral
Browser agent	Compares, clicks, fills forms, or buys	A possible lead or transaction
Malicious bot	Scrapes, attacks, or commits fraud	Cost and risk

Have bots really overtaken humans on the web?

Bots currently generate more web requests than humans in several major network datasets, while people still dominate the internet’s heaviest activities.

Imperva estimated that automated systems produced 53% of web traffic during 2025, up from 51% a year earlier. Cloudflare has also recorded automated requests passing human requests for HTML pages, and Fastly recently measured bots at close to half of application requests on its network. The crossover is real across several large infrastructure providers.

A request is a poor way to measure the whole internet, though. One crawler can request 10,000 small pages, while one person streaming a film may transfer far more data through only a handful of connections. Video remains the biggest category by data volume, followed by other human-heavy uses such as cloud applications, social media, gaming, and communications.

In plain English, machines now knock on more website doors. People still account for most of the attention, entertainment, and heavy data use happening behind them.

What is being measured?	Who appears to lead?	What the result really tells us
HTML or application requests	Bots can lead	Machines contact many pages very frequently
Total data transferred	Humans still dominate	Video and other rich media outweigh text crawling
Time and attention	Humans dominate	People watch, read, communicate, and play
Economic intent	Mixed	Bots create both fraud and genuine commercial demand

Are AI agents really behind the bot takeover?

AI has accelerated the bot takeover, but genuine user agents still represent a small slice of the automated traffic websites face.

Imperva split 2025 traffic into roughly 47% human activity, 40% bad bots, and 13% other automated traffic. That 40% includes account attacks, scraping, fraud, and other long-running forms of automation. AI can make those bots cheaper and more capable. Very few are personal agents working for consumers.

Cloudflare’s latest purpose data shows how far training has spread. Its share of crawler requests rose from 22% in spring 2025 to 52% by mid-2026, while mixed-use crawlers represented more than 36%. Today’s machine-heavy web mixes old bots, AI-assisted attacks, and industrial crawling. Useful autonomous agents are only beginning to appear.

For websites, the current pressure comes from machine-scale reading by systems whose purpose and value are often unclear. Mass autonomous shopping has barely started.

If you want more recent data on this point, please see our latest agentic AI market report.

How fast is AI traffic growing now?

AI traffic is currently growing several times faster than human traffic, so a modest share can become expensive for a website surprisingly quickly.

Fastly followed the same customer group from January through May 2026 and saw AI requests rise about 30%. Human requests grew by less than 5% over the same period, making AI traffic’s growth rate roughly 6.5 times faster. Claude-related traffic rose especially sharply, although from a smaller starting point.

Cloudflare had already seen the shift toward live retrieval. Between May 2024 and May 2025, GPTBot traffic roughly quadrupled while ChatGPT-User requests, which are more closely tied to immediate user prompts, rose more than twentyfold. Percentages change from one network to another, yet the same trend keeps appearing: crawling expands, and user-triggered fetching expands even faster.

A website can feel the impact long before AI consumes a large share of global bandwidth. A doubling of uncached requests can hurt a small publisher or documentation site long before anyone notices a problem at internet-backbone level.

Can AI traffic actually overwhelm a website?

Individual websites can already be overwhelmed by AI traffic, even while the physical internet as a whole remains comfortable.

Text crawling is tiny beside global video, cloud, and gaming traffic. The pressure appears at the website’s origin server, where databases, storage, and application code do the expensive work. Fastly found that fewer than 9% of human requests in its latest study had to travel back to the origin, compared with more than 51% of automated AI requests.

That sixfold difference comes from browsing habits. Human visitors gather around a site’s popular pages, which are usually cached. Crawlers roam through archives, old PDFs, obscure product pages, and constantly changing endpoints. Fastly has seen individual AI fetchers exceed 39,000 requests per minute, enough to resemble an attack even when the operator intended no harm.

So the internet backbone is coping. Smaller websites are the ones being squeezed, particularly when a burst of machine traffic reaches pages that were never designed for mass retrieval.

Why do AI bots cost websites so much more than human visitors?

AI bots often cost more because they request the least cacheable parts of a site and repeatedly fetch content that has barely changed.

Wikimedia saw this imbalance clearly. Bots produced about 35% of pageviews but at least 65% of the expensive traffic reaching its core data centers. Automated systems were heavily downloading files from Wikimedia Commons, pushing multimedia bandwidth up by roughly 50% from its earlier baseline and reducing the spare capacity available for genuine demand spikes.

Read the Docs found an even more visible before-and-after result. Once it blocked abusive crawlers from repeatedly downloading large documentation files, daily bandwidth fell from around 800 gigabytes to 200 gigabytes. Three-quarters of the previous load disappeared without removing a meaningful human audience.

Cloudflare has separately estimated that more than half of good-bot crawling revisits pages that have not changed. That waste is built into the way broad crawlers work. They want complete coverage, while human interest usually concentrates on a small set of pages.

Example	Machine behavior	Measured effect
Fastly network	AI requests reached origin servers far more often	Over 51% of AI requests reached origin, versus under 9% of human requests
Wikimedia	Bots downloaded pages and media at scale	35% of pageviews created at least 65% of costly core traffic
Read the Docs	Crawlers repeatedly fetched large files	Blocking them cut bandwidth by about 75%
Cloudflare network	Good bots revisited unchanged pages	More than half of crawling added no freshness

Do AI companies send enough visitors back to websites?

AI companies currently return only a tiny fraction of the traffic their systems consume, and those referrals remain far too small to replace search.

TollBit’s 2025 publisher data found roughly 1,700 OpenAI crawler visits for each human referral. Perplexity’s ratio was around 369 to one, while Anthropic’s reached about 73,000 to one. Another large website study by Ahrefs found that ChatGPT, Perplexity, and Gemini together generated only around 0.1% of referral traffic; Google sent about 345 times more visits.

The exact ratios change with the sites and period being measured, yet every study finds the same lopsided exchange: AI companies read hundreds or thousands of pages to produce a small number of outbound clicks.

The spectacular growth rates around AI referrals can hide this gap. A channel can grow fivefold from a tiny base and still replace almost nothing when Google visits fall by 20% or 30%. These days, AI referrals give some sites useful extra traffic. They still come nowhere close to replacing search.

If you want more recent data on this point, please see our latest agentic AI market report.

Are AI answers already taking clicks away from publishers?

AI answers are already cutting clicks to publishers, and the drop now appears in both user-level studies and industry traffic data.

Pew Research Center studied nearly 69,000 Google searches made by 900 US adults. People clicked a normal search result in 15% of visits without an AI summary and only 8% when a summary appeared. Links cited inside the AI summary received clicks in just 1% of visits. In practice, the answer box roughly halved the chance that a user would leave Google through a standard result.

Publisher traffic is falling in the same direction. Chartbeat data used by the Reuters Institute showed organic Google traffic dropping 33% globally across more than 2,500 sites between November 2024 and November 2025. US sites lost 38%. AI Overviews explain only part of the loss; algorithm changes, shifting news demand, and older zero-click features also played a role. Still, Pew shows exactly how part of the wider decline happens.

Publishers are feeling the damage first in pages built to answer a clear question: explainers, lifestyle advice, reference pages, reviews, and evergreen service content. When Google or an assistant can compress the useful part into a few sentences, the source page becomes optional.

Are the people who click from AI more valuable?

The smaller group arriving from AI is currently more likely to buy, which makes the channel attractive for retailers and far less useful for publishers.

Adobe analyzed more than one trillion visits to US retail websites and found AI-driven traffic up 393% year over year in the first quarter of 2026. By March, those visitors converted 42% better than traffic from non-AI sources. They also spent 48% longer on the site, viewed 13% more pages, and generated 37% more revenue per visit.

That reversal happened quickly. A year earlier, AI-referred shoppers converted 38% worse than other visitors. The latest results suggest that people now use assistants to compare products, narrow choices, and arrive with a clearer idea of what they want.

For merchants, those are excellent visitors. A publisher gets a much worse deal because the assistant can deliver the product—the information—before any click happens.

Can ad-funded websites survive when AI gives the answer directly?

Many ad-funded information sites will struggle because an AI answer removes the pageview that was supposed to pay for the underlying work.

The old search bargain was simple enough: a crawler indexed a page, Google displayed a link, and some users visited the source. The publisher could then show ads, sell a subscription, or earn an affiliate commission. AI answers keep the crawling step while often removing the visit.

User agents make the gap even clearer. An assistant may read a review, compare five products, and recommend one without rendering the ads that funded the review. The user receives real value, while the website records server costs and no monetizable attention.

Cloudflare’s latest controls show that website owners are done treating every recognized crawler as welcome. Website owners can now manage Search, Training, and Agent bots separately, and Cloudflare has announced special protection for ad-supported pages. That product decision reflects a practical reality: a page carrying ads was built for a person to see, not merely for a machine to absorb.

Advertising will remain strong where people want the experience itself—video, games, communities, entertainment, and distinctive brands. Straightforward informational pages have a much harder future because their useful content is easy to extract and summarize.

If you want more recent data on this point, please see our latest agentic AI market report.

Why are more websites blocking AI crawlers now?

Websites are blocking AI crawlers because the old promise of “visibility in exchange for access” no longer feels credible.

Fastly recently found billions of requests from bots previously treated as desirable being blocked by customers. Those blocks accounted for about 4% of all bot traffic in the sample. Publishers are no longer assuming that a recognized crawler automatically helps their business.

Cloudflare has pushed the change further. New website owners can block training crawlers by default, and all customers can now separate search bots from agents. Cloudflare has also announced stricter defaults for ad-monetized pages and lets selected crawlers be charged. One year after launching its first pay-per-crawl tools, Cloudflare said it had counted more than 50 licensing agreements between publishers and AI companies since 2023.

The web will probably stay open to people, while machine access becomes much more selective: free pages for people, limited access for search engines, paid access for training, structured feeds for trusted agents, and hard blocks for unidentified automation. That keeps information online while making machine access far less open than ordinary human browsing.

Can a website tell a helpful AI agent from a dangerous bot?

Most websites still cannot tell with enough confidence, and that uncertainty is becoming one of the web’s biggest practical headaches.

A shopping assistant, a price scraper, and an account-takeover bot can all run a real browser, execute JavaScript, and imitate normal clicking. Their code may look almost identical from the website’s side, even though one carries a customer’s permission and another is trying to steal an account.

Robots.txt offers little protection on its own because it is only a voluntary instruction file. Cooperative crawlers respect it; evasive ones ignore it or change their identity. TollBit measured suspected robots.txt bypasses in 13.26% of identifiable AI-bot requests during the second quarter of 2025, up from 3.3% two quarters earlier. Cloudflare has also documented undeclared crawlers changing user-agent strings and network addresses after sites tried to block them.

Security teams are also dealing with a much faster threat curve. Imperva found AI-driven bot attacks increasing 12.5 times during 2025, with 27% of bot attacks targeting APIs. Those APIs are also where legitimate agents increasingly need to check inventory, retrieve data, and execute actions.

Cryptographic systems such as Web Bot Auth could give each agent a verifiable identity. That would help a site know who sent the request and whether the agent has authorization from a user. Verification still leaves room for bad behavior, although it gives websites far more than browser movements and IP addresses to work with.

Will AI agents help online stores more than they hurt them?

For online stores, AI agents currently look like a promising new sales channel.

Retailers earn money when a purchase happens, so they can benefit even when an assistant controls discovery. Shopify reported that AI-driven traffic to its stores grew eightfold year over year in the first quarter of 2026, while orders from AI-powered searches rose nearly thirteenfold. New buyers placed orders through AI channels at almost twice the rate seen in other channels.

Shopping through AI is also moving beyond simple referrals. Shopify and Google co-developed the Universal Commerce Protocol so agents can understand products, prices, inventory, discounts, and checkout through a shared format. Every Shopify merchant can now expose synchronized commerce data to AI surfaces without building a separate integration for each assistant.

The cost may show up in the customer relationship. Merchants may gain sales while losing control over product discovery, cross-selling, loyalty, and brand presentation. An assistant can compare many stores instantly and send the order to whichever merchant best satisfies its ranking rules.

Agent-driven sales can grow while merchants become more dependent on the companies running the agents. That resembles the shift from independent websites to Amazon marketplaces: more demand, paired with a new gatekeeper.

If you want more recent data on this point, please see our latest agentic AI market report.

Are people actually letting AI agents buy things yet?

Only in small numbers; AI-assisted shopping is growing quickly, while fully delegated purchasing remains an early market.

Millions of people now ask assistants to find products, compare prices, or narrow a shortlist. Adobe’s retail data and Shopify’s order growth show that this behavior has moved well beyond experiments. Yet most purchases still end with a person reviewing the choice and approving checkout.

Visa said it and its partners completed hundreds of secure agent-initiated transactions during 2025. That was an important technical milestone and a rounding error beside ordinary ecommerce volumes. Payment networks are still building the trust layer: agents need clear spending limits, merchant verification, proof of user consent, and rules for disputes or returns.

The difference between “help me choose a hotel” and “book any suitable hotel under $250” is enormous. Consumers are comfortable with the first instruction today. The second hands over money, preferences, and liability. Adoption will depend less on whether agents can click the button and more on whether people trust what happens when the choice is wrong.

Will fewer website visits lead to less original content?

Useful public content will shrink if creators keep losing visits, revenue, and recognition while AI products reuse their work.

Wikipedia offers a warning because its model depends on readers becoming editors, donors, and volunteers. After improving its bot detection, the Wikimedia Foundation found human pageviews down roughly 8% from the same period a year earlier. It linked the decline partly to search engines and generative AI answering questions with material drawn from Wikipedia.

Commercial publishers face the same loop with less protection. The Reuters Institute’s 2026 survey found news leaders expecting search referrals to fall another 43% over the next three years. Fewer visits mean less advertising, weaker subscription funnels, and less information about what readers actually value.

Plenty of content will survive. Companies publish documentation to support products. Governments release data because they must. Universities, enthusiasts, and communities create knowledge for reasons beyond advertising. Premium outlets can charge users or license archives.

The first losses will probably come from independent information that looks replaceable: niche guides, specialist reviews, local reporting, technical tutorials, and carefully maintained reference pages. We may end up with abundant AI answers built from a thinner and older pool of public sources.

Can websites make AI companies pay for what they use?

Websites can increasingly charge AI companies for access, although the money and standards are still far too limited to fund the open web.

Publisher licenses already show that AI companies will pay for valuable material. The trouble is that private contracts favor large media groups with lawyers, unique archives, and enough scale to negotiate. A small expert website cannot realistically sign a bespoke agreement with every model provider.

Pay-per-crawl gives smaller sites a possible route, although a crawl is a crude unit of value. One page might be downloaded ten times and never influence an answer; another might be fetched once and support thousands of answers. Cloudflare is now testing pay-per-use models with companies including Ceramic.ai and You.com, so payment can follow an answer or request rather than raw retrieval.

Authentication and payment standards are also taking shape. Web Bot Auth can verify the operator behind a request. HTTP 402-style systems can let an agent pay automatically for a page or API call. Freshness signals can tell crawlers when a page has changed, cutting pointless repeat visits.

The likeliest result is a two-tier web: open pages for people and paid interfaces for machines. That would beat unlimited scraping, while giving infrastructure and AI companies enormous influence over prices. The core questions now are who gets paid, how much they receive, and whether small sites can take part.

Proposed fix	What it improves	What still goes wrong
Publisher licensing	Pays selected content owners	Large publishers have far more bargaining power
Pay per crawl	Gives small sites a price for access	Downloads do not equal value created
Pay per use	Links payment to an answer or action	Attribution across many sources is difficult
Verified agent identity	Separates known operators from anonymous bots	A verified agent can still behave badly
Machine payments	Lets agents buy pages or API calls instantly	Standards and prices still vary
Freshness signals	Cuts unnecessary repeat crawling	Both sides must support and respect them

Is AI agent traffic killing the internet?

Partly: AI traffic is currently weakening the open web’s business model far more than it is threatening the internet’s technical infrastructure.

The physical network is in no danger. Machine requests can hurt individual origins, but text crawling remains tiny beside video, cloud services, and other data-heavy human activity. Rate limits, caching, and better agent identification can control much of the technical load.

The economic damage is already visible. Bots now produce more than half of web requests. AI crawlers read far more pages than AI services return as visits. Google users click less when summaries appear. Organic Google traffic across more than 2,500 sites fell by one-third in a year, while AI referrals remain a very small traffic source.

Retailers have a plausible upside because a qualified visitor or authorized agent can still place an order. Publishers and reference sites face the harsher bargain: their information can satisfy the user before anyone reaches the page that paid to create it.

The internet will keep expanding. The open informational web will probably become smaller, more gated, and more commercial. Search engines may retain free access. Training crawlers will face more blocks and fees. Trusted agents will use authenticated feeds. Premium sources will move behind subscriptions, licenses, and paid APIs.

Our judgment is clear. The internet people use for video, communication, software, and shopping will keep growing. AI traffic is already eroding the system that funded a huge amount of freely accessible knowledge, and without better payments, attribution, and direct audiences, the web will still work while offering fewer independent places worth visiting.

If you want more recent data on this point, please see our latest agentic AI market report.

Who is the author of this content?

NEW MARKET PITCH TEAM

We track new markets so founders and investors can move faster

We build living "market pitch" documents for emerging markets: AI, synthetic biology, new proteins, and more. Instead of outdated PDFs or hallucinated LLM answers, our clients get a clean, visual, always-updated view of what's really happening: key players, deals, regulations, and signals that matter. Learn more about us.