On October 17, 2026, The Verge reported on websites blocking AI agents — the autonomous assistants from OpenAI, Meta, Google and others that promise to book your flights, manage your inbox and shop on your behalf. The same week, Meta quietly removed AI-training language from its commercial terms after publishers and legal experts accused it of claiming a license to scrape news sites for AI. After years of the open web feeding AI systems for free, the doors are slamming shut. Here is the verified story of what just happened, and what comes next.
Websites Blocking AI Agents: Why It’s Happening
Autonomous agents are the hottest thing in AI. OpenAI, Meta, Google, Instinct and others promise their agents can scour the internet for you, doing anything from booking a dinner reservation to buying new shoes. But a full copy of The Verge’s October 17 report details what happens when the rest of the internet won’t play nice: in recent weeks, users of services like Meta’s Muse, xAI’s Grok, OpenAI’s Dots and Instinct have reported blocks preventing their AI agents from performing tasks across top websites.
The named blockers are not obscure forums. Yelp told TechCrunch it only allows AI agents that pay for its data licensing program. Delta Air Lines is preventing agents from booking tickets on behalf of users. JPMorgan Chase and Charles Schwab have confirmed they are blocking AI agents from accessing accounts, citing security concerns. And it is not just the big names: some users say sites that previously worked with their AI agents are no longer accessible — including Cars.com and Amtrak — suggesting other companies could be quietly blocking them too.
Amazon was already fighting this battle last year, when it banned Perplexity’s agentic shopping feature and sued the AI startup. At the time, Amazon claimed it had “repeatedly requested” that Perplexity stop allowing its AI agents to shop on its website, adding that third-party companies that purchase items on behalf of customers “should operate openly and respect service provider decisions whether or not to participate.”
Meta, for its part, pushed back on the trend earlier this month. “Turning away a personal agent means turning away the customer behind it,” the company said, adding that it would rather work with businesses to address the underlying concerns than see them lose that customer. Instead of fighting blockers head-on, Meta has partnered with retailers including Walmart, Gap, Sephora and Best Buy to let Muse shop on their sites — and Shopify has opened its network of storefronts to AI agents. The Verge reached out to OpenAI and Google for comment on the roadblocks their agents face but did not immediately hear back; Instinct declined to comment.
Even where there is no deliberate ban, agents are colliding with the web’s anti-bot infrastructure. Reports on Reddit suggest Google is blocking some accounts that use Muse due to “suspicious activity,” while others have reported problems checking out on Walmart despite its Meta partnership. As one expert put it in the report, many of these systems were not even built to block agents — they were built to block bots and programs trying to scam or hack, and AI agents trip the same safeguards.
The motives are layered: security fears, the cost of serving bot traffic, and money.
Meta’s Terms U-Turn: The AI-Training Clause That Vanished
The second half of the story broke just days before The Verge’s piece. On October 15, Nieman Lab surfaced the wording of Meta’s updated commercial terms, scheduled to take effect October 30, 2026. The terms would have had business accounts on Facebook, Instagram or WhatsApp automatically consent to Meta’s web scrapers crawling and collecting content from websites “associated with” those accounts — simply by continuing to use the platforms. The key passage said Meta could use content from those domains, web pages or URLs to “provide, develop, improve, research, train, and maintain the integrity of” its products. As Ars Technica documented, the update also required business account holders to let Meta deploy “automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents)” — and even to “create test credentials” — to access their sites.
Three legal experts who reviewed the terms for Nieman Lab read that wording as a potential license to train Meta’s AI systems, including Llama, on publishers’ own websites — no deal required. Matt Topic, a media and intellectual property attorney at Loevy & Loevy, called it “a remarkable provision to bury in terms of service,” noting it would give Meta access to material “that would likely cost many millions of dollars in a licensing market.” Cornell professor of digital and information law James Grimmelmann said the terms could let Meta train AI on any content from business account holders’ sites — and critics worried the clause could let Meta sidestep the robots.txt files sites use to block AI crawlers, since virtually every news organization operates at least one Meta business account.
Meta denied the interpretation before revising anything. On October 16, Andy Stone, Meta’s vice president of communications, called the reports false, said there was no change to how Meta uses web crawlers, and said the update was about pulling product information from third-party sites for ad and content recommendations — not scraping news sites. In a follow-up statement, spokesperson Daniel Roberts added: “We’ve heard you and want to clarify: the crawling terms we consolidated only applied to our ads and business products, and we’ve clarified that in the terms themselves. This is not about scraping news sites, and these updates don’t give us new rights that supersede those preferences.”
Then, sometime around October 16–17, the offending language disappeared — a before-and-after comparison shows the word “train” stripped out and crawling tied to ads and business products instead, while the crawling permission, the test-credentials power and the October 30 effective date all remain. The Verge’s October 17 report (by Terrence O’Brien) confirmed the passage no longer appears on Meta’s commercial terms page.
The Wider Crackdown: How the Web Learned to Say No
The Meta fight is the loudest episode of a trend that has been building all year. The web’s original truce with bots — the robots.txt file, a voluntary “please don’t crawl this” note — is giving way to something harder. An analysis of robots.txt data across the internet’s top domains found that of 1,530 domains with a reachable robots.txt file, 292 — 19.1% — now disallow at least one of the four major AI crawlers (GPTBot, ClaudeBot, Google-Extended or PerplexityBot), a sharp jump from a year earlier when blocking was still a fringe move. Amazon blocks all five major AI crawlers outright (GPTBot, ClaudeBot, Google-Extended, PerplexityBot and Bytespider), and Instagram, X, Pinterest and TikTok sit alongside it. Sites on Cloudflare’s network denied 13.47% of AI bot requests in the third quarter of 2026, up from 5.98% a year earlier, per Cloudflare Radar data analyzed by TechnologyChecker.io and reported October 9 — while search-triggered AI crawl requests nearly doubled in the same period. AI systems want to read the web more than ever; owners keep saying no.
The biggest lever moved by Cloudflare. On July 1, 2026, the company began classifying AI traffic by purpose, sorting crawlers into three buckets: Search (indexing), Training (scraping data to build models) and Agent (bots acting on a user’s behalf in real time). Effective September 15, 2026, new domains onboarding to Cloudflare — plus existing free-tier customers and new sites from existing customers — get the Training and Agent categories blocked by default on any page that carries advertising, while search indexing stays welcome. Cloudflare’s new controls end the long-standing trade-off that forced publishers to choose between search discoverability and control over their content. “This is how we make the Internet better: preserving the openness that makes search valuable while giving the people and businesses behind the web meaningful control over how their work is used,” said Matthew Prince, Cloudflare’s co-founder and CEO. According to Cloudflare, 17% of site owners now restrict AI training while fewer than 1% block search crawlers — and mixed-use crawlers, which blend search, agent and training use in one bot, account for 36.6% of verified crawler traffic.
Declared policy and real enforcement still diverge. The HasData AI Crawler Block Index found that 39.5% of sites whose robots.txt bans GPTBot still served it a live page — meaning the ban lived in a text file while the firewall waved the bot through. Publishers, though, block AI crawlers 5.5 times as often as the open web. And the lighter-touch alternatives have flopped: many sites now publish llms.txt files to guide AI systems, but Ahrefs’ server-log research found 97% of those files received zero requests in May 2026.
Lawmakers are stepping in as well. In June, New York’s State Assembly passed a bill requiring AI crawlers to identify themselves to news publishers, with fines of $15,000 a day for violations. Meanwhile, blocking has costs that give site owners pause: a Rutgers–Wharton study found news publishers blocking large-language-model crawlers lost about 7% of weekly visits within six weeks, and BuzzStream found that 92.3% of top news sites blocking Google’s AI controls still appeared in AI citations anyway — meaning the block cost them traffic without buying them invisibility. The question every publisher now faces is not just how to block AI agents, but whether blocking actually achieves anything.
What It Means for You
If you use AI tools: expect friction. Your agent may hit walls on sites it handled fine last month — flights, restaurant data, account logins — and may get your own accounts flagged for “suspicious activity.” The era when an agent could roam the whole web on your behalf is narrowing to the sites that have cut a deal with your agent’s maker. Check whether a task failed because of a real block before blaming the assistant.
If you run a website or small business: this is a genuine fork in the road. Blocking training and agent bots protects your content and bandwidth, but it can cost you visibility in AI answers — and, per the research, roughly 7% of weekly visits. The Cloudflare default now makes blocking the path of least resistance on ad-supported pages, but a deliberate choice beats a forgotten toggle: a security plugin or copied robots.txt template may already be blocking AI crawlers you actually want. For small businesses, the block-or-allow decision is a strategy question that sits next to the fundamentals — it’s worth revisiting how to write a business plan before deciding whether the short-term safety of blocking is worth the traffic risk. And if you’re figuring out how to start an online store with no money to spend on licensing or ads, remember the one asset you fully control is a website that search engines — and AI assistants — can still find.
If you market or publish content: the game is becoming pay-to-be-read. Yelp’s licensing-only stance and Meta’s Walmart/Shopify partnerships point to a future where agent access is negotiated, priced, and metered — the same direction training data already went with publisher licensing deals. The licensing market that Meta’s critics priced at “millions, or billions” is real, and your content may be inventory in it.
What to Watch Next
October 30 is the first date on the calendar: Meta’s revised commercial terms take effect, and publishers will be watching whether the crawling permissions stay narrowly scoped to ads and business products. The New York crawler-identification bill will test whether legislation can do what robots.txt could not. The Amazon–Perplexity lawsuit will reveal whether “agents shopping on your site without permission” is a terms-of-service violation, a security incident, or a business model. And the market experiment — Yelp charging for agent access, Walmart and Shopify partnering with Meta’s Muse — will reveal whether the web settles on licensing, partnership, or permanent blockade. The walls went up fast in 2026; whether they stay up depends on who pays to take them down.
Frequently Asked Questions
Which websites are blocking AI agents?
Per The Verge’s October 17, 2026 report, users have hit blocks at Yelp (which requires paid data licensing), Delta Air Lines (which blocks agent ticket bookings), JPMorgan Chase and Charles Schwab (security-based account blocks), plus reported trouble at Cars.com and Amtrak. Amazon blocked Perplexity’s agentic shopping feature last year and sued the company. Amazon also blocks all five major AI crawlers from scraping its site.
Did Meta really try to let itself scrape news sites for AI training?
Meta says no. Nieman Lab reported on October 15 that Meta’s updated commercial terms contained language — “provide, develop, improve, research, train, and maintain the integrity of” products — that three legal experts read as permission to train AI on publishers’ websites. After publisher backlash, Meta removed the AI-training language around October 16–17, tying crawling to ads and business products instead. Meta insists its crawler practices never changed and that robots.txt preferences are still honored.
Can websites actually stop AI crawlers with robots.txt?
Only partly. Robots.txt is a voluntary signal many established crawlers respect, but it can’t be enforced — crawlers can ignore it or misidentify themselves. The HasData index found 39.5% of sites that ban GPTBot in robots.txt still served it a page. Real enforcement happens at the CDN or firewall layer, which is why Cloudflare’s default block on ad-supported pages matters more than any text file.
Why is Cloudflare blocking AI agents by default?
Cloudflare says it wants to end the trade-off between search visibility and content control. Since September 15, 2026, new and free-tier sites get “Training” and “Agent” crawler categories blocked by default on ad-supported pages, while search crawlers stay welcome. CEO Matthew Prince framed it as giving site owners meaningful control over how their work is used; 17% of site owners already restrict AI training.
Will AI agents eventually have to pay for access?
That is where the trend points. Yelp already requires paid data licensing for agent access, Meta is partnering with retailers like Walmart and Shopify instead of fighting blockers, and analysts argue the standoff likely ends in licensing fees similar to AI training-data deals. The question is who pays: the agent companies, the users, or both.
Conclusion
October 2026 will be remembered as the month the web started charging admission. The Verge’s reporting showed the doors closing on AI agents at some of the internet’s biggest sites; Meta’s terms U-turn showed how fiercely publishers will fight any attempt to turn a platform account into a scraping license; and Cloudflare’s defaults showed that the infrastructure layer has already picked a side. The open web was built on the assumption that bots were guests. In 2026, websites decided the guests had overstayed — and sent them the bill.
