---
title: "Meta's AI Crawler Terms Start Nov 20: Should You Block Meta-ExternalAgent?"
slug: "meta-ai-crawler-terms-nov-20-block-meta-externalagent-robots-txt"
type: "blog-post"
url: "https://automationbyexperts.com/blog/meta-ai-crawler-terms-nov-20-block-meta-externalagent-robots-txt"
date: "2026-11-12"
category: "Web Scraping"
tags: ["meta-externalagent", "robots.txt", "ai crawlers", "ai scraping", "website monitoring", "geo"]
read_time: 7
author: "Youssef Farhan"
---

# Meta's AI Crawler Terms Start Nov 20: Should You Block Meta-ExternalAgent?

> Meta delayed its business terms to November 20, 2026 amid a row over AI crawling. Here is what is disputed, which Meta bots exist, and how to verify your robots.txt.

**Short answer:** Meta's revised business terms take effect on November 20, 2026, and critics say they could let Meta crawl websites tied to business accounts for AI purposes. Meta says its crawling is unchanged and that robots.txt is still the way to opt out, so the practical move is to decide your policy now, put it in robots.txt, and then check that it is actually being served.

## What is changing in Meta's business terms on November 20?

Meta moved the effective date of its revised commercial terms from October 30 to November 20, 2026, after pushback from publishers and regulators, according to [Search Engine Journal](https://www.searchenginejournal.com/meta-delays-business-terms-crawling/594388) and [Android Headlines](https://www.androidheadlines.com/2026/10/meta-delays-business-terms-ai-scraping.html). The dispute is about whether the wording lets Meta crawl sites connected to business accounts for AI use. [PPC Land](https://ppc.land/metas-new-terms-let-it-crawl-business-websites-to-train-ai-from-oct-30/) covered the original October 30 version of the terms.

Reports disagree on how much is really different. Some outlets say Meta removed or withdrew language that appeared to permit scraping for AI training (for example [MediaNama](https://www.medianama.com/2026/10/223-meta-website-scraping-ai-training/)), while others say the crawling permission has not been removed and only the date moved. Legal commentators also point out that the revised wording does not clearly mention robots.txt. Read Meta's current terms yourself before relying on any summary, including this one.

## What does Meta say about robots.txt?

Meta's public position, as reported, is that robots.txt remains the way site owners signal their preferences and that the update does not give it new rights that override those signals. Critics reply that a promise to respect robots.txt is a policy choice, not a contractual commitment, and trade body WAN-IFRA asked Meta to make robots.txt binding in the terms. Legal commentary quoted in the coverage adds that robots.txt is a statement of preference and does not by itself create an enforceable legal right.

That gap is why the sensible reading is: robots.txt is the opt-out that exists today, but it only works if it is correct, live and covers the right bot names.

## Which Meta crawlers should you know about?

| User agent | What Meta says it is for | What blocking it does |
| --- | --- | --- |
| Meta-ExternalAgent | Training foundation AI models or improving products by indexing content directly | Does not remove your site from Meta AI search results |
| Meta-WebIndexer | Indexing for Meta AI search | Does not opt you out of model training |

This distinction comes from the [PPC Land explainer on Meta-ExternalAgent](https://ppc.land/meta-externalagent/) and Meta's webmaster documentation as it is summarised there. Meta's documentation says robots.txt changes can take up to 24 hours to register, so do not test the same day and conclude it failed.

## Should you block Meta's AI crawler?

It depends on what you sell and what you want from AI search. There is no universal answer, but a simple way to decide:

- **Publishers and paid-content sites:** usually block training crawlers, because the content is the product.

- **Local and service businesses:** often allow indexing bots, because being quoted in AI answers can bring enquiries, and block only training agents if they care.

- **Ecommerce and marketplaces:** check whether product pages are being fetched heavily by any bot before deciding, and rate limit where needed.

Whatever you pick, write it down as a rule per user agent. A vague "we block AI" policy is how a bot gets missed.

## How do you check that your robots.txt really does what you think?

Most of the mistakes we see described in the wild are boring ones: a typo in the user agent, a rule placed under the wrong group, a CDN or plugin serving a different robots.txt than the file you edited, or a firewall that blocks or allows bots regardless of the file. A short checklist:

- Open `yourdomain.com/robots.txt` in a private window and confirm the live content, not the file in your repository.

- Confirm there is a group for each bot you care about, for example `User-agent: Meta-ExternalAgent` followed by `Disallow: /`.

- Check that your CDN or bot-management layer is not overriding the file. Our post on [Cloudflare blocking Googlebot and AI crawlers](https://automationbyexperts.com/blog/cloudflare-blocking-googlebot-ai-crawlers-check-your-site) covers that failure mode.

- Wait up to 24 hours for Meta to pick up the change, per its documentation.

- Re-check after every site migration or plugin update, because both can rewrite robots.txt.

The [SEO, GEO & AEO Audit actor](https://automationbyexperts.com/apify/seo-geo-aeo-audit) includes an AI-crawler access check and an llms.txt check in the same run as its page scores and schema validation, so one run gives you a fix list instead of a manual review. Be realistic about llms.txt though: see [does llms.txt work](https://automationbyexperts.com/blog/does-llms-txt-work-ai-crawlers-ignore-it-2026) for the log-based evidence that most AI crawlers ignore it.

## How can you watch for terms and policy changes like this one?

The Meta story is a reminder that the rules you operate under can change on a date you set in a calendar and then miss. If you rely on a platform's terms, you can monitor the terms page itself. The [Website Change Monitor actor](https://automationbyexperts.com/apify/website-change-monitor) watches any page, returns a diff of what was added and removed, supports keyword watching (for example "AI", "crawl", "training") and sends alerts to Slack, Discord, Telegram or a webhook. The first run saves a baseline, and later runs report only changes. For the same idea applied to vendor pricing, see [monitoring SaaS pricing page changes](https://automationbyexperts.com/blog/monitor-saas-pricing-page-changes-before-renewal).

## Does this change whether scraping is legal?

Not directly. The terms are about what Meta may do with *your* website, not about what you may do with Meta's. Meta's own terms have long barred automated access without permission, and it sued the scraping company Octopus in 2022 on terms-of-service and DMCA grounds, though a court ruled that scraping publicly accessible information does not violate the Computer Fraud and Abuse Act (as summarised in the coverage above). Legality depends on the data, the jurisdiction and how you use it, and nothing here is legal advice. Our [public data scraping checklist](https://automationbyexperts.com/blog/is-scraping-public-data-legal-logged-out-rule-2026-checklist) goes through the questions to ask first.

If your work involves checking how AI engines treat your brand, browse the [SEO and AI visibility actors](https://automationbyexperts.com/apify/category/seo-ai-visibility).

## What does a sensible robots.txt look like for AI bots?

Here is a minimal example that blocks Meta's training and indexing crawler while leaving everything else untouched. Adapt it to your policy, and add one group per bot you want to control:

```
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-WebIndexer
Disallow: /
```

Only include the second group if you also want to stay out of Meta AI search indexing. As the table above shows, the two bots do different jobs, so blocking one is not a substitute for the other. Keep the file short, put each rule in its own group, and avoid clever wildcard patterns that a bot may parse differently than you expect.

## What if you also run crawlers or scrapers yourself?

The same discipline applies in reverse. If you collect data from other sites, read their robots.txt and terms first, identify your requests honestly, keep volumes reasonable and collect only what you need. The coverage of this dispute shows how quickly sentiment shifts when a large company is seen to read the web on its own terms, and small operators get less benefit of the doubt. Prefer official APIs or licensed data where they exist, and keep a record of why you decided a source was fine to collect from.

It also helps to separate three questions that often get blurred together: whether a bot is *allowed* by the site's stated preferences, whether it is *legal* in your jurisdiction, and whether it is *wise* for your business given the relationship you want with that site. A yes on one does not guarantee a yes on the others.

## A five-minute plan before November 20

- Decide your policy per bot: allow, block or rate limit.

- Write it into robots.txt and confirm the live file.

- Check your CDN or firewall is consistent with it.

- Set a monitor on Meta's terms page with keywords such as "crawl" and "AI".

- Put a reminder in your calendar to re-check robots.txt a day after the change, since Meta says updates can take up to 24 hours.

## FAQ

**When do Meta's new business terms take effect?**Reports say November 20, 2026, moved from October 30 after publisher and regulator pushback. Check Meta's own terms page for the current date.

**Does blocking Meta-ExternalAgent remove my site from Meta AI search?**According to the PPC Land summary of Meta's documentation, no. Meta-ExternalAgent is described as the training and indexing crawler, and a separate bot, Meta-WebIndexer, relates to search. Blocking Meta-WebIndexer does not opt you out of training.

**Is robots.txt legally binding?**Legal commentators quoted in the coverage say it expresses a site owner's preference but does not by itself create an enforceable legal right. Meta says it still honours it, but critics note that this is a policy, not a contract term.

**How long does a robots.txt change take to work for Meta?**Meta's webmaster documentation says changes can take up to 24 hours to register.

## Check your setup before November 20

Run the [AI-crawler audit and a terms-page monitor on Apify](https://apify.com/fayoussef?fpr=youssef&fp_sid=blog), which has a free plan to try both, and use the result to decide your policy on purpose rather than by default.

Last updated: November 2026

## Actors used in this guide

- [Website Change Monitor - Page & Price Change Alerts](https://automationbyexperts.com/apify/website-change-monitor): It monitors any web page for changes - competitor pricing, products, terms of service, job boards - and returns a cle…
- [SEO, GEO & AEO Audit - AI Search Readiness Checker](https://automationbyexperts.com/apify/seo-geo-aeo-audit): It audits any website for Google SEO and AI-search visibility in one run - returning 0–100 SEO, GEO and AEO scores pe…
- [AI Account Watch - Custom Sales Trigger Alerts for Any Company List](https://automationbyexperts.com/apify/ai-account-watch): It watches a list of companies and alerts you when one of them does what you describe in plain English - raises fundi…
