Why Robots.txt Cannot Block Brave AI Alone

Brave crawls public websites through its search crawler, which does not advertise a differentiated Brave user agent. That means a User-agent: Bravebot rule does nothing: Brave documents no such token, and robots.txt does not prevent a URL from remaining indexed. To remove a page from Brave’s index, publish a page-level noindex directive, leave the page accessible long enough for Brave to see it, and submit the URL for re-fetching.

For B2B teams that already block GPTBot, ClaudeBot or PerplexityBot, this is the critical difference. Those services publish bot-specific controls. Brave does not provide an equivalent way to block only its crawler while preserving Google crawling.

Select the crawlers you want to control, paste your robots.txt, and check the result for the requested path.

Brave Uses an Unidentified Search Crawler

Brave says its search engine maintains its own crawler and independent web index. The crawler does not advertise a differentiated user agent because, according to Brave, doing so could cause sites that allow only Google to exclude it. Brave also says that if a domain or page is not crawlable by Googlebot, its crawler will not crawl it either (Brave Search crawler documentation).

This creates two separate control questions:

• Can you stop Brave from fetching a page without also affecting Google?
• Can you stop the page from appearing in Brave’s index and downstream products?

Brave provides no documented user-agent token for the first task. A rule such as the following has no documented crawler to match:

User-agent: Bravebot
Disallow: /

“Bravebot” is a convenient label for discussing Brave in crawler inventories, not a user-agent value Brave tells publishers to use.

A block applying to all crawlers or to Googlebot can indirectly stop Brave from fetching the affected URL under Brave’s stated policy. That is not a Brave-specific control, however. It can also sacrifice Google crawling and prevent compliant crawlers from seeing a later noindex directive.

Other AI Controls Do Not Transfer to Brave

GPTBot, ClaudeBot and PerplexityBot have named user-agent tokens. Their operators provide instructions for using robots.txt to manage the corresponding crawlers. OpenAI documents GPTBot in its crawler and user-agent guidance, Anthropic documents ClaudeBot in its web-crawling guidance, and Perplexity documents PerplexityBot in its bot documentation.

A team can therefore create a named group for one of those bots and apply a path-specific or sitewide disallow rule. As with any robots.txt instruction, this is a request to a compliant crawler rather than an authentication mechanism.

Google-Extended belongs in the same audit but works differently. Google describes it as a standalone product token rather than a crawler. A Google-Extended rule controls the uses specified in Google’s documentation without controlling inclusion or ranking in Google Search (Google crawler documentation).

Brave has no corresponding publisher token. Adding named blocks for four other AI services does not cover Brave by implication, and a wildcard block changes access for more than Brave.

Brave Can Discover URLs Through Its Crawler and WDP

Brave’s ordinary crawler is not its only discovery input. Its crawler is partially powered by the opt-in Web Discovery Project, or WDP. Participating Brave browsers can contribute anonymous data such as visited URLs, page metadata and engagement signals. Brave says these contributions help identify useful or novel pages and improve relevance (Brave Web Discovery Project documentation).

Brave distinguishes routine WDP contributions from page-content collection. It says routine metadata contributions do not include page content. WDP can also process selected fetch jobs from Brave’s crawler and return fetched HTML to Brave; its documentation says these jobs are limited to a preselected set of innocuous, reputable domains.

Site owners can also submit URLs manually. Submission requests a fetch. It does not promise that Brave will index, rank, quote or cite the page.

Discovery, crawling and indexing remain separate stages:

1. Brave learns that a URL exists.
2. Its crawler attempts to fetch the resource.
3. Brave decides whether to retain the resource in its index.
4. A later query may retrieve the page for search results, an AI answer or an API product.

Blocking one discovery route does not necessarily remove a URL already held in the index.

Noindex Controls Brave Indexing, Not Initial Access

Brave explicitly says robots.txt should not be used to prevent indexing. It directs site owners to publish a noindex directive instead.

For an HTML page, place a robots meta tag with the value noindex in the document head. For a PDF or another non-HTML resource, return an X-Robots-Tag: noindex HTTP response header.

The crawler must fetch the resource to see either instruction. If the URL is blocked first, Brave may be unable to discover the new directive. Google’s technical documentation confirms both implementations and explains why a robots block can prevent a crawler from seeing noindex (Google Search Central).

The practical removal sequence is:

1. Add noindex to the page or response.
2. Make sure the resource is not blocked from the crawler that must read the directive.
3. Submit the URL to Brave for re-fetching.
4. Check Brave later to see whether the indexed result has disappeared.

Brave says it must fetch the updated page before it can delist it. Its documentation does not provide a guaranteed processing time, so no precise removal interval is available.

This process controls future index presence. Brave’s public crawler documentation does not say that delisting deletes data an API customer may already have been permitted to store.

Match the Control to the Business Goal

Goal	Appropriate Control	Main Trade-Off
Appear in Brave search and AI answers	Keep the page public, Googlebot-accessible and free of noindex	Eligibility does not guarantee retrieval or citation
Remove an indexed page	Add noindex, permit re-fetching and submit the URL again	Removal waits for Brave to process the update
Stop Brave from fetching a section	Block Googlebot or all compliant crawlers	No Brave-only isolation; Google visibility may also be lost
Keep material confidential	Require authentication or deny unauthorized server requests	The content cannot remain publicly discoverable

robots.txt is not access control. Confidential documents, customer portals, paid resources and unpublished files should be protected by authentication or server authorization rather than crawler instructions.

For public demand-generation pages, the usual policy is different. Keep valuable product, comparison and educational pages crawlable and indexable, then measure whether Brave retrieves them for buyer questions. Apply noindex only where index exclusion is the actual business requirement.

Brave’s Index Can Supply AI Answers and APIs

Brave’s index supports more than its conventional search results. Brave offers search results, AI-oriented context and answer products to outside developers through the Brave Search API.

Indexing therefore creates eligibility for several forms of retrieval. It does not mean every indexed page will be quoted, cited or delivered to an AI application. Query interpretation, ranking and answer generation happen after indexing.

This distinction also matters when comparing Brave with systems that operate their own named crawlers or retrieve pages through search partners. The pathways are examined separately in our guide to how Claude crawls and retrieves websites.

For measurement, check ordinary search visibility and AI-answer citations independently. A page can rank without being cited in an answer, while a cited page may generate little qualified traffic. Record the query, cited URL and subsequent visits rather than treating the citation itself as a business outcome.

Audit Brave Access Without Looking for Bravebot Logs

Start with the response Brave would receive. Confirm that a page intended for discovery returns a successful response, exposes its substantive text without a login and does not carry an accidental noindex directive.

Next, review Googlebot access. Under Brave’s published policy, a URL that is not crawlable by Googlebot will not be crawled by Brave either. Check both named Googlebot groups and wildcard groups in robots.txt, including path-specific rules inherited from old migrations or staging configurations.

Do not expect server logs to contain an authoritative Bravebot label. Brave’s lack of a differentiated user agent means a user-agent filter cannot reliably isolate its requests.

After publishing a priority page or adding noindex, submit that URL to Brave. Then search Brave using the exact page title and distinctive text from the page. Failure to appear in one query is a useful diagnostic, not definitive proof that the URL is absent from every part of the index.

News Corp Alleges Brave Masks Its Crawlers

On July 21, 2026, News Corp sued Brave Software and alleged that Brave “masks its web crawlers such that publishers cannot detect or reliably block them.” News Corp also alleged that Brave sells approximately 250-word content snippets to AI companies through a “Data for AI API,” characterizing the conduct as “theft by a masked intruder.” These are allegations, not court findings (Semafor).

The dispute followed Brave’s February 2025 suit against News Corp concerning a cease-and-desist. Brave has argued that its indexing, snippets and high-level summaries are fair use, while News Corp disputes Brave’s collection and commercial use of its content (TheWrap).

The litigation does not establish that every Brave crawl or API result is unlawful. It does make the technical limitation more consequential for publishers: there is no published Brave-only crawler identity they can block while continuing to welcome Googlebot.

For a B2B marketing site seeking discovery, the operational choice remains whether each page should be public and indexable. For publishers whose articles are the product, the inability to separate Brave crawling from Google access may require a combined technical, commercial and legal decision rather than another routine robots.txt entry.