A 429 Tells Googlebot to Slow Down—Fix the Limit Before Chasing Crawl Budget

An HTTP 429 Too Many Requests response tells Googlebot that the server is overloaded or rate-limiting requests. Google treats it like a server error and temporarily reduces crawling; after the server resumes 2xx responses, Google gradually raises the crawl rate again (Google’s HTTP status guidance).

That does not mean every small business site has a crawl-budget problem. Google’s crawl-budget guide is aimed mainly at sites with at least 1 million moderately changing pages, sites with 10,000 or more rapidly changing pages, and sites with many URLs marked “Discovered—currently not indexed.” For a smaller site whose new pages are crawled promptly, Google says an updated sitemap and regular Page Indexing checks are generally enough (Google’s crawl-budget guide).

The practical question is narrower: Are 429s stopping Googlebot from reaching pages that matter to the business?

Diagnose the source before changing crawl controls

Start with the period when 429s appeared. In Search Console, open Settings → Crawl stats, inspect the response-code breakdown and compare the affected dates with hosting, CDN and application logs. Google says the report shows request volume, server responses and availability issues, although its example URLs are representative rather than comprehensive (Search Console Crawl Stats documentation).

For each 429 request, record:

• timestamp, hostname and requested URL;
• source IP and user agent;
• whether the response came from the CDN, web application firewall, web server or CMS security/rate-limit plugin;
• the rule or threshold that fired;
• whether normal visitors or other bots were affected at the same time.

Do not trust a Googlebot user-agent string by itself. It can be spoofed. Verify requests with reverse DNS or Google’s published IP ranges before creating any exception (Googlebot documentation).

Then classify the incident:

Evidence	Likely issue	Appropriate response
A security rule rejects verified Googlebot while the origin has spare capacity	Misconfigured bot or rate-limit rule	Correct the rule narrowly; do not disable protection for all bots
The origin is slow or saturated and several request types fail	Real capacity or application problem	Fix slow queries, caching, rendering or hosting capacity
A large set of duplicate, filtered or low-value URLs absorbs requests	URL-inventory problem	Consolidate duplicates and use durable crawl controls where appropriate
Only a few isolated 429s appear and important URLs continue receiving 200 responses	Transient event	Monitor rather than launch a crawl-budget project

Use 429 only as temporary protection

If Googlebot is genuinely overwhelming an unhealthy server, a temporary 429 is preferable to returning a fake 200 page. But it is an emergency brake, not a permanent crawl-management setting.

Google recommends temporary 429 or 503 responses when its crawling overloads a server, says Googlebot will retry affected URLs for about two days, and warns that continuing those responses for more than a few days can cause URLs to be dropped from the index (Google’s crawling-error guidance). Stop issuing the errors once load is controlled.

Do not try to solve this by repeatedly submitting the same sitemap, intermittently hiding directories in robots.txt, or blocking important pages. A sitemap helps discovery but does not require immediate crawling. If excessive URLs are the underlying problem, address them deliberately: consolidate duplicates, remove obsolete URLs correctly and block only areas that should not be crawled over the long term. For a broader technical baseline, review accessible and indexable page priorities.

Prove recovery with a short evidence chain

A passing one-off URL test is useful, but it is not enough. Keep monitoring until all four conditions hold:

1. Delivery: CDN, WAF and origin logs show that verified Googlebot requests now receive 200, valid redirects, 304, or intentional 404/410 responses instead of 429s.
2. Availability: Search Console’s Crawl Stats response breakdown and Host status no longer show a continuing incident.
3. Recrawl: Logs show Googlebot returning to a small sample of commercially important pages, and URL Inspection can fetch them without a host-load warning.
4. Indexing: Page Indexing and URL Inspection show whether those pages remain eligible and indexed. Crawling alone does not guarantee indexing.

Choose the sample before checking: the homepage, a high-value service or product page, one recently updated article and one page that returned a 429. Record the date of the fix, status codes before and after, last observed Googlebot request and Search Console result. This creates a defensible recovery record without implying that every fluctuation in crawl requests changed rankings or pipeline.

Escalate to the host, CDN or developer when 429s continue across important URLs, robots.txt itself returns 429, verified Googlebot remains blocked after a rule change, or server response times and errors rise together. Otherwise, stable successful responses and confirmed recrawls are stronger evidence than trying to maximize crawl volume for its own sake.