“Server error (5xx)” means Googlebot asked for the page and your server answered with an error in the 500 range instead of the page. Google retries later. If the error persists, it slows down crawling of the whole site and eventually drops the affected URLs from the index.
The first question is whether the error is still happening. Many 5xx reports come from a short outage, a deploy or a rate limit that has since passed.
Is it still happening?
- Open an affected URL in URL Inspection and click Test live URL. If the live test fetches the page, the error was temporary.
- Request the same URL several times from outside:
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/page. - Open Settings → Crawl stats and look at By response: a steady share of server errors is a real problem; a spike on one day points to an outage.
A real case from our own audits: on one site, 152 internal pages answered 500 during a crawl. Requested again a few seconds later, most of them answered 200, and a later audit found no errors at all. The server, or a firewall in front of it, was refusing a burst of requests from one address. To a visitor the site worked; to a crawler it looked broken. Googlebot also crawls in bursts.
What each code usually means
| Code | Usual cause |
|---|---|
| 500 Internal Server Error | A bug or an exception in the application, a failed database query, a broken plugin |
| 502 Bad Gateway | The proxy or CDN could not get an answer from the application server |
| 503 Service Unavailable | Maintenance mode, overload, or a rate limit. Google treats 503 as temporary |
| 504 Gateway Timeout | The application took too long; slow queries or external calls |
How to fix it
- Read the server logs for the affected URLs at the times Google reported. The error log usually names the failing code path.
- Check capacity and limits. Worker or connection limits, memory, database connections. If a firewall or CDN rate-limits bots, make sure verified Googlebot is not limited.
- Fix templates that fail for some pages only. When the errors share a URL pattern, one template or one missing data field is often the cause.
- Use 503 with
Retry-Afterfor planned maintenance, not 500 and not a 200 page saying “back soon”. - Monitor 5xx rates, so the next outage is noticed before Google reports it.
Once fixed, click Validate fix on the reason.
How to check that it worked
- Live tests of affected URLs succeed.
- Server errors in Crawl stats fall back to near zero.
- The reason's count drops during validation.
What SignalCrawler checks here
URLs that answer with a 5xx during the audit's crawl are asked again, one at a time, after a pause, and once more after a longer pause. Pages that fail every time are reported as server errors; pages that work on the second try are reported separately as errors during the crawl, a sign of load problems or rate limiting. The audit also flags a robots.txt that answers with a server error and slow server responses.