Search Console Crawl Errors: 404s, Redirects, Server Errors & More
Crawl errors occur when Google tries to visit a URL on your website but encounters a problem. These errors prevent Google from indexing your pages, which means those pages cannot rank, cannot get impressions, and cannot generate traffic. While crawl errors are not as nuanced as index coverage issues, they are more urgent — a crawl error means a page is completely inaccessible to Google. Here is every type of crawl error and how to fix it.
What are crawl errors?
A crawl error is any error Google encounters when trying to access a URL on your website. When Googlebot (Google\'s crawler) visits a page, it sends an HTTP request and expects a response — typically a 200 OK status with the page content. If anything prevents this — a 404, a server error, a redirect loop, a blocked page — Google logs a crawl error.
Crawl errors are important because they directly prevent indexing. If Google cannot crawl a page, it cannot index it. If it cannot index it, the page is invisible in search results. A single crawl error on an important page can eliminate all traffic to that page.
Google Search Console reports crawl errors in the Page Indexing report, categorised by type. Each error type has a different cause and a different fix.
Types of crawl errors
| Error Type | HTTP Status | What It Means |
|---|---|---|
| 404 Not Found | 404 | The page does not exist at the URL Google tried to crawl |
| Server error (5xx) | 500, 502, 503 | Your server is down, overloaded, or misconfigured |
| Redirect error | Various | A redirect loop, broken redirect, or too many redirects |
| Soft 404 | 200 (false) | The page returns 200 but has no real content (e.g., 'page not found' message) |
| Blocked by robots.txt | N/A | Your robots.txt file prevents Google from crawling the URL |
| DNS error | N/A | Google cannot resolve your domain name to an IP address |
404 errors: the most common crawl error
A 404 error means the page Google tried to crawl does not exist. This happens when a page has been deleted, a URL has been changed, or an internal/external link points to a URL that was never created. Google finds these URLs by following links — from your own site, from external sites, or from its previous crawl data.
Deleted pages with remaining links
If you delete a page but other pages still link to it, Google will try to crawl the link and get a 404. Fix: either restore the page, add a 301 redirect to a relevant alternative, or remove the links pointing to the deleted URL.
URL changes without redirects
If you change a page URL (e.g., /old-page to /new-page) without adding a redirect, Google will try the old URL and get a 404. Fix: add a 301 redirect from the old URL to the new one.
Broken external links
Other websites may link to URLs on your site that do not exist. You cannot control their links, but you can redirect common misspellings to the correct pages. Google will stop reporting 404s for URLs that return redirects.
Not all 404s are bad. If a page was intentionally deleted and no equivalent exists, a 404 is the correct response. Google will eventually stop trying to crawl it. Only fix 404s that affect pages you want indexed — redirect them to relevant alternatives, or restore the content.
Server errors (5xx)
Server errors (500, 502, 503) occur when Google tries to crawl your page but your server fails to respond properly. Unlike 404s, these errors mean your server is having a problem — not that the page does not exist.
500 Internal Server Error
Your server encountered an unexpected condition. This is often a code error, a database failure, or a configuration issue. Fix: check your server error logs and fix the underlying issue.
502 Bad Gateway
Your server received an invalid response from an upstream server. This often happens with proxy servers or load balancers. Fix: check your server infrastructure and upstream services.
503 Service Unavailable
Your server is overloaded or undergoing maintenance. Google understands this is temporary, but if it persists, Google may reduce crawl frequency. Fix: ensure your server can handle traffic, or return a Retry-After header.
Redirect errors
Redirect errors occur when a redirect does not work properly. Google encounters these when following a redirect chain that is broken, loops back on itself, or has too many hops.
Redirect loops
Page A redirects to Page B, which redirects back to Page A. Google cannot resolve this and gives up. Fix: break the loop by ensuring the final destination returns a 200 without redirecting.
Broken redirect chains
Page A redirects to Page B, but Page B does not exist (404). Fix: update the redirect to point to a valid destination.
Too many redirects
Google follows up to 5 redirects. If your chain is longer, Google stops and reports an error. Fix: shorten the chain by pointing the first redirect directly to the final destination.
Soft 404s: the hidden crawl error
A soft 404 is one of the trickiest crawl errors. The page returns a 200 OK status (meaning the server is responding), but Google determines the page is effectively a 404 because it has no real content. This often happens when:
Empty search results pages
When an internal search returns no results, the page often says "no results found" but returns a 200 status. Google sees this as a soft 404. Fix: return a proper 404 status, or noindex search results pages.
Thin or empty pages
Pages with very little content (e.g., a product page with no description, or a category page with no products) can be flagged as soft 404s. Fix: add substantial content or return a proper 404.
"Page not found" messages without 404 status
Some CMS systems display a "page not found" message but return a 200 status instead of 404. Fix: configure your CMS to return the correct 404 HTTP status code for missing pages.
Blocked pages and robots.txt errors
Sometimes crawl errors are intentional — your robots.txt file blocks Google from crawling certain URLs. This is not always a problem; many sites block admin pages, internal search, or duplicate content intentionally. The issue arises when you accidentally block important pages.
Robots.txt blocks
A disallow rule in your robots.txt prevents Google from crawling matching URLs. Check your robots.txt file to ensure you are only blocking pages you do not want indexed.
Noindex tags
A noindex meta tag or X-Robots-Tag header tells Google not to index a page. This is intentional but can be accidentally added to important pages. Check your page headers if important pages are not being indexed.
DNS errors
If Google cannot resolve your domain name to an IP address, it cannot crawl any pages. This is usually a temporary DNS issue. If it persists, check your DNS configuration with your hosting provider.
How to fix crawl errors step by step
- 1Open Search Console and check the Page Indexing report for crawl errors.
- 2Identify the error type (404, server error, redirect error, soft 404, blocked).
- 3Use URL Inspection to get details about the specific URL and error.
- 4Fix the underlying issue: restore deleted pages, fix server errors, resolve redirect loops, add content to thin pages, or update robots.txt.
- 5Request re-crawling via URL Inspection to validate the fix.
- 6Monitor the report over the following days to confirm the error is resolved.
Crawl errors vs index coverage issues
Crawl errors and index coverage issues are related but distinct. Crawl errors occur when Google cannot access a URL — the page is unreachable. Index coverage issues occur when Google can access a page but decides not to index it — the page is reachable but not included in the index.
A page can be crawled successfully (no crawl error) but still not be indexed (coverage issue) because Google decided the content was too thin or duplicated. Both types of issues need attention, but crawl errors are more urgent because they completely block access. For a deeper dive into both topics, see our guides on technical SEO issues and index coverage.
Related resources
Frequently asked questions
Are all 404 errors bad for SEO?
No. If a page was intentionally deleted and no equivalent exists, a 404 is the correct response. Google will eventually stop trying to crawl it. You should only fix 404s for pages you want indexed — redirect them to relevant alternatives or restore the content. Having some 404s is normal and does not harm your site overall.
How long does it take Google to re-crawl a fixed page?
After you request re-crawling via URL Inspection, Google typically re-crawls within a few days. However, it can take longer for the fix to be reflected in search results — Google needs to re-crawl, re-index, and update its serving infrastructure. Expect 1–2 weeks for the fix to fully take effect.
Can crawl errors affect my entire site?
Server errors (5xx) and DNS errors can affect your entire site if your server is down or DNS is broken. These are urgent because they prevent Google from crawling any pages. 404s and redirect errors typically affect individual pages. If you see a sudden spike in crawl errors across many pages, check your server status immediately.
Should I block pages with robots.txt to prevent crawl errors?
Only block pages you genuinely do not want Google to crawl — admin pages, internal search, duplicate content. Do not block pages to "fix" crawl errors; that just hides the problem. If a page should be indexed, fix the underlying issue rather than blocking it.