Issue library · 8 crawl and server checks
Connect rendered-page advice to the responses and discovery paths a crawler actually encounters.
A technical crawl is a bounded request-based review of how important URLs are discovered, fetched, interpreted, and linked. Prioritize failed responses and broken internal paths before lower-impact metadata refinements.
A strong page cannot perform as intended when crawlers receive an error, follow unnecessary redirects, encounter an accidental exclusion, or cannot reach the page through useful internal links.
The crawler could not retrieve the URL successfully or received a 4XX/5XX response.
Restore the intended page, repair the destination, or remove and replace references to a URL that should no longer exist.
Request the exact URL again and confirm the intended successful response without an unexpected redirect.
The request passed through more than one redirect before reaching its final destination.
Update internal links, sitemap entries, and canonicals to the final preferred URL, then remove unnecessary intermediate hops.
Trace the URL again and confirm one deliberate redirect at most.
The applicable robots.txt rule prevents StrictSEO's crawler from fetching the URL.
Keep the restriction when deliberate; otherwise revise the narrow rule without opening unrelated private paths.
Test the exact URL against the deployed robots.txt rules and the intended crawler user agent.
The URL was declared in a sitemap but no reviewed HTML page linked to it.
Add a useful contextual or navigational path, or remove the sitemap entry if the page should not be independently discovered.
Repeat the crawl and confirm at least one appropriate source page links to the URL.
One or more crawled pages link to a URL that failed or returned an error response.
Repair the destination or update every source link to the correct working page.
Re-crawl the affected sources and confirm the destination succeeds.
The X-Robots-Tag response header asks compliant search engines not to index the resource.
Retain the directive only for deliberate exclusions and align it with HTML, sitemap, and canonical signals.
Inspect the final response headers and confirm the directive matches the intended indexing policy.
The document response exceeded StrictSEO's local review threshold during this crawl.
Repeat the test before prioritizing work, then inspect redirects, origin processing, caching, CDN routing, and payload delivery.
Compare several controlled requests and field evidence when available.
The local crawler stopped reading a document after its bounded safety limit.
Reduce excessive document markup or inspect the response using a specialized transfer and payload tool.
Re-run after reducing the response and confirm complete analysis occurs under the limit.