Technical SEO starts with a simple chain: a crawler must discover a URL, fetch it successfully, understand which version is canonical and be allowed to index it. A beautiful page cannot rank if that chain is broken.
Discovery is not the same as indexing
A search engine can know a URL exists without indexing it. Sitemaps and links help discovery, while content quality and technical signals influence indexing decisions.
A URL can also be indexed without appearing for valuable queries. Indexing is only the prerequisite, not the final goal.
Check status at the URL level when diagnosing a specific page instead of assuming a sitewide issue.
Robots rules and noindex do different jobs
robots.txt controls crawling instructions; a noindex directive tells a search engine not to keep the page in its search index after it can process that directive.
Blocking a URL from crawling can prevent a crawler from seeing page-level instructions. Use each mechanism for the job it is designed to do.
Test staging rules carefully before launch. Accidentally carrying noindex or blocking directives into production is a common avoidable mistake.
Canonicals should match your preferred URL strategy
Use canonical tags to identify the preferred version when several URLs contain substantially the same content.
Keep signals consistent: internal links, sitemap URLs and redirects should generally support the same canonical destination.
Do not use canonical tags as a universal fix for every low-quality or duplicated page. Sometimes the correct solution is a redirect, removal or better site architecture.
XML sitemaps should be clean
Include canonical, indexable URLs that return successful responses and deserve discovery.
Remove old redirects, 404s, parameter duplicates and pages intentionally excluded from search.
Split very large or specialized sitemaps when it helps monitoring, but do not create complexity without a diagnostic benefit.
HTTP status codes tell crawlers what happened
Use successful responses for live pages, redirects for moved content and appropriate error responses for removed URLs.
Avoid soft 404s where a missing page returns a normal success code with an error message.
Keep redirect chains short and update internal links to the final destination after migrations.
Frequently asked questions
Does submitting a sitemap guarantee indexing?
No. A sitemap helps discovery and communicates preferred URLs, but search engines still decide what to crawl and index.
Should every page be indexable?
No. Utility, duplicate, account and low-value pages may not belong in search. The goal is to index the right pages, not the maximum number of pages.