Build an inventory of pages that should exist
Start with a list of intended public URLs. Group them by page type: homepage, category, article, product, policy or another meaningful function. Identify the pages that serve a real reader need and the pages that should remain private or excluded from search.
Compare that list with the pages actually reachable through navigation and internal links. A valuable page that exists only in a sitemap may be difficult for readers to discover. A site with several near-identical routes may be sending inconsistent signals about the preferred version.
For each page, record its purpose, title, canonical URL, status code and main internal entry points. This is a practical inventory, not a reason to create extra pages for every wording variation of a query.
Google’s search-process documentation separates crawling, indexing and serving results. A problem at one stage needs a different diagnosis from a problem at another.
Check HTTP status and direct access
Important pages should load at their intended address without a chain of unnecessary redirects. A missing page should return a genuine not-found response rather than a successful status with a generic error message.
| Check | What a good result looks like | Priority |
|---|---|---|
| Main public URL | A successful response with the expected page. | Critical if broken. |
| Old URL with a replacement | A direct appropriate permanent redirect. | High for valuable old links. |
| Unknown URL | A useful error page with a real 404 status. | High if soft errors are widespread. |
| HTTP and alternate hostnames | A consistent route to the chosen HTTPS hostname. | High for consistency and security. |
| Important files | Styles, scripts and images load correctly. | High when rendering or use is affected. |
Check direct entry to deeper pages, not only navigation from the homepage. A site can work when clicked through while failing on a bookmarked route because of server configuration.
When retiring content, decide whether a relevant replacement exists. Redirect to that replacement when appropriate; do not send every removed page to the homepage merely to avoid a 404.
Separate crawl controls from index controls
A robots.txt rule controls crawling for compliant crawlers; it is not a privacy boundary or a dependable way to remove a URL from search. A blocked URL can still be discovered through links. Google’s robots.txt introduction explains that distinction.
Use appropriate indexing directives when a public page should not appear in search, and protect private information with actual access control. Do not place confidential files at a public URL and assume a crawler instruction makes them private.
Watch for staging settings left behind at launch. A site-wide noindex directive, a disallow-all robots file or authentication on the production site can prevent intended discovery. Conversely, an unprotected staging copy can create duplicate public content.
Check headers as well as HTML. Indexing instructions can be delivered through response headers, and a proxy or hosting configuration may add behavior that is not visible in the source template.
Make canonical signals consistent
Choose the preferred public address for each page. Use it consistently in internal links, canonical annotations and the sitemap. Canonical signals help describe the preferred version; they are not a guarantee that a search engine will choose it.
Common duplicates include HTTP and HTTPS, www and non-www, trailing-slash variations, explicit index filenames and tracking parameters. Handle these deliberately instead of linking to whichever form happens to appear first.
Do not point every article’s canonical to the homepage or use canonical tags to merge pages with different reader purposes. A canonical should describe a true preferred equivalent, not compensate for an unclear content strategy.
Google’s canonical documentation describes the available signals. After implementation, inspect representative pages and check that the preferred URL is actually reachable and indexable.
Check titles, headings and internal links in context
Write a title that identifies the page’s actual subject and a description that accurately previews its value. Avoid repeating a generic title across every article. A clean heading hierarchy helps readers scan; headings should organize the content rather than decorate it with keywords.
Use descriptive links to related decisions and explanations. A hosting review can link to a cost calculator, a direct comparison and a migration checklist because those support the reader’s next step. Links added only to repeat a phrase create little value.
Make sure essential content and links exist in the delivered or correctly rendered page. If the site relies on JavaScript, verify what happens when data loading fails and how the crawler can reach the content. A loading placeholder is not a useful article.
Inspect the mobile navigation and long tables. Technical accessibility and reader accessibility often meet in the same defects: hidden links, clipped text or controls that cannot be reached by keyboard.
Use structured data to describe reality
Structured data should match the visible page and the supported content type. An article can identify its headline, author and publication date. Breadcrumb markup can describe the actual navigation hierarchy.
Do not invent aggregate ratings, customer reviews, prices or test results to make markup look richer. If a page does not display a factual offer or a genuine review set, the structured data should not pretend otherwise.
Validate the syntax and the relevant feature requirements. Google’s structured-data introduction makes clear that valid markup does not guarantee a particular search appearance.
Keep dates meaningful. Updating a date without changing the substance of an article does not make the information current. If product terms were checked, record what was checked and preserve the evidence.
Prioritize fixes and measure the right outcome
Resolve an accidental site-wide block, broken checkout or widespread server error first. Next address important missing pages, redirect problems and unusable mobile content. Treat small metadata refinements as lower priority than defects that prevent reading or indexing.
Use Search Console and server information to investigate patterns, while recognizing that reporting can lag. Separate a technical failure from a page that is indexed but does not yet meet a competitive search intent.
For performance, use the Core Web Vitals guide. For content, ask whether the page gives a useful, original answer and supports its claims. Technical correctness creates a foundation; it does not guarantee traffic or first-place rankings.
Frequently asked questions
Does a sitemap guarantee indexing?
No. It helps search engines discover URLs, but crawling and indexing depend on other factors. Keep the site navigable and the pages useful.
Can robots.txt hide private content?
No. Use authentication or another genuine access control for private information. A crawler instruction does not prevent public access.
Will valid schema guarantee rich results?
No. Structured data must match the page and the relevant guidelines, and eligibility does not guarantee a particular display.
Sources & editorial notes
Sources checked on September 6, 2026. Product details can change by country, platform and billing term. Prices shown are snapshots, not live quotes.
- Google: crawling, indexing and serving
- Google: robots.txt limitations
- Google: canonical signals
- Google: structured-data requirements
- Google: JavaScript SEO
This guide combines published documentation with our editorial analysis. We have not measured provider performance or conducted an independent security audit. Read our methodology. Report a correction.

