How to Tell Crawlability Problems from Indexing Decisions

Summary

• Keep the questions separate. Crawling concerns access to a URL; indexing concerns whether its content is eligible to appear in search results. A URL can be indexed even when a current crawl is blocked.
• Treat reports as evidence with limits. For a property you control, use Search Console URL Inspection to review a URL’s reported status. A site: search can offer an indication, but it does not confirm complete index coverage or current crawl access.
• Follow the evidence in order. Trace internal links and sitemap presence, review the matching rule in the root robots.txt, and then inspect the page’s HTML for an unintended noindex directive. Verify each change where it was made; don’t assume it guarantees indexing or a recrawl.

The diagnostic boundary between crawl access and indexability

Crawlability describes whether a search engine can access a URL and explore its content. Indexability concerns whether a page can be included in the search index. They relate to different stages of discovery, and they are not interchangeable. The Google overview of crawling and indexing treats crawling and indexing as distinct topics for that reason.

That distinction matters when a report says a page is “indexed but not crawlable.” The wording can sound contradictory, but it describes two different things: a URL may already be in the index while a current crawl is restricted. For example, a robots.txt rule can block crawling without, by itself, guaranteeing that the URL will be excluded from the index. A page’s prior index status and its present crawl access need separate checks.

So treat index status as an outcome to investigate, not as proof that a search engine can access the page now. The reverse is also true: being able to access a page does not establish that it is indexed or eligible to appear in search results.

A useful diagnosis records both findings separately:

• Crawl access: Can the search engine reach and crawl the URL under the current site rules?
• Indexability: Does the page have a signal, such as an unintended noindex directive, that would prevent it from being considered for indexing?

Avoid changing crawl settings just because a URL is absent from search results. First identify whether the issue is discovery, access, a page-level signal, or only uncertainty about the reported status. Each calls for a different next step.

Establish what the URL’s reported status actually means

Start with the exact URL and the exact wording of the report; don’t diagnose from a vague impression that a page is “missing.” Record the full address you’re investigating, including its path, and note what the report says about it. That gives you a specific item to compare against the site’s crawl and page-level signals.

If you control the property, use Google Search Console’s URL Inspection for that URL and record the status it reports. URL Inspection is limited to properties the operator controls. It can’t be used to verify the status of someone else’s site or a page outside the property.

A site: search is a different kind of evidence. It can indicate that a URL or related content appears in search results, but the results are samples, not rigorous proof of full index coverage. A result also doesn’t demonstrate that the URL is crawlable now. Don’t treat a site: query as a substitute for the report in URL Inspection or for checking access rules.

Keep the reported status separate from verified checks. For example, your notes might say:

• Reported: Search Console URL Inspection gives a particular status for the exact URL.
• Not yet verified: Whether an internal link leads to the URL, whether it appears in the sitemap, whether robots.txt blocks its path, and whether the page contains a noindex directive.

That distinction prevents a report label from becoming an assumed cause. A status tells you what the tool reports; the follow-up checks help explain why. If the tool reports an issue but you can’t identify a matching restriction or page-level signal, record that uncertainty rather than guessing at a cause.

Trace discovery and crawl access before judging indexability

Before deciding that an indexing problem is caused by a page-level signal, check whether the URL has a path through the site and whether crawl rules allow access. Discovery and access are related but separate: an accessible page may be difficult to discover if there is no clear internal path, while a page that can be found may still be restricted from crawling.

Start with the site’s internal links. Look for a link from a relevant page to the exact URL under investigation. If you can’t find one, record that the discovery path is unclear and identify where a suitable internal link would belong. Don’t infer from sitemap presence alone that the page is accessible or indexed.

Then check the site’s XML sitemap and look for that exact URL. Record whether it appears. If it does not, note the omission as a discovery signal to investigate; if it does, record that fact without treating it as proof of crawl access or index status. The check answers a narrow question: whether the URL is listed there.

Next, open the site’s root robots.txt and compare its rules with the URL path. Determine whether a rule matches the path and whether that restriction is intentional. For example, a rule aimed at a broad directory could affect a page within that directory even if nobody meant to block that individual page. The task is to identify the relevant rule and its purpose, not to assume that any restriction is an error.

If a matching block is unintended, correct the rule at its source and verify the resulting root robots.txt to confirm that the URL path is no longer matched. If the block is intentional, keep it and continue the diagnosis without treating the URL’s index status as proof that the rule is harmless. A change to robots.txt does not guarantee an immediate recrawl or a particular indexing outcome.

Read robots.txt as a crawl control, not an index verdict

A rule in robots.txt restricts crawling; it does not, by itself, guarantee that a URL is excluded from the index. The distinction is easy to miss because blocking access may affect what a crawler can retrieve, but index status and crawl access still need separate assessment.

The admitted guidance differs in how broadly it describes the indexing effect of a block. Lumar’s crawlability article explains that a URL blocked by robots.txt can still be indexed if its address is known, for example through an external link. By contrast, BetterLinks’ crawlability guide describes a crawl block as often preventing a page from being indexed. These positions should not be flattened into a rule that a block always removes a URL from the index or always leaves it there.

The bounded conclusion is more useful: a matching rule can restrict crawling, but it does not settle the index question. Check the rule and its purpose before changing anything. If the restriction is deliberate, don’t remove it solely because a report uses the phrase “indexed but not crawlable.” If it is unintended, correct the rule and verify what the file now says about the URL’s path.

There’s an additional reason to avoid treating the report wording as a complete diagnosis. A user-reported Google Search Console thread about “indexed but not crawlable” describes that notification and uncertainty about what caused it. It illustrates the wording people may encounter, but it does not establish the cause for another site.

After any edit, verify the resulting rule in the root robots.txt. That confirms the file’s current crawl instruction. It does not prove the URL has been recrawled, removed from the index, or added to it. Keep those outcomes distinct in your notes.

Inspect page-level indexability signals after access is addressed

Once you’ve assessed discovery and crawl access, check the page’s HTML for a noindex directive. This is a page-level indexability signal, not the same thing as a robots.txt crawl restriction. A page can be crawlable yet have a noindex directive, so a clean access check does not answer every indexing question.

Inspect the HTML output for the exact URL and determine whether a noindex directive is present. If it is, decide whether that instruction is intentional. A deliberate directive may match the page’s purpose; an accidental one may have been introduced at the page or template source. Don’t remove a directive just because the URL is absent from results: first establish that the instruction is unintended.

If the noindex directive is unintended, correct it in the page or template source that produces the page. Then inspect the updated page output and confirm that the directive is no longer present. This verifies the change that was actually made. It does not guarantee that the URL will be indexed, and it doesn’t establish when a search engine will next process the page.

If no noindex directive appears, record that as one checked signal, not as proof that every indexability question is resolved. Return to the reported status and the other evidence you’ve collected. The page may still have a discovery or access issue, or the available checks may not identify the reason for its reported status.

Keep your findings in two columns, even if only in a note: crawl access and indexability. Under access, record the internal path and the matching robots.txt rule, if any. Under indexability, record whether the page HTML contains noindex and what URL Inspection reports for a property you control. This prevents an access change from being mistaken for proof of an index outcome.

Apply the distinction to a hypothetical URL

Assume the URL is https://example.com/guides/indexing-check/, the operator controls the relevant Search Console property, and URL Inspection reports “indexed but not crawlable.” These are hypothetical inputs, not findings from a test. The first step is to preserve the report wording and check the URL’s actual path against the root robots.txt.

Now assume the file contains a rule matching /guides/indexing-check/, and that the block is intentional. In that case, don’t remove it solely to address the report. Record that crawl access is restricted by an intentional rule, then separately review the available indexability evidence. The status does not show that the block is irrelevant, nor does it prove that the page should be crawlable.

If the matching block is unintended, adjust the rule at the source where the root robots.txt is maintained. Then open the resulting file and verify that its rules no longer match /guides/indexing-check/. After that, revisit URL Inspection for the same controlled property and exact URL to assess the reported status again. Record what the report says; don’t assume the edit guarantees a recrawl, indexing, or a change in ranking.

If no matching robots.txt rule exists, don’t invent one as the cause. Check whether an internal link leads to the exact URL and whether it appears in the XML sitemap. Then inspect the page HTML for noindex. Those checks help separate a discovery question from a page-level indexability question, but none should be presented as proof of a result they do not measure.

The point of the example is the decision sequence, not a promised outcome: identify the report, verify the relevant rule, make only a justified change, and check the evidence again.

Close the diagnosis with a verified, conditional next step

End with one of three findings: the discovery path is unclear, crawl access is restricted, or access has been checked but indexability remains in question. These are practical summaries of the evidence you have, not guarantees about what a search engine will do next.

If the discovery path is unclear, record whether an internal link leads to the URL and whether it appears in the XML sitemap. If crawl access is restricted, record the matching rule in the root robots.txt, whether the block is intentional, and any change made. If access has been checked but the indexability question remains, record whether the HTML contains noindex and what URL Inspection reports for a property you control.

After a change, recheck the specific evidence that the change was meant to affect: the resulting robots.txt rule or the updated page HTML. Then review the reported URL status again when that evidence is available. Don’t substitute a site: result for a full index-status confirmation, and don’t describe a verified file edit as proof of indexing.

This approach keeps the diagnosis honest and actionable: identify the signal, change only what the evidence supports, and state clearly what remains unknown.

Article updated on October 5, 2026.