Respect Canonical URLs

Syntax: select Yes or No button

This option enables respecting when pages indicate that there's a canonical version of themselves, expressed with <link rel="canonical">.

If Y (the default), then if a page has a <link rel="canonical"> that is different from the fetched URL, the original page will NOT be indexed and the canonical URL will instead be added to the index.

Note that if the canonical URL is not be allowed due to walking rules (matches exclusions, off-site, etc), then the canonical behavior does not apply and the original, non-canonical URL will be indexed.

If Placeholder=Y is set, a placeholder of the non-canonical URL is saved. This helps efficiency with walks that have cross-site links.