Host Aliases from robots.txt

Syntax: select Yes or No button

If Yes, then the walk process will look for Host: directives in `robots.txt`. If one is found, and the host listed there is different than the host it was launched on, it will save that pair as a new Host Alias (see above), and convert the content.

For example, if your Base URL is, and the contents of ` contains the line


Then a Host Alias will automatically be created, from the alias host to the canonical host, and then will then be indexed.

Copyright © Thunderstone Software     Last updated: Apr 15 2024
Copyright © 2024 Thunderstone Software LLC. All rights reserved.