Host Aliases from robots.txt

Syntax: select Yes or No button

If Yes, then the walk process will look for Host: newsite.com directives in `robots.txt`. If one is found, and the host listed there is different than the host it was launched on, it will save that pair as a new Host Alias (see above), and convert the content.

For example, if your Base URL is https://www.oldsite.com, and the contents of `https://www.oldsite.com/robots.txt contains the line

Host: www.newsite.com

Then a Host Alias will automatically be created, from the alias host www.oldsite.com to the canonical host www.newsite.com, and then https://www.newsite.com will then be indexed.