Robots.txt

The Search Appliance comes with no robots.txt file. If you want to control how web walking robots visit your search pages you can upload a robots.txt file. This robots.txt file will apply to all pages returned bythe Search Appliance including admin, documentation, and searches whether accessed by HTTP or HTTPS. See https://www.robotstxt.org for the syntax of robots.txt files.

If you no longer wish to have a robots.txt check Delete.

Note: You may also control robots for searches on a profile by profile basis by using meta robots within the top HTML of the search settings or the custom XSL if you're using an XSL style sheet for search results. See https://www.robotstxt.org for the syntax of the meta robots tag.