Robots Crawl-delay

Syntax: decimal number

This gives the minimum and maximum robots.txt Crawl-delay values to allow; values found outside this range will be changed to the appropriate minimum or maximum. -1 means no limit. The defaults are 0 and 10. These values can be used to set reasonable bounds to sites' Crawl-delay values.

Note that a Crawl-delay seen (modified to these limits) is only used if Robots robots.txt is Y, and overrides Walk Delay (here). Thus, to use the greater of robots.txt Crawl-delay or Walk Delay (e.g. if the walker is bandwidth-limited, and sites' robots.txt delays are to be followed), keep Min equal to Walk Delay. To always use Walk Delay but still respect other robots.txt directives (i.e. just ignore Crawl-delay), keep Min and Max equal to Walk Delay.