Syntax: select Yes or No button
When enabled (the default), support the Google AJAX crawling scheme. This allows AJAX URLs (which are usually not walkable) to be walked, if the site being walked also supports the scheme.
AJAX URLs which contain anchors/fragments (
not normally walkable because anchors are never sent in HTTP
not include AJAX so the anchor is not processed by the walker either.
Thus AJAX anchor links look like duplicates and are never fetched, or
if fetched, do not return the anchor-specified content.
With the Google AJAX crawling scheme, walkers temporarily rewrite certain links with conforming anchors - those that begin with an exclamation point - by placing the anchor into the query string. Since query strings are sent in HTTP requests, the server sees the anchor, and can return the appropriate content. Moreover, the returned content can be static so that the walker can index it.
Specifically, URLs of the form:
will be requested from the server as (respectively):
This temporary rewrite is only used for the walker fetch: search results still return the original AJAX-anchor version of the link, so that browsers can still take advantage of the AJAX features of the site.
Note that this scheme requires web site support: the site must respond
to any URL with the
_escaped_fragment_ query parameter with the
appropriate, full, static HTML content that corresponds to the AJAX
anchor state of the same value.