User-agent: Mediapartners-Google* Disallow: # See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file User-agent: * Crawl-delay: 10 Disallow: /json* Disallow: /js* Disallow: /tags* Disallow: /bookmarklet*