# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file User-agent: Yahoo! Slurp User-agent: Googlebot User-agent: msnbot Allow: /browse/* Disallow: /share Disallow: /search Disallow: /*? Disallow: /*/Web-Search User-agent: msnbot-media Crawl-delay: 1 Allow: /browse/* Disallow: User-agent: Adsbot-Google User-agent: Yahoo!-AdCrawler User-agent: Mediapartners-Google User-agent: Teoma Crawl-delay: 10 Allow: /browse/* Disallow: User-agent: * Crawl-delay: 10000 Allow: /browse/* Disallow: /Health Disallow: /health Disallow: /Autos Disallow: /autos Disallow: /Travel Disallow: /travel Disallow: /search Disallow: /*?