# See http://www.robotstxt.org/wc/faq.html#robotstxt # # User-agent: Slurp # Disallow: / #Crawl-delay: 10