# Robots.txt file to control search engine crawlers # All robots will spider the domain User-agent: * Disallow: /_private/ Disallow: /cgi-bin/ Disallow: /mlist/ Disallow: /guestbook/