# robots.txt # Robots Exclusion Protocol # Convention to prevent cooperating web spiders and other web robots from accessing # all or part of a website which is otherwise publicly viewable # = comment to be ignored User-agent: * Disallow: