# www.robotstxt.org/ # Allow crawling of all content User-agent: * Disallow: /cron Disallow: /pages Disallow: /search Disallow: /*/add-review Disallow: /*/directions Disallow: /*/edit