Fetch and analyze any website's robots.txt file. Test whether specific paths are allowed or blocked for Googlebot and other search engine crawlers.
The robots.txt file tells search engine crawlers which pages they can and can't access on your site. It's located at /robots.txt on every domain. While it's not a security measure, it's important for controlling how crawlers index your site.