Test robots.txt

Type an address on your site: we say whether our crawler may read it, using your robots.txt as the crawler last read it and the crawler's own rules.

How to change what it reads

Use the token AskRobotsBot after User-agent:; rules for * apply to it too. Examples: block it, slow it, keep it out of some paths.