Add a page to AskRobots

Missing a page? Type its address. Our crawler reads it right away, and once it passes the same checks as every page, it is searchable in seconds.

One address at a time. A sitemap for a whole site goes in the tools for your site.

What happens

We check the address, then read the page the way our crawler always does: as AskRobotsBot, following your robots.txt, and never faster than our usual pace for your site. A page that passes the same quality checks as every page we read goes into the index, and the status page shows the moment it is searchable.

After that it is part of our normal crawl: we read it again now and then to keep it fresh.

What it doesn't do

Adding a page gets it seen; it never moves it up in results. Pages are ranked the same way whether someone added them or our crawler found them.

We don't read pages that robots.txt keeps us out of, sites that answer with a bot challenge, private or local addresses, or sites on our adult, malware or phishing lists.

Run the site?

The tools for site owners test your robots.txt, show what we know about a page, and, once you prove the site is yours, read pages again, take a sitemap and show your site's report. For scripts, the Search API has the same thing: POST /api/v1/submit.