95

My robots.txt in Google Webmaster Tools shows the following values:

User-agent: *
Allow: /

What does it mean? I don't have enough knowledge about it, so looking for your help. I want to allow all robots to crawl my website, is this the right configuration?

8

I understand that this is fairly old question and has some pretty good answers. But, here is my two cents for the sake of completeness.

As per the official documentation, there are four ways, you can allow complete access for robots to access your site.

Clean :

Specify a global matcher with a disallow segment as mentioned by @unor. So your /robot.txt looks like this.

User-agent: *
Disallow:

The hack :

Create a /robot.txt file with no content in it. Which will default to allow all for all type of Bots.

I don't care way :

Do not create a /robot.txt altogether. Which should yield the exact same results as the above two.

The ugly :

From the robots documentation for meta tags, You can use the following meta tag on all your pages on your site to let the Bots know that these pages are not supposed to be indexed.

<META NAME="ROBOTS" CONTENT="NOINDEX">

In order for this to be applied to your entire site, You will have to add this meta tag for all of your pages. And this tag should strictly be placed under your HEAD tag of the page. More about this meta tag here.

37

If you want to allow every bot to crawl everything, this is the best way to specify it in your robots.txt:

User-agent: *
Disallow:

Note that the Disallow field has an empty value, which means according to the specification:

Any empty value, indicates that all URLs can be retrieved.


Your way (with Allow: / instead of Disallow:) works, too, but Allow is not part of the original robots.txt specification, so it’s not supported by all bots (many popular ones support it, though, like the Googlebot). That said, unrecognized fields have to be ignored, and for bots that don’t recognize Allow, the result would be the same in this case anyway: if nothing is forbidden to be crawled (with Disallow), everything is allowed to be crawled.
However, formally (per the original spec) it’s an invalid record, because at least one Disallow field is required:

At least one Disallow field needs to be present in a record.

7

It means you allow every (*) user-agent/crawler to access the root (/) of your site. You're okay.

133

That file will allow all crawlers access

User-agent: *
Allow: /

This basically allows all user agents (the *) to all parts of the site (the /).

  • 7
    Correct, unless you need to negate the allow part. There is not "allow" so make that: "User-agent: * Disallow:" like they show here: robotstxt.org/robotstxt.html – vsdev Jan 8 '15 at 13:46

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.