100

I would like Google to ignore URLs like this:

http://www.mydomain.example/new-printers?dir=asc&order=price&p=3

In other words, all the URLs that have the parameters dir, order and price should be ignored. How do I do so with robots.txt?

0

3 Answers 3

170

Here's a solutions if you want to disallow query strings:

Disallow: /*?*

or if you want to be more precise on your query string:

Disallow: /*?dir=*&order=*&p=*

You can also add to the robots.txt which url to allow

Allow: /new-printer$

The $ will make sure only the /new-printer will be allowed.

More info:

http://code.google.com/web/controlcrawlindex/docs/robots_txt.html

http://sanzon.wordpress.com/2008/04/29/advanced-usage-of-robotstxt-w-querystrings/

9
  • this will disallow new-printers I only want to disorder the querystring part Feb 5, 2012 at 15:02
  • so you want to allow /new-printer but not /new-printers?dir=*&order=*&p=*?? Feb 5, 2012 at 15:05
  • 1
    Are those advanced wildcards and the allow directive supported well? Jan 15, 2013 at 14:34
  • 11
    According to robotstxt.org/robotstxt.html - "there is no "Allow" field" Apr 22, 2013 at 9:38
  • Taking the new-printers example a bit further, what if different combinations and orders of parameters are possible on that file. Can you specify in a single query that a specific file should be disallowed if any kind of parameters are added to it without explicitly specifying them? Would... Disallow: /new-printer?* work?
    – AdamJones
    Aug 27, 2014 at 21:20
38

You can block those specific query string parameters with the following lines

Disallow: /*?*dir=
Disallow: /*?*order=
Disallow: /*?*p=

So if any URL contains dir=, order=, or p= anywhere in the query string, it will be blocked.

6
  • Does this means that the whole page will not be crawled as long as the above condition is satisfied. Aug 2, 2017 at 8:10
  • 3
    Beware: this will also block parameters which partially match expression, so not only example.com?p=test but also example.com?top=test.
    – rob006
    Dec 1, 2019 at 16:32
  • 3
    If you would like to ignore those parameters regardless their position in the URL (first position or next) you can try that : Disallow: /*?dir=* Disallow: /*?order=* Disallow: /*?p=* Disallow: /*&dir=* Disallow: /*&order=* Disallow: /*&p=* Feb 20, 2020 at 18:21
  • Can the ? be ignored? Nov 4, 2021 at 9:20
  • If I Disallow: /*?*order=, will it also disallow requests that contain reorder=? Jan 21, 2022 at 10:27
0

Register your website with Google WebMaster Tools. There you can tell Google how to deal with your parameters.

Site Configuration -> URL Parameters

You should have the pages that contain those parameters indicate that they should be excluded from indexing via the robots meta tag. e.g.

3
  • 3
    While the original question mentions Google specifically, it's important to note that the Google WebMaster Tools would only block Google. Adding the Disallow rules in the robots.txt file would address other search engines as well.
    – Matt V.
    Jan 14, 2013 at 20:37
  • 1
    True. It should also be clarified that robots.txt does not stop Google indexing pages but stops it reading their content. The best solution is using the robots meta tag on the page itself. This is supported by all systems. Jan 15, 2013 at 14:35
  • 4
    Note that this doesn't work anymore since they removed that functionality, see developers.google.com/search/blog/2022/03/…
    – Joël
    Aug 4, 2022 at 8:15

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.