37

I am creating two pages on my site that are very similar but serve different purposes. One is to thank users for leaving a comment and the other is to encourage users to subscribe.

I don't want the duplicate content but I do want the pages to be available. Can I set the sitemap to hide one? Would I do this in the robots.txt file?

The disallow looks like this:

Disallow: /wp-admin

How would I customize to the a specific page like:

http://sweatingthebigstuff.com/thank-you-for-commenting

3 Answers 3

58
Disallow: /thank-you-for-commenting$

in robots.txt

Take a look at last.fm robots.txt file for inspiration.

The dollar sign marks an end. We need that so other pages starting with the same url, for example /thank-you-for-commenting-another-page, will be indexed.

3
  • 16
    I'm relieved someone had the foresight to disallow harm to humans. Apr 20, 2011 at 2:12
  • 1
    Surely this will suffer false positives, because the standard says "any URL that starts with this value will not be retrieved".
    – ChrisJJ
    Sep 29, 2016 at 23:19
  • edited this answer so a $ sign and explanation is included
    – Wim Feijen
    May 25 at 11:32
13

robots.txt files use regular expressions to match pages, so to avoid targeting more pages than you intend, you may need to add a $ to the end of the page name:

Disallow: /thank-you-for-commenting$

If you don't you'll also disallow page /thank-you-for-commenting-on-this-too

0
9

You can also add a specific page with extension in robots.txt file. In case of testing, you can specify the test page path to disallow robots from crawling.

For examples:

 Disallow: /index_test.php
 Disallow: /products/test_product.html
 Disallow: /products/     

The first one Disallow: /index_test.php will disallow bots from crawling the test page in root folder.

Second Disallow: /products/test_product.html will disallow test_product.html under the folder 'products'.

Finally the last example Disallow: /products/ will disallow the whole folder from crawling.

Not the answer you're looking for? Browse other questions tagged or ask your own question.