9

I have one problem

I have domain name for example www.testing.com and new.testing.com so i do not want to new.testing.com display in any search engine. I have added one robots.txt to the new.testing.com. And both site has same parent directory

--httpdoc
----testing.com
----new.testing.com

So i want to know that can i handle both site using one robots.txt of testing.com is it possible???

Please suggest me solution if possible.

3 Answers 3

11

The best thing you could do is to add separate robots.txt files. Put one in each directory. You should have

  • testing.com/robots.txt and
  • new.testing.com/robots.txt

After adding robots.txt file to new.testing.com, you should add following code to keep search engines away.

User-agent: *
Disallow: /
1
  • 2
    if domain and subdomain same directory how can disallow only subdomain? disallow: /new.domain.com/ that true? or have any another idea? please help...
    – gokhan
    Dec 17, 2014 at 17:22
6
RewriteEngine on
RewriteCond %{HTTP_HOST} ^subdomain.website.com$
RewriteRule ^robots\.txt$ robots-subdomain.txt

Then add the following to /robots-subdomain.txt:

User-agent: *
Disallow: / 
1
  • 1
    Exactly what i was looking for. Thanks!
    – ANDiTKO
    Mar 1, 2017 at 11:01
0

Following rules inside works for me,

RewriteEngine On
RewriteCond %{HTTP_HOST} ^subdomain\.maindomain\.com$ [NC]
RewriteRule ^/robots.txt$ /nobots.txt [L]

And add 'nobots.txt' to root directory as follows,

User-agent: *
Disallow: / 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.