6

I have used robots.txt to restrict one of the folders in my site. The folder consists of the sites in under construction. Google has indexed all those sites which are in testing phase. So I used robots.txt. I first submitted the site and robots.txt is enabled. Now the status is success for the www.mysite.com/robots.txt. But the google is still listing those test links. Here is the code i have written for robots.txt...

User-agent: *
Disallow: /foldername/

Can anyone suggest me what the problem may be? Thanks in advance

2
  • 2
    Could it be that Google cached your pages before you introduced the robots.txt
    – Opal
    Commented Sep 8, 2010 at 3:57
  • 1
    Thanks Lee, what can i do then ?
    – user75472
    Commented Sep 8, 2010 at 4:10

2 Answers 2

3

Did you test your robots.txt following Googles instructions? http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

To test a site's robots.txt file:
  • On the Webmaster Tools Home page, click the site you want.
  • Under Site configuration, click Crawler access
  • If it's not already selected, click the Test robots.txt tab.
  • Copy the content of your robots.txt file, and paste it into the first box.
  • In the URLs box, list the site to test against.
  • In the User-agents list, select the user-agents you want.


Webmaster tools: https://www.google.com/webmasters/tools/home?hl=en

2
  • Thanks Mondain......... I think robots.txt is downloaded successfully. On Test robots.txt tab, there are three columns like robots.txt file, Downloaded and Status and for each there is a message www.mysite.com/robots.txt, 17 hours ago, 200(success) respectively. I think crawl is done
    – user75472
    Commented Sep 8, 2010 at 4:09
  • 1
    Webmaster Tools are part of Google Search Console now. From this support question one can access the robots.txt tester: support.google.com/webmasters/answer/6062598?hl=en
    – Developer
    Commented Nov 24, 2022 at 9:19
3

See Requesting Removal of Content from our Index from the Google Webmaster Blog. You can expedite the removal process by submitting a removal request using the Google Webmaster Tools; otherwise, the pages will eventually be dropped out of the index when it is recrawled (i.e. updating a robots.txt file does not have an immediate change; the change takes place on subsequent crawls).

2
  • Thank you Michael, I used this too. My remove url is www.mysite.com/foldername/. The status is removed. But still not working.
    – user75472
    Commented Sep 8, 2010 at 4:12
  • 1
    I would wait just a bit... there may be some lag before it takes effect. Commented Sep 8, 2010 at 4:21

Not the answer you're looking for? Browse other questions tagged or ask your own question.