2

Possible Duplicate:
How to detect fake users ( crawlers ) and cURL

some pages of my Website is getting crawled undesirably .

say pages like

abc.com/smarty/templates/1.html

abc.com/smarty/templates/2.html

abc.com/images/1.jpg

abc.com/images

  • I want to avoid indexing of these page
  • also want to remove these pages from google indexing.

I know to do this using apache setting.But since i am using shared network, i dnt hv access to these.

Please help

4
  • Do you have a robots.txt file and sitemap files in place?
    – Mike Brant
    Dec 11, 2012 at 16:50
  • I really dont know why php-ers put their "core" files into webroot...
    – user823738
    Dec 11, 2012 at 16:51
  • block those page in robots.txt you dont want to Crawling Dec 11, 2012 at 16:51
  • how to make rules in robot.txt , if i want to avoid all pages begining with abc.txt/smarty to not be crawled Dec 11, 2012 at 16:53

3 Answers 3

3

You can use a text file robots.txt, that the search motors find and which tells them what pages they can and can't index in your website.

Here is a good article about how to write this file: Robots.txt: What it is, Why it’s Used and How to Write it


To remove a page from Google was discussed here

3
  • thanx , also how to remove existing indexed pages? Dec 11, 2012 at 16:56
  • And how to remove directory say i dnt want abc.com/aa/ to be indexed but i want abc/aa/a.php to be indexed Dec 11, 2012 at 16:58
  • @PriyaSehpal just use Disallow: aa/* or a* to be more discrete.
    – user823738
    Dec 11, 2012 at 18:02
0

in robots.txt add rules like

User-agent:    *
Disallow:   /smarty/*
Disallow:   /images/*
0
0

www.google.com/webmasters/tools/

Pic of webmaster tools!

You can fetch pages as google You can remove pages from google's index you can upload your sitemap and much more. I think this is exactly what you are looking for.

Not the answer you're looking for? Browse other questions tagged or ask your own question.