Care to share your thoughts about what to put in robotics.txt file to limit spidering? I read someone who had file limiting access to certain folders. What does the spider need to see and what should be kept off limits? Thanks and regards, felipe.
-
-
I have no suggestions but I do know that is robots.txt not robotics.txtDan Blomberg -
"The word genius isn't applicable in football. A genius is a guy like Norman Einstein!" ... Joe TheismanComment
-
I generally tell the spiders to stay away from image directoriesa dn my cache directory.
This stops cached files from being spidered, if they get spidered in the first place, as they don't contain the full layout for my website.
It all depends on what you want the spider to see and what not. Bear in mind though, some "bad" search engines don't pay attention to robots.txt
David-----
Do you fear the obsolescence of the metanarrative apparatus of legitimation?Comment
Comment