Robot.txt: where is it?

[felipe808]

I've been trying to find this file, robot.txt, and cannot. Please tell me where it is kept and/or delivered to the robots. Thanks in advance. Felipe

[www]

As far as I know,

This file does not exist without you making it. It should be kept in your main "www" or "public_html" folder. Robots look for it there.

[djn]

If you want to make one look at this:

http://www.robotstxt.org/wc/exclusion-admin.html

[Buddha]

robots.txt not robot.txt

[felipe808]

Robots.txt

Thanks for the reply. I cannot find robots.txt anywhere. I just converted the site to TextPattern and it must have it somewhere else (in php?), because now cpanel reports how many times the robots have read the file and that never happened before. Just curious to see what it says.
Thanks

[Elite]

Hmmm - that is odd - On sites where I dont have a robots.txt it is just reported in the 404 requests

Which stats are you using?

[felipe808]

Roger that 404

Yeah, affirmative. Now I have zero 404 errors, because TextPattern also has some sort of automatic re-routing of bad requests to the home page. Now all the cross scripters requesting access to php, java pages go to the home page, which is pretty cool. But, I'll be damned if I can find that robots.txt file.

[djn]

I've just googled a little and found this:

http://forum.textdrive.com/viewtopic.php?id=2658

In plain English, if your visitor requests a non-existent URL (like a bot checking for /robots.txt) he is served the home page or the section index instead of a 404 Not Found header or your custom error page. Same for favicons, I've read...

[felipe808]

Question answered: Thanks

Looks like you found it. Thanks. There is no robots.txt and that's why I cannot find it. AwSTats must be reporting it read because the robots are not receiving a "clean mode 404" as reported in the link you provided. I had searched that site for robots.txt and got zero results. Anyway, thanks for the insight.