Search engine bots causing massive bandwidth usage

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Nitron
    Member
    • Apr 2004
    • 37

    Search engine bots causing massive bandwidth usage

    Various search engine bots, mainly from Google (googlebot) and MSN, are causing massive bandwidth usage on my vBulletin 3, I'm already at 22GB usage this month, when usually I rarely go past 8GB.

    To make a long question short, I have my vB3 in a subdirectory, how would I construct a robots.txt file to deny all search engine bots just for that subdirectory and everything in it?
    Z Force Network
  • Amitabh
    Member
    • Mar 2004
    • 78

    #2
    Create a robots.txt in your root directory with the following contents:
    Code:
    User-agent: *
    Disallow: /<forum-folder-name>/

    Comment

    Working...