WHM "Bandwidth" Settings

[Frank Hagan]

I have a trouble ticket in for a weird problem I'm having with an unidentified bot following the Amazon.com store links on two of my sites, and consuming a bunch of bandwidth (it doesn't look like it obeys "robots.txt", but we'll see).

But in adjusting the bandwidth on the affected accounts, I noticed something weird. In WHM with the "X" skin, when you edit a package, it asks for the bandwidth limit in "MegaBytes". If you put 2000 in there, you would expect the account to have just under 2 gigs of bandwidth available, but it doesn't ... you have to put in 20,000 "MegaBytes" to get the response: Bandwidth limit (20971520000) ... so I think that's really a "bytes" measurement in WHM and not a "MegaBytes" measurement. Has anyone else seen this?

[AndrewT]

It certainly is in megabytes, not bytes.

20,971,520,000 bytes = ~20GB

Edit: I also did just check one of your domains which you referenced in the ticket and it is in fact assigned ~20GB. Since it is using nearly 18GB, a 2GB limit certainly would not suffice.

[Frank Hagan]

You're right ... it is correct. Its a ton of bandwidth compared to what I usually have. I'll have to check to see if the new robots.txt is being obeyed by the new "experimental" bot that was indexing my forums.

[Frank Hagan]

These guys are killing me!

Unknown robot (identified by 'robot') 163669+23 6.45 GB 21 May 2007 - 18:10

And that's after excluding the php script portions of my site. That's 6.45 GB today alone. I edited a robots.txt file to have just two lines in it ...

User-agent: *
Disallow: /

I'm hoping that will work. Otherwise, how do you use a .htaccess directive to ban a range of IP addresses?

[Elite]

These guys are killing me!



And that's after excluding the php script portions of my site. That's 6.45 GB today alone. I edited a robots.txt file to have just two lines in it ...

User-agent: *
Disallow: /

I'm hoping that will work. Otherwise, how do you use a .htaccess directive to ban a range of IP addresses?

I feel for you

for one:

deny from 123.456.78.90

and a range

deny from 123.456.78

[Frank Hagan]

Thanks!

I actually found there's an "IP Deny Manager" in Cpanel that writes the value out, so I was able to block the range of IP addresses the bot is coming from ... I probably blocked too much and should narrow it down a bit, but its last visit was before the IP blocking, so it looks like it is working, at least.

I sent the bot owner an email, but haven't heard back from him.

Google bot also visited the site again, and I'm hoping it is obeying the robots.txt file; I'll have to look at the stats again tomorrow and see if they increment any.

I think robots.txt needs to be enhanced, so that bots will know which pages to crawl, and where to stop crawling. You can put entire folders out of reach (if the bot obeys), but not all pages after the first page in a folder (as an example).

[Snowbaby]

How do you see what uses up what amount of BW? I'd be quite interested as my site gets a lot of bots visiting constantly.

[AndrewT]

You can see this via cPanel or under "View Bandwidth Usage" in WHM.

[Buddha]

How do you see what uses up what amount of BW? I'd be quite interested as my site gets a lot of bots visiting constantly.

Log files. Awstat.

[Frank Hagan]

How do you see what uses up what amount of BW? I'd be quite interested as my site gets a lot of bots visiting constantly.

AWStats will tell you the name of the bot, if it identifies itself, under the heading "Robots / Spiders". To see the IP addresses, you have to look at the "Access Logs", which you can download from Cpanel under "Raw Access Logs".

It downloads the log in the unix-style ".gz" archive format which some archivers support; a free one that does is 7-Zip at http://www.7-zip.org/. You may have to rename the log file within the archive, which 7-Zip allows. The log uses your domain name and if it ends in ".COM", your virus protection will go crazy trying to stop you from extracting it. Rename it with a .TXT and open it in a text editor.

Search for "robots.txt" and you'll see the visitors that first asked for the file before continuing on to surf your site. Their IP address is given.

I think there's a Windows-based log viewer out there; I should find out because its a pain to sort through the raw logs.

[bjacobs]

Here's a windows based Log Viewer