I've noticed that one of my accounts is regularly increasing it's monthly bandwidth usage. Is there something that I can use that will allow me to figure out what's causing it? It's my personal domain, so I know that there is nothing new happening as far as uploads/downloads. And I can't think of anything that would attract much attention. I know that I've been getting quite a bit of spam going through my wordpress site, but I don't know why that'd increase the bandwidth so much.
Bandwidth Increasing regularly
Collapse
X
-
Take a look at the "Web/FTP Stats" in Cpanel ... click on that icon, and then click on ""Awstats" ... when Awstats loads, scroll down until you see what is causing the extra bandwidth. Look at the "Robots / Spiders" section to see if the robots from the search engines are consuming too much bandwidth.
If you don't see anything in Awstats that looks to be responsible, submit a trouble ticket for Dathorn to check the bandwidth stats. Not sure of the reason, but I did have a problem with bandwidth when I was on an older server ... it had to do with how Cpanel or WHM was calculating it, IIRC. -
Slurp is Inktomi, I think. The question I'd have to ask is why they are crawling your site so much? Do you have a php store, or link application on it that has thousands of links?
You can limit the folders the robots crawl through the use of a "robots.txt" file you place in the public_html folder. If you have a lot of graphics on the site, Google will index them and eat up a bit of bandwidth. You can disallow all files with a .jpg extension, for instance. Here's an example of one of my "robots.txt" files that disallows graphics from indexing, as well as some folders I certainly don't want to show up on a Google search:
Code:User-agent: Mediapartners-Google* Disallow: /.cgi$ Disallow: /.jpg$ Disallow: /.gif$ Disallow: /.bmp$ Disallow: /images/ Disallow: /cgi-bin/ Disallow: /js/ Disallow: /phpAdsNew/ Disallow: /styles/ User-agent: googlebot Disallow: /.cgi$ Disallow: /.jpg$ Disallow: /.gif$ Disallow: /.bmp$ Disallow: /images/ Disallow: /cgi-bin/ Disallow: /js/ Disallow: /phpAdsNew/ Disallow: /styles/ User-agent: Slurp Disallow: /.cgi$ Disallow: /.jpg$ Disallow: /.gif$ Disallow: /.bmp$ Disallow: /images/ Disallow: /cgi-bin/ Disallow: /js/ Disallow: /phpAdsNew/ Disallow: /styles/ Crawl-delay: 10 User-agent: * Disallow: /.cgi$ Disallow: /.jpg$ Disallow: /.gif$ Disallow: /.bmp$ Disallow: /images/ Disallow: /cgi-bin/ Disallow: /js/ Disallow: /phpAdsNew/ Disallow: /styles/ Crawl-delay: 10
See the Google recommendations at https://www.google.com/webmasters/to.../en/about.html for more information. Another good option is to create a sitemap to Google's specifications, and specifically list the URLs in it ... and tell Google how often they really change. Google will then crawl only the URLs in the sitemap, and will *presumably* honor the "change frequency" you put in it. Details are on the Google page linked above.Comment
-
Bandwidth issues
I am going to post in here first about my bandwidth issue.
I have 2 questions and after reading this thread I might have the googlebot problem solved. Googlebot is eating my bandwidth. I did the robots.txt.
Other question:
Why is my server IP in the list of Top Host in hits?
69.56.171.57 21859 21859 1014.11 MB 17 Mar 2007 - 05:36
1GB for only 17 days.
DwayneComment
Comment