Bandwidth Increasing regularly

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Tuxhedoh
    Junior Member
    • May 2004
    • 12

    Bandwidth Increasing regularly

    I've noticed that one of my accounts is regularly increasing it's monthly bandwidth usage. Is there something that I can use that will allow me to figure out what's causing it? It's my personal domain, so I know that there is nothing new happening as far as uploads/downloads. And I can't think of anything that would attract much attention. I know that I've been getting quite a bit of spam going through my wordpress site, but I don't know why that'd increase the bandwidth so much.
  • Frank Hagan
    Senior Member
    • Mar 2004
    • 724

    #2
    Take a look at the "Web/FTP Stats" in Cpanel ... click on that icon, and then click on ""Awstats" ... when Awstats loads, scroll down until you see what is causing the extra bandwidth. Look at the "Robots / Spiders" section to see if the robots from the search engines are consuming too much bandwidth.

    If you don't see anything in Awstats that looks to be responsible, submit a trouble ticket for Dathorn to check the bandwidth stats. Not sure of the reason, but I did have a problem with bandwidth when I was on an older server ... it had to do with how Cpanel or WHM was calculating it, IIRC.

    Comment

    • Tuxhedoh
      Junior Member
      • May 2004
      • 12

      #3
      That was helpful.... It shows 3Gbs reported as- Not viewed traffic includes traffic generated by robots, worms, or replies with special HTTP status codes.

      Googlebot 1.62 GB and Inktomi Slurp 1.02 Gb

      Comment

      • Frank Hagan
        Senior Member
        • Mar 2004
        • 724

        #4
        Slurp is Inktomi, I think. The question I'd have to ask is why they are crawling your site so much? Do you have a php store, or link application on it that has thousands of links?

        You can limit the folders the robots crawl through the use of a "robots.txt" file you place in the public_html folder. If you have a lot of graphics on the site, Google will index them and eat up a bit of bandwidth. You can disallow all files with a .jpg extension, for instance. Here's an example of one of my "robots.txt" files that disallows graphics from indexing, as well as some folders I certainly don't want to show up on a Google search:

        Code:
        User-agent: Mediapartners-Google*
        Disallow: /.cgi$
        Disallow: /.jpg$
        Disallow: /.gif$
        Disallow: /.bmp$
        Disallow: /images/
        Disallow: /cgi-bin/
        Disallow: /js/
        Disallow: /phpAdsNew/
        Disallow: /styles/
        
        User-agent: googlebot
        Disallow: /.cgi$
        Disallow: /.jpg$
        Disallow: /.gif$
        Disallow: /.bmp$
        Disallow: /images/
        Disallow: /cgi-bin/
        Disallow: /js/
        Disallow: /phpAdsNew/
        Disallow: /styles/
        
        User-agent: Slurp
        Disallow: /.cgi$
        Disallow: /.jpg$
        Disallow: /.gif$
        Disallow: /.bmp$
        Disallow: /images/
        Disallow: /cgi-bin/
        Disallow: /js/
        Disallow: /phpAdsNew/
        Disallow: /styles/
        Crawl-delay: 10
        
        User-agent: *
        Disallow: /.cgi$
        Disallow: /.jpg$
        Disallow: /.gif$
        Disallow: /.bmp$
        Disallow: /images/
        Disallow: /cgi-bin/
        Disallow: /js/
        Disallow: /phpAdsNew/
        Disallow: /styles/
        Crawl-delay: 10
        Technically, you only have to have the last block, with "useragent: *" to have the rules apply to all robots (well, all the well behaved ones, anyway). Google doesn't observe the "crawl-delay" rule, but slurp and msn do; it prevents them from crawling too fast and slowing down the server.

        See the Google recommendations at https://www.google.com/webmasters/to.../en/about.html for more information. Another good option is to create a sitemap to Google's specifications, and specifically list the URLs in it ... and tell Google how often they really change. Google will then crawl only the URLs in the sitemap, and will *presumably* honor the "change frequency" you put in it. Details are on the Google page linked above.

        Comment

        • DWalley
          Junior Member
          • Jul 2004
          • 25

          #5
          Bandwidth issues

          I am going to post in here first about my bandwidth issue.

          I have 2 questions and after reading this thread I might have the googlebot problem solved. Googlebot is eating my bandwidth. I did the robots.txt.

          Other question:

          Why is my server IP in the list of Top Host in hits?
          69.56.171.57 21859 21859 1014.11 MB 17 Mar 2007 - 05:36

          1GB for only 17 days.

          Dwayne

          Comment

          • AndrewT
            Administrator
            • Mar 2004
            • 3653

            #6
            Your best bet would be to look at the actual logs and see what the requests are.

            Comment

            • ZYV
              Senior Member
              • Sep 2005
              • 315

              #7
              I am not sure, but I think that a lot of traffic classified as "not viewable" might be eaten up by pingback/trackback stuff? Can't think of anything else apart from crawling bots as per Frank Hagan.

              Comment

              Working...