I am investigating a problem that I noticed on some of my sites. It showed up that I am not the only one noticing it.
Search Engine crawlers are becoming quite annoying. They spider sites far to often and far to aggresive. Usualy they do not follow instructions in robots.txt. Ot seems that the only solution is to simply block IP addresses where bots come from and get rid of them. This, of course is no god solution, since that means you wil loose any visibilty on search engines, which is hardly something one would like to do.
Forums, blogs and other dynamic content sites are most vurnelable, since most bots do not recognize cyclical links that are usual for this kind of sites and they crawl same pages again and again. IT is also problem that each connection bot makes to the site is presented as new session, so site statistics is messed out, but also, it is not uncommon that bots do not recognize session ID's provided by sites, so they see the same page as new page just because it has different SID.
There is also one more problem, some spammers are using crawling techniques to promote. They simply crawl your site but put their own url in http Referer (yes, referer may be easily faked). It causes that if you have statistics of your site (and usualy you do), their site shows up as referer link. Most ridiculous examle is spammer who puts Gogole search url as referer, but with predefined keys that lead to his site.
I am stating ths topic because I found out that one forum (which is usualy not resource hog) I host on Dathorn, suffered bot and spam attack that lasted for seven days. It was under constant load by search engine bots and spammers so it spent enormous traffic and number of connections. I am wondering how Dathorn stuff did not notice and suspend my site for that.
The only solution I hve is to simplu block IP or IP ranges of hosts where such connections came from. I hoipe, it wil at least lower ammount of unwanted traffic.
Long tiem ago I set robots.txt for that forum to instruct spiders to take it is on my board. For soem of them it works, for others it does not.
Here is an contents of robots.txt for those who want to implement it on their forums (this is fopr phpBB forum). Just save it in file named robots.txt and place it in the root of your site (forum must be installed in rot to, otherwise alter file and directory paths to match those on your site).
Unfortunately this is not enough, so I am looking for a ways to further control search engine spiders on site. There is an obvious soultion to change forum code so it recognizes and handles spiders differently, but that is not easy to implement and it is actualy against spider policy as they demand to see exaclty the same contents as ordinary user, otherwise, they snadbox site.
One thing that came to my mind is to traffic shape connections coming from spiders. That would allow us to slow down all connections coming from the spiders. It actually may be quite effective, especialy if you really slow it down. First, load to site would be definitely decreased simpy do e to slow connections, and second you won't be blocking spiders, meaning, they wil stil be able to crawl but slower.
So, you get both good effects: site is releived of load and stil indexed by search engines. Bad thing is that indexing would not be fast, but it is slow anyways (because of many other factors), so it won't make soem difference.
Traffic shaping could easily solve another problem: some spiders are unpolite ant they create many concurent connections to your site, incirising load further. Polite spiders crwal single page at a time. For such unplite spiders solution is to limit number of connections thay are able to create. Remember, all this limitations should be used only on known spam bot IP addresses.
Well, this seems fine, but here is one huge problem. we cannot set such shaping. Only Dathorn stuff can, and if they do, it would affect all sites. I do not expect them to be willing to do this, but, at least I mentioned it.
I am open to other ideas to control spam bots but keep site indexed on search engines.
Search Engine crawlers are becoming quite annoying. They spider sites far to often and far to aggresive. Usualy they do not follow instructions in robots.txt. Ot seems that the only solution is to simply block IP addresses where bots come from and get rid of them. This, of course is no god solution, since that means you wil loose any visibilty on search engines, which is hardly something one would like to do.
Forums, blogs and other dynamic content sites are most vurnelable, since most bots do not recognize cyclical links that are usual for this kind of sites and they crawl same pages again and again. IT is also problem that each connection bot makes to the site is presented as new session, so site statistics is messed out, but also, it is not uncommon that bots do not recognize session ID's provided by sites, so they see the same page as new page just because it has different SID.
There is also one more problem, some spammers are using crawling techniques to promote. They simply crawl your site but put their own url in http Referer (yes, referer may be easily faked). It causes that if you have statistics of your site (and usualy you do), their site shows up as referer link. Most ridiculous examle is spammer who puts Gogole search url as referer, but with predefined keys that lead to his site.
I am stating ths topic because I found out that one forum (which is usualy not resource hog) I host on Dathorn, suffered bot and spam attack that lasted for seven days. It was under constant load by search engine bots and spammers so it spent enormous traffic and number of connections. I am wondering how Dathorn stuff did not notice and suspend my site for that.
The only solution I hve is to simplu block IP or IP ranges of hosts where such connections came from. I hoipe, it wil at least lower ammount of unwanted traffic.
Long tiem ago I set robots.txt for that forum to instruct spiders to take it is on my board. For soem of them it works, for others it does not.
Here is an contents of robots.txt for those who want to implement it on their forums (this is fopr phpBB forum). Just save it in file named robots.txt and place it in the root of your site (forum must be installed in rot to, otherwise alter file and directory paths to match those on your site).
Code:
User-agent: * Disallow: /cgi-bin/ Disallow: /img Disallow: /images Disallow: /admin/ Disallow: /_decomas/ Disallow: /_smarty/ Disallow: /smarty/ Disallow: /login.php Disallow: /posting.php Disallow: /privmsg.php Disallow: /search.php Disallow: /memberlist.php Disallow: /groupcp.php Disallow: /statistics.php Disallow: /viewonline.php Disallow: /rules.php Disallow: /modcp.php Disallow: /logout.php Disallow: /album.php Disallow: /printview.php Crawl-delay: 360
One thing that came to my mind is to traffic shape connections coming from spiders. That would allow us to slow down all connections coming from the spiders. It actually may be quite effective, especialy if you really slow it down. First, load to site would be definitely decreased simpy do e to slow connections, and second you won't be blocking spiders, meaning, they wil stil be able to crawl but slower.
So, you get both good effects: site is releived of load and stil indexed by search engines. Bad thing is that indexing would not be fast, but it is slow anyways (because of many other factors), so it won't make soem difference.
Traffic shaping could easily solve another problem: some spiders are unpolite ant they create many concurent connections to your site, incirising load further. Polite spiders crwal single page at a time. For such unplite spiders solution is to limit number of connections thay are able to create. Remember, all this limitations should be used only on known spam bot IP addresses.
Well, this seems fine, but here is one huge problem. we cannot set such shaping. Only Dathorn stuff can, and if they do, it would affect all sites. I do not expect them to be willing to do this, but, at least I mentioned it.
I am open to other ideas to control spam bots but keep site indexed on search engines.
Comment