We hate spammers
lionel	Need help with software development? consulting and outsourcing marketplace, where you can post your project free and receive bids from a global pool of knowledge workers and service providers. I'm a spammer and I'm stupid.
Almar Joling	The MSN bot is horrible. It downloaded about 80 times YALG before I noticed the download logs (yes, I keep track of all downloads on my site, just for stuff like this). It's like 800mb. Not that Icare that much (800mb isn't really much), but it would simply have continued for weeks and months. If you search on some of the IP's eric, you can find some info on most bots, and possibly a) ban them or b) use robots.txt (which is ignored anyway by most search engines, imo) I'm simply returning an error from the download counter script, if the IP matches the MSN bot one.
Eric Coleman	Here are some interesting graphs. The numbers at the bottom are Month / Day /Year. The graphs show daily activity for 2004. This shows browser referrals. Most people come from google. Practically 0 people are coming from either the Yahoo or MSN website. VBgamer aparently doesn't rank very high with them. Download Attachment: [url="http://www.vbgamer.com/msgboard/uploaded/eric coleman/20041222105448_Daily_Search_Engines.png"][img]icon_paperclip.gif[/img]Daily_Search_Engines.png[/url] 9.42 KB This shows the spider or bot's daily activity. Around Aug 1 is when bot activity dramatically increased. Google has the highest hit count for a single day. Download Attachment: [url="http://www.vbgamer.com/msgboard/uploaded/eric coleman/20041222105412_Top_Spiders.png"][img]icon_paperclip.gif[/img]Top_Spiders.png[/url] 5.76 KB However, if you look at this graph, you can see that overall the MSN robot is more active than either Google or Yahoo(Inktomi Slurp is Yahoo). Download Attachment: [url="http://www.vbgamer.com/msgboard/uploaded/eric coleman/20041222105515_Daily_Spider_Activity.png"][img]icon_paperclip.gif[/img]Daily_Spider_Activity.png[/url] 7.77 KB What's interesting about all this is comparing bot activity versus the number of hits from those search engines. Google brings in the most hits, so it's robot activity is somewhat usefull. MSN downloads lots of stuff but no one ever finds vbgamer by searching with MSN. By comparing pages downloaded to search engine referrals we get the following ratios: Google: 1.4 pages downloaded for every referal MSN: 98.7 pages downloaded for every referal Yahoo: 16.9 pages downloaded for every referal MSN just sucks!
VBBR	I think we're getting more MSN bot hits because the MSN search system is being re-created or something (they say to better it), so I think it needs to crawl everything again. Anyway, Google rocks.
Almar Joling	Yeah, I read somewhere the MSNBot is quite aggressive :)
VBBR	...wait. I think that if VBGamer wanted advertising in the website, it could use Google or something like that. Or am I wrong? Did the staff authorize that?
Eric Coleman	this is probably just a bot of some sort. I guess it's stime to start locking down different areas of the website.
VBBR	Shouldn't the topic be deleted? We're advertising it yet more by replying. And what about the "numbers-and-letters-mixed-with-lines-so-a-bot-can't-register" approach? [:D]
Eric Coleman	I'm not sure what kind of software I would have to install on the server to do stuff like that. Instead I was thinking more along the lines of a simple test. A few questions with multiple answers. Bots aren't smart enough to understand language yet but the questions would be obvious. What is two plus 2? A couple of answers could be 2, two, 4, etc. Or maybe something like E=mc^2? The answers could be Albert Einstein, George Bush, Your mother. I removed the link from the post above. I think we can continue this post on how best to defeat spam bots. Lately VBgamer is being crawled very heavily by search engines. MSN, Inktomi, and Google are really big sources of bot traffic, but there are lots of other bots though. I suspect they're mostly email harvesters.
VBBR	You could display a simple picture of a circle, square or triangle (randomly), then ask the user "What is that?" -> "A circle" / "A Square" / "A Triangle" / "None of the above". Maybe the 5th option could be "I'm a bot". [:D]
Krisc	I think it might even be enough to just create a "Click yes to verify you are not a bot..."