June
28th
2006
Locking out search engines..
... has cut down 99% of my 'bad' traffic. Just magically stopped. A thing of beauty, really.
I avoided using certain words here, to avoid getting back on their radar, even though my site is now blocking search engines.
A pure robots.txt solution. I also went to their websites and used their ‘remove content’ links - which forces them to re-check robots.txt. =) I did this specific to google and the Internet Archives; leaving the others to see if they would just ‘pick it up’ - and they seem to have. I don’t have much in the way of search engine bots hitting here anymore.
I do have some hits coming in from older content on Yahoo and MSN - so I’ll go and ask them to remove that content to clear it out.
Overall, though - a thing of beauty.
But I’m sure there’s something in a shade of grey…
Something in between…
Duh, you know I already knew that, for some odd reason I guess I just had to confirm. :P I’m actually only having issues with the Slurp/Inktomi bot, but I just went ahead and set it disallow Slurp. They say that should work. we’ll see I guess.
But yeah it would be nice to be free of bots for a while, but I did just open up a few new websites, so I’m open for bots coming in for now :/ Just not Slurp… That bot is crazy!
RSS
Flickr