[thesite] new spidert blocks..

Daniel J. Cody djc at starkmedia.com
Mon Oct 22 13:03:00 CDT 2001


i've increased the number of user-agents that are going to be blocked(as 
explained in the apache/bad robots article) to :

EmailSiphon
EmailWolf
CherryPickerSE
CherryPickerElite
Crescent
EmailCollector
EmailSiphon
MCspider
bew
Deweb
FEZhead
Fetcher
Getleft
GetURL
HTTrack
IBM_Planetwide
KWebGet
Monster
Mirror
NetCarta
OpaL
PackRat
pavuk
PushSite
Rsync
Shai
Spegla
SpiderBot
SuperBot
tarspider
Templeton
WebCopy
WebFetcher
WebMiner
webvac
webwalk
w3mir
XGET
Wget
WebReaper
WUMPUS
FAST-WebCrawler

in the last month, these have all entered the facticious directory i set 
up in our robots.txt file to trap them

if anyone notices anything odd give a holler :)

.djc.





More information about the thesite mailing list