[thesite] new spidert blocks..
Daniel J. Cody
djc at starkmedia.com
Mon Oct 22 13:03:00 CDT 2001
i've increased the number of user-agents that are going to be blocked(as
explained in the apache/bad robots article) to :
EmailSiphon
EmailWolf
CherryPickerSE
CherryPickerElite
Crescent
EmailCollector
EmailSiphon
MCspider
bew
Deweb
FEZhead
Fetcher
Getleft
GetURL
HTTrack
IBM_Planetwide
KWebGet
Monster
Mirror
NetCarta
OpaL
PackRat
pavuk
PushSite
Rsync
Shai
Spegla
SpiderBot
SuperBot
tarspider
Templeton
WebCopy
WebFetcher
WebMiner
webvac
webwalk
w3mir
XGET
Wget
WebReaper
WUMPUS
FAST-WebCrawler
in the last month, these have all entered the facticious directory i set
up in our robots.txt file to trap them
if anyone notices anything odd give a holler :)
.djc.
More information about the thesite
mailing list