[thelist] Google is indexing me......

Michael Pemberton mpember at phreaker.net
Sun Sep 9 10:20:09 CDT 2001


It might even be worth using http-agent detection to block the spider
from accessing that directory.  This will stop it from putting 404 pages
in the google result.

CDitty wrote:
> 
> As far as I can tell, there are not any users yet hitting these links.  It
> is just the Google spider causing all these errors.   Since it is the
> spider causing problems, my 404 page will have no effect.
> 
> Will the mod_rewrite take care of links like this -
> /One_Time/Household_Items/Lawn_and_Garden/\forum\ ?
> 
> Chris
> 
> At 09:57 AM 9/9/2001, you wrote:
> 
> >One option would be to just go with the flow and use mod_rewrite to
> >correct the URL.
> >
> >That is, if you have mod_rewrite available to you. The other option is
> >using the 404 document, but I don't think that will fix your errorlog
> >problem
> >
> >hth,
> >
> >--
> >mattwarden
> >mattwarden.com
> >
> >On Sep 9, CDitty had something to say about [thelist] Google is indexing...
> >
> > >Google is indexing me and is filling up my error log.  For some reason,
> > >they keep trying to hit my forums and are trying to call it with a \forum
> > >instead of /forum (see below).
> > >
> > >[Sun Sep 9 01:12:53 2001] [error] [client 216.239.46.153] File does not
> > >exist: /One_Time/Household_Items/Lawn_and_Garden/\forum\
> > >
> > >
> > >They are doing this on almost every single page.  Can anyone offer any
> > >advice on fixing this?  I have verified all my links and they are
> > >correct.  I would hate to have to block Google as I get tons of traffic
> > >from them.
> >
> >
> >---------------------------------------
> >For unsubscribe and other options, including
> >the Tip Harvester and archive of TheList go to:
> >http://lists.evolt.org Workers of the Web, evolt !
> 
> ---------------------------------------
> For unsubscribe and other options, including
> the Tip Harvester and archive of TheList go to:
> http://lists.evolt.org Workers of the Web, evolt !

-- 
Michael Pemberton
mpember at phreaker.net
ICQ: 12107010





More information about the thelist mailing list