[thelist] Google is indexing me......
Michael Pemberton
mpember at phreaker.net
Sun Sep 9 10:20:09 CDT 2001
It might even be worth using http-agent detection to block the spider
from accessing that directory. This will stop it from putting 404 pages
in the google result.
CDitty wrote:
>
> As far as I can tell, there are not any users yet hitting these links. It
> is just the Google spider causing all these errors. Since it is the
> spider causing problems, my 404 page will have no effect.
>
> Will the mod_rewrite take care of links like this -
> /One_Time/Household_Items/Lawn_and_Garden/\forum\ ?
>
> Chris
>
> At 09:57 AM 9/9/2001, you wrote:
>
> >One option would be to just go with the flow and use mod_rewrite to
> >correct the URL.
> >
> >That is, if you have mod_rewrite available to you. The other option is
> >using the 404 document, but I don't think that will fix your errorlog
> >problem
> >
> >hth,
> >
> >--
> >mattwarden
> >mattwarden.com
> >
> >On Sep 9, CDitty had something to say about [thelist] Google is indexing...
> >
> > >Google is indexing me and is filling up my error log. For some reason,
> > >they keep trying to hit my forums and are trying to call it with a \forum
> > >instead of /forum (see below).
> > >
> > >[Sun Sep 9 01:12:53 2001] [error] [client 216.239.46.153] File does not
> > >exist: /One_Time/Household_Items/Lawn_and_Garden/\forum\
> > >
> > >
> > >They are doing this on almost every single page. Can anyone offer any
> > >advice on fixing this? I have verified all my links and they are
> > >correct. I would hate to have to block Google as I get tons of traffic
> > >from them.
> >
> >
> >---------------------------------------
> >For unsubscribe and other options, including
> >the Tip Harvester and archive of TheList go to:
> >http://lists.evolt.org Workers of the Web, evolt !
>
> ---------------------------------------
> For unsubscribe and other options, including
> the Tip Harvester and archive of TheList go to:
> http://lists.evolt.org Workers of the Web, evolt !
--
Michael Pemberton
mpember at phreaker.net
ICQ: 12107010
More information about the thelist
mailing list