[thelist] Google is indexing me......

Patrick Berry pberry at piratehaven.org
Sun Sep 9 20:31:28 CDT 2001


>>> Michael Pemberton had the thought that... <<<

> It might even be worth using http-agent detection to block the spider
> from accessing that directory.  This will stop it from putting 404 pages
> in the google result.

Use a robots.txt file.

http://www.robotstxt.org/wc/exclusion.html#robotstxt

Pat




More information about the thelist mailing list