It's not specific crawlers/bots I'm worried about; I need to stick some "include" content somewhere, and if I put it "above" the site root (which I realize is optimal for security, etc.) then my paths get long, plus there are some issues of convenience in terms of creating the pages that pull toward a "downward" location. So while I'm happy to have the crawlers find and index the pages using the pulled content, I don't want to confuse things by having the material show up twice as far as the bots are concerned. Does that make sense? I have a feeling I'll end up moving the content above the site root in the end, but I was hoping for something to support what is essentially my construction-laziness. -----Original Message----- From: thelist-bounces at lists.evolt.org on behalf of Eduardo Hutter Sent: Sat 3/29/08 0:56 To: thelist at lists.evolt.org Subject: Re: [thelist] Hide entire directory from search engines? Or if you want to disable specific bots and crawlers, you could use htaccess to block them * Hassan Schroeder wrote, On 28/03/2008 4:10 PM: > On Fri, Mar 28, 2008 at 12:01 PM, Gersten, John <JGERSTEN at lchb.com> wrote: > > >> Does anyone know of a way to conceal entire directories from crawlers, etc.? >> > > Password protecting it would be one, since robots.txt is only a serving > suggestion :-) > > -- * * Please support the community that supports you. * * http://evolt.org/help_support_evolt/ For unsubscribe and other options, including the Tip Harvester and archives of thelist go to: http://lists.evolt.org Workers of the Web, evolt !