[thelist] Hide entire directory from search engines?

[Gersten, John]

It's not specific crawlers/bots I'm worried about; I need to stick some "include" content somewhere, and if I put it "above" the site root (which I realize is optimal for security, etc.) then my paths get long, plus there are some issues of convenience in terms of creating the pages that pull toward a "downward" location. So while I'm happy to have the crawlers find and index the pages using the pulled content, I don't want to confuse things by having the material show up twice as far as the bots are concerned. Does that make sense?

I have a feeling I'll end up moving the content above the site root in the end, but I was hoping for something to support what is essentially my construction-laziness.


-----Original Message-----
From: thelist-bounces at lists.evolt.org on behalf of Eduardo Hutter
Sent: Sat 3/29/08 0:56
To: thelist at lists.evolt.org
Subject: Re: [thelist] Hide entire directory from search engines?
 
Or if you want to disable specific bots and crawlers, you could use
htaccess to block them

* Hassan Schroeder wrote, On 28/03/2008 4:10 PM:
> On Fri, Mar 28, 2008 at 12:01 PM, Gersten, John <JGERSTEN at lchb.com> wrote:
>
>   
>> Does anyone know of a way to conceal entire directories from crawlers, etc.?
>>     
>
> Password protecting it would be one, since robots.txt is only a serving
> suggestion  :-)
>
>   

-- 

* * Please support the community that supports you.  * *
http://evolt.org/help_support_evolt/

For unsubscribe and other options, including the Tip Harvester 
and archives of thelist go to: http://lists.evolt.org 
Workers of the Web, evolt !