[thelist] Hide entire directory from search engines?

[Joe Flintham]

Gersten, John wrote:
> I need to stick some "include" content somewhere, ...and if I put it "above" the site root (which I realize is optimal for security, etc.) then my paths get long, plus there are some issues of convenience in terms of creating the pages that pull toward a "downward" location. So while I'm happy to have the crawlers find and index the pages using the pulled content, I don't want to confuse things by having the material show up twice as far as the bots are concerned. Does that make sense?
>
> I have a feeling I'll end up moving the content above the site root in the end, but I was hoping for something to support what is essentially my construction-laziness.
>
>
> -----Original Message-----
> From: thelist-bounces at lists.evolt.org on behalf of Eduardo Hutter
> Sent: Sat 3/29/08 0:56
> To: thelist at lists.evolt.org
> Subject: Re: [thelist] Hide entire directory from search engines?
>  
> Or if you want to disable specific bots and crawlers, you could use
> htaccess to block them
>
>   
Hi John

I think Eduardo's suggestion still stands: you can still 'include' your 
includes using SSI, php, [or whichever scripting environment you're 
using], even though the directory may be unreadable via any HTTP request 
by virtue of htaccess (and therefore unreadable by bots).  And if (as 
you suggest) the bots will get to see the content of your includes at 
least once _anyway_, then you don't need to worry about excluding them 
from a second, independent crawl.

Joe