[thelist] evolt robots.txt [was More Smart Tags]

[Daniel J. Cody]

Erik Mattheis wrote:


> ... and went to grab an example - Try to dialectize an evolt URL ... 
> interesting result.
> 
> http://rinkworks.com/dialect/
> 
> Is this because someone at evolt actually has requested to be removed or 
> is is comming from the robots.txt file:
 

Not sure why that comes up for the dialectizer thing.. We haven't 
blocked it on purpose, thats for sure..

 
> User-agent: *
> Disallow:
> 
> And how can evolt be all over google with that robots.txt? ...

Actually the evolt.org robots.txt files doesn't block anything. A
Disallow: /

would block everything.
Disallow:

blocks nothing.. Check out 
http://www.robotstxt.org/wc/exclusion-admin.html for more info :)

.djc.