[thelist] evolt robots.txt [was More Smart Tags]
Daniel J. Cody
djc at starkmedia.com
Mon Jun 25 10:34:20 CDT 2001
Erik Mattheis wrote:
> ... and went to grab an example - Try to dialectize an evolt URL ...
> interesting result.
>
> http://rinkworks.com/dialect/
>
> Is this because someone at evolt actually has requested to be removed or
> is is comming from the robots.txt file:
Not sure why that comes up for the dialectizer thing.. We haven't
blocked it on purpose, thats for sure..
> User-agent: *
> Disallow:
>
> And how can evolt be all over google with that robots.txt? ...
Actually the evolt.org robots.txt files doesn't block anything. A
Disallow: /
would block everything.
Disallow:
blocks nothing.. Check out
http://www.robotstxt.org/wc/exclusion-admin.html for more info :)
.djc.
More information about the thelist
mailing list