[thelist] web site check (again)

Mark Groen markgroen at gmail.com
Mon Jun 13 09:30:45 CDT 2005


> Having said that, I don't know with any certainty whether robots crawl
> files via https at all.

There's a lot of good content that just happens to be on an ssl server,
as long as there isn't a secure log-in or robots exclusion file it's
fair game for Google. This site has been crawled for instance:

https://www.linux-community.de/

A beta link that might be useful to some, the Google Site Map:

https://www.google.com/webmasters/sitemaps/docs/en/about.html

way off topic :-)

cheers,

        Mark



More information about the thelist mailing list