[thelist] CMS: dynamic pages and search engines/spiders

[nets at searchtools.com]

At 4:44 PM -0700 5/20/02, Chris W. Parker wrote:
>  > It indexes querystrung pages on my org's site wonderfully. Dunno the
>  > mechanism tho.. Guess it's all just link-following

AltaVista also indexes some pages with query strings, but many other
search engines just ignore them.  It's a good idea to convert the
urls to slashes, then you're sure to get search engine robots.

>so then if the kb article is not linked somewhere on the site then
>obviously the spider will not reach it right? (that's probably an
>obvious 'right!'.)

Not entirely.  If anyone anywhere links to it, spiders can find it.
You have no control over who links to it.  You should protect it with
a password if you want to keep it private.

>one more question...
>
>would it be worthwhile to have a link that read like this... "Show All
>Articles"... and on that page list every article (or maybe every article
>in a certain category or something) as a link so that the spider had
>some more articles to crawl?

Yes, that's an excellent idea.  It makes it very easy for the robot
spiders to follow the links.   And if anyone actually wants to read
all your articles, they can do that too :-)   Be sure to date the
articles, so humans have a little chronological context.

Avi

--
Complete Guide to Search Engines for Web Sites and Intranets
    <http://www.searchtools.com>