[thelist] [SPIDERS] Can Dynamic pages be crawled?

Rob Whitener rwhitener at DesignOptions.com
Fri Nov 22 10:10:01 CST 2002


Yes, they can.  I read about it on the internet.  I can't remember any
specifics about it, but in the meta-tags if you include index or content
follows attribute, the spiders will access your pages.  Look around on
google for meta tags, web crawlers, and dynamic content.

-----Original Message-----
From: Benjer [mailto:futureweb at macmail.com]
Sent: Friday, November 22, 2002 10:05 AM
To: thelist at lists.evolt.org
Subject: [thelist] [SPIDERS] Can Dynamic pages be crawled?


On a site with lots of dynamic content being pulled from the datatbase, can
these pages be crawled by spiders?

I remember reading about the url itself may cause the problem?

Benjer


--
* * Please support the community that supports you.  * *
http://evolt.org/help_support_evolt/

For unsubscribe and other options, including the Tip Harvester
and archives of thelist go to: http://lists.evolt.org
Workers of the Web, evolt !



More information about the thelist mailing list