[thelist] [SPIDERS] Can Dynamic pages be crawled?
Rob Whitener
rwhitener at DesignOptions.com
Fri Nov 22 10:10:01 CST 2002
Yes, they can. I read about it on the internet. I can't remember any
specifics about it, but in the meta-tags if you include index or content
follows attribute, the spiders will access your pages. Look around on
google for meta tags, web crawlers, and dynamic content.
-----Original Message-----
From: Benjer [mailto:futureweb at macmail.com]
Sent: Friday, November 22, 2002 10:05 AM
To: thelist at lists.evolt.org
Subject: [thelist] [SPIDERS] Can Dynamic pages be crawled?
On a site with lots of dynamic content being pulled from the datatbase, can
these pages be crawled by spiders?
I remember reading about the url itself may cause the problem?
Benjer
--
* * Please support the community that supports you. * *
http://evolt.org/help_support_evolt/
For unsubscribe and other options, including the Tip Harvester
and archives of thelist go to: http://lists.evolt.org
Workers of the Web, evolt !
More information about the thelist
mailing list