[thelist] [SPIDERS] Can Dynamic pages be crawled?

Russ russ at unrealisticexpectations.com
Fri Nov 22 10:16:01 CST 2002


Yes, they can.  I say this with no authority or certainty; as I started
typing it, I started to see a loophole since I provide a link to the
specific content, and that could be spidered, as well.

I have content that is dynamically generated that gets spidered all the
time and shows up via a specifically defined URL in Google, etc.
searches all the time.



> -----Original Message-----
> From: thelist-admin at lists.evolt.org
> [mailto:thelist-admin at lists.evolt.org] On Behalf Of Rob Whitener
> Sent: Friday, November 22, 2002 10:11 AM
> To: 'thelist at lists.evolt.org'
> Subject: RE: [thelist] [SPIDERS] Can Dynamic pages be crawled?
>
>
> Yes, they can.  I read about it on the internet.  I can't
> remember any specifics about it, but in the meta-tags if you
> include index or content follows attribute, the spiders will
> access your pages.  Look around on google for meta tags, web
> crawlers, and dynamic content.
>
> -----Original Message-----
> From: Benjer [mailto:futureweb at macmail.com]
> Sent: Friday, November 22, 2002 10:05 AM
> To: thelist at lists.evolt.org
> Subject: [thelist] [SPIDERS] Can Dynamic pages be crawled?
>
>
> On a site with lots of dynamic content being pulled from the
> datatbase, can these pages be crawled by spiders?
>
> I remember reading about the url itself may cause the problem?
>
> Benjer
>
>
> --
> * * Please support the community that supports you.  * *
http://evolt.org/help_support_evolt/

For unsubscribe and other options, including the Tip Harvester and
archives of thelist go to: http://lists.evolt.org Workers of the Web,
evolt !
--
* * Please support the community that supports you.  * *
http://evolt.org/help_support_evolt/

For unsubscribe and other options, including the Tip Harvester
and archives of thelist go to: http://lists.evolt.org
Workers of the Web, evolt !




More information about the thelist mailing list