[thelist] [SPIDERS] Can Dynamic pages be crawled?

Morgan Kelsey morgan at morgankelsey.com
Fri Nov 22 10:27:01 CST 2002


> Yes, they can.  I say this with no authority or certainty; as I started
> typing it, I started to see a loophole since I provide a link to the
> specific content, and that could be spidered, as well.
>
> I have content that is dynamically generated that gets spidered all the
> time and shows up via a specifically defined URL in Google, etc.
> searches all the time.
>

depends how big the site is.
from the horses mouth:
http://www.google.com/webmasters/2.html#A1
1. Reasons your site may not be included.

  a.. Your pages are dynamically generated. We are able to index dynamically
generated pages. However, because our web crawler can easily overwhelm and
crash sites serving dynamic content, we limit the amount of dynamic pages we
index.
disguising your query strings as directories is still the best way to go for
high search engine rankings.


nagrom




More information about the thelist mailing list