[thelist] Are search engine spiders JavaScript-enabled?
Madhu Menon
webguru at vsnl.net
Fri Nov 9 13:07:34 CST 2001
At 11:02 PM 11/9/2001, Kevin wrote:
>Check your server logs. The answer is in there. Most legit crawlers
>clearly identify themselves. It's also worth noting that many search sites
>(Yahoo, Google) do not use crawlers to gather site info.
That's not completely correct. Yahoo is a manual directory system and uses
Google technology for its web site search. Google, however, *does* crawl
sites. Its spider is called GoogleBot. How else can it index the content of
web pages?
Regards,
Madhu
<<< * >>>
Madhu Menon
User Experience Consultant
e-mail: webguru at vsnl.net
Weblog: http://madman.weblogs.com
More information about the thelist
mailing list