[thelist] googlebot strange behaviour

listman at hazard0us.org listman at hazard0us.org
Thu Aug 29 02:44:00 CDT 2002


I 've been using URL mapping on a website, for a while now to generate
friendly URLs.

/script?type=utils maps to something like -> /home/pages/utils/index.html

Google incidentally has refused to index this website beyond the 1st page.
I was checking up the http logs for this site recently and i realized
googlebot is making requests on :

/home/page/utils/ (and returning with a 404)

instead of

/home/page/utils/index.html (inspite of all the HREFs pointing to
/home/pages/utils/index.html)

Other search spiders  like Scooter (altavista)  pick up the links properly.

Anybody ever had a similar problem ?

(For some technical reasons I cannot create a valid mapping to
/home/page/utils/ that could be identical
to /home/page/utils/index.html - so that route is out of the question).

------
Ashok Hariharan
http://www.unganisha.org
------








More information about the thelist mailing list