[thelist] To Crawl... or Not To Crawl

[Rob Smith]

Hey list,

We've got the Google Mini appliance as our search engine. The paradox
I'm in is that we want it to crawl and capture all the SKU's of all our
products. The rub is that the pricing is on the same page as the SKU's.
We don't want the pricing info to be cached. What I've done to trick out
the Mini is have a page off our department listing page that goes to a
page with the SKU's, and without the pricing. Upon entering the page,
you're redirected to the real page with the pricing. The Mini finished
recrawling our site last night and had all my trickster pages listed as
"Info: redirected URL" ... it crawled it successfully, but didn't store
it on the search index as good URLs to list. 

I don't want to create a comprehensive page of all SKU's per product. My
initial go 'round with that was a bloated 6 MB worth of plain text; not
web friendly to say the least. 

My lame next thought would be to store all SKU's on the same page as the
initial product listing:

Sunset Photo eSatin Paper 300g <div
style="font-size;1px;color:white">(3PES851150,3PES8511,...etc.)</div>

{next product in department}

Suggestions?


Rob Smith
LexJet
rob.smith at lexjet.com
http://www.lexjet.com
(800)453-9538
(941)330-1210 Int'l
(941)330-1220 Fax
1680 Fruitville Road, 3rd Floor
Sarasota, FL 34236