[thelist] Links and Spiders

[David Dorward]

On Sun, Dec 29, 2002 at 07:44:08AM -0800, Administrative HQ wrote:
> 1. How can I know if another site has linked to mine
> (and, preferably, who it is)?

With difficulty. There are a few approaches:

1.

You can usually examine your server log and find entries with referrers
from other sites. This will show when someone follows a link from
another site to yours. (Not 100% reliable, according to the http
specification sending a referrer is optional and some people don't)

2.

You can do a search for links to a site in some search engines.

(Watch the wrap)

http://www.google.com/search?q=link:debian.org&hl=en&ie=UTF-8
&lr=lang_en&sa=X&oi=lrtip7

3.

You can implement a technology such as pingback. This isn't widely
supported (at least currently), however it is rather nifty:

http://www.hixie.ch/specs/pingback/pingback

> 2. Can search engine robots access pages which require
> login with user name and password (where my outward
> bound links are located)?

In theory - yes, providing you gave them a username and password.
In practice - probably not.

--
David Dorward                                   http://david.us-lot.org/
"You cannot rewrite history, not one line."
                                      - The Doctor (Dr. Who: The Aztecs)