[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: A surprising discovery?

On Mon, 26 Apr 1999 13:10:48 -0700 (PDT), Ray Keefer
<Ray.Keefer@eng.Sun.COM> wrote:

> These files are at the web site. Anyone with the correct URL can access them.
> That is the way HTML and the World Wide Web works. 
> 
> If there are ways to lock them up then please let me know.

I think the key here is the search engines (like Altavista) generally end
up referencing the URL of a message in the archive directly.

I know some crawler-type search engines respond to a number of meta-tags
that give them information, including whether or not to go any deeper.

Check out http://info.webcrawler.com/mak/projects/robots/exclusion.html

Later,
Jon

--------------------------------------------------------------
   Jon Hylands      Jon@huv.com      http://www.huv.com/jon

  Project: Micro Seeker (Micro Autonomous Underwater Vehicle)
           http://www.huv.com