Note 62: Altavista: Scooter in the case

Rubric: Our networks

Altavista: Scooter in the case

Anton Nosik <anton@beseder.com>

In the last issue of our column, we talked about the practical aspects of using the Altavist search engine (http://altavista.digital.com/). Today we’ll talk about its anatomy, internal structure and principles of work.

First of all, we are puzzled by the question: where is Altavist looking for those documents whose addresses she gives us? Of course, not on all the big Internet: in this case, serving any request would take years. Documents viewed by the search engine (any - not only Altavist, but also Lycos, Webcrawler, Inktomi, Yahoo, Magellan, etc.), are stored in a special indexed database on its local disk, along with those addresses that can find the originals of these documents on the network. The Altavista uses the technology developed by Digital (full name - Dec, Digital Equipment Corporation). In fact, the search robot is one large advertising of the efficiency and power of this technology, as well as the software and hardware products of the DEC in which it is used.

From this method of indexing the network, one regrettable, but inevitable conclusion: the search results Altavists by definition are initially unreliable. Because home pages are constantly changing; Each of 30,000,000 documents, copies of which are stored in the Altavistan database, can disappear at any time or undergo cardinal transformations, the automatic reflection of which in the DEC database is fundamentally impossible. The average life of the document on WWW is 52 days, and Altavista does not delete documents from his base for several months, and correctly, in general, does. Now I will even explain why. To do this, you should refer to the question of how the database Altavists is equipped and supported.

The creation and replenishment of this base is engaged in a robot named Scooter. For three years before the official announcement of the discovery, Altavists (December 1995) scooter periodically traveled along the web, “read” all the documents found there and drove into one large base along with addresses. This base was indexed, and the contents of the documents became part of the overall search space.

All links (references to other addresses or documents found in the text of the indexed page) recorded a scooter, and writes to this day, in his address book, in the order of receipt. That is, by otindexing a page that contains links to another forty addresses, the scooter not only enriches the altavist database with the text of this document, but also adds four dozen new addresses to the route of its future excursions. Of course, they are added to the end of his list, so if there is already a million other unexplored addresses there, then forty freshly -worn -up links should wait for a long time.

The situation has changed somewhat since the activity of the scooter from the secret has become obvious. From this day, forty million visitors to the web were allowed to "feed with" an omnivorous scooter, adding their pages to its targeted base using a special interface located at:

http://altavista.digital.com/cgi-bin/query?pg=adDURL

And this interface today works in a completely amazing way. You add a new address and click on Submit. The robot first visible its base of 40-50 million addresses, checking if he had time to visit this page before. If the address is familiar - the robot says "Thank you, I was already there." Otherwise, he immediately goes to the page submitted. If he discovers it, then instantly pours the contents into his base. One and a half seconds after pressing the Submit button, we get a message about this content on the screen:

"Thank you for telling us a new address. The page you specified was considered from the network for 0.3 (for example) seconds and added to our base. It will be included in the search base within a day, and within a few days it will be completely indexed."

Strictly speaking, this is not at all not true, but this is not a very accurate message. We are not talking about any hours or days. At that very second, when we find out that the scooter considered our page from the network, all its contents have already been listed in the Altavistan base, and when indicating the corresponding keywords, it can be found there. I can imagine how many high -forgotten Internet experts, having read this phrase, will consider it an extra confirmation of my general computer illiteracy and ignorance. Therefore, I propose to anyone who doubts to conduct a very simple experiment:

1. Create a page on which there would be some inconceivable word, such as Golovozhopoe. Here is the approximate text of such a page:

<Title> Proverka Altavisty </ Title>

</head> <body>

ETA Stranica Ne Neset Nikakoj Smyslovoj Nagruzki. Vse Pretenzii po etomu povodu k <a href = mailto: nosik@usa.net> nosiku </a>.

</body> </ html>

I wrote the text of the page in broadcast (in Russian in Latin letters), however, if you have Russian keyboard drivers, then Altavist is the same successfully indexes Alt, CP1251, KOI8, and any Hebrew text.

2. Place this page on the www server available to you. If you do not have it, make yourself a directory in Geocities, at http://www.geocities.com/. All pleasure (including reading instructions) will take 15-25 minutes, depending on your acquaintance with the Internet and English.

3. Go to the Altavist at the above address of the adding interface and indicate there the full address (URL) of your freshly built page.

4. After receiving the confirmation that the page is considered from the network, follow the Golovozhopoe for the word in Altavist.

5. If the Altavista does not immediately give you the address and the contents of your newly registered page, let me know about this at the famous address, and I am playing in the lamers in publicly.

Unfortunately, we will have to interrupt this note today, so read the end of the story about Altavist in tomorrow's release of the newspaper.

Tomorrow we will complete the trip to the bowels of the Altavists with a reference to Scott Pakin, who gathered a lot for himself valuable from our publication over the past environment. After that - already, apparently, next week - we will step on the ground of the cherished archipelago of the Easter cakes ...