The problem they were solving
By around 1990, the internet connected many universities and research sites, and a lot of useful material sat on public file servers. Anonymous file transfer let anyone download files without a personal account. But there was no master list. If you did not already know where a program or document was, you had to ask around or browse server after server.
Early search tools were built to fix this. They were small projects, often started by students or staff at universities, and they were not companies in the modern sense.
Searching file names
One of the best-known early tools, Archie, was developed around 1990 at McGill University in Montreal. It periodically contacted public file servers, collected their directory listings, and combined them into a searchable database. A user typed part of a filename and received a list of servers holding matching files.
The limits were obvious. Archie searched names, not contents, so a file called something unhelpful could not be found by what it contained. It also depended on file naming habits, which varied.
Menus and their search tools
Around 1991, Gopher was introduced at the University of Minnesota. It presented information as nested menus, which was easy to navigate and ran well on text terminals. Gopher servers multiplied quickly at universities. Because there were so many menus, additional tools were created to search them. The best known, Veronica, searched menu titles across Gopher servers, and a companion tool called Jughead did something similar for a single server. The names played on the comic strip characters of Archie.
Another system of the same era, WAIS, searched the text of documents in indexed collections, which was closer to full-text search. It was used mainly with specialized collections rather than the open internet.
Moving to web pages
As the web spread in the early 1990s, people wanted to search it too. Pages were scattered across servers and linked to one another in no fixed order. A tool could find pages by following links, a process called crawling, then record words from each page in an index.
Several experiments appeared in the early to mid-1990s. Some catalogued page titles and headers only. Others stored more of each page’s text. Around the middle of the decade, a number of crawler-based services launched to the public, including systems from universities and from computer companies, and some claimed to index very large portions of the web. Details such as which one was first or largest are disputed and depend on definitions, so any single “first” claim should be treated carefully.
How a crawler-based engine worked
In general terms, these services had three parts:
- A crawler. A program that started from a list of known pages, downloaded each one, and extracted its links to find more pages.
- An index. A database mapping words to the pages containing them, so that a query did not require reading every page again.
- A query system. A page where users typed words and got a list of matching pages in some order.
This basic structure is still recognizable in modern engines, although the scale and methods are very different.
Why results were often poor
Early results had several weaknesses:
- Coverage was incomplete. The web grew faster than crawlers could revisit it, so pages were missing or stale.
- Ranking was simple. Results were often ordered by how often the query words appeared or how close they were to the top of a page. This was easy for page authors to exploit by repeating words or hiding text, a form of spam.
- Queries were literal. A search for a word matched that word, and not necessarily the idea behind it.
- Dead links were common. Pages moved or disappeared, and indexes took time to catch up.
Users learned workarounds, such as using exact phrases, adding specific terms, or trying several services. Some sites even combined the results of multiple engines into one page.
Competition with directories
Search engines were not the only way to find pages. Human-edited lists of sites were popular at the same time, because a person had already judged what belonged in each category. Directories were often more reliable for broad topics, while crawlers were better for specific, obscure queries. For some years the two approaches coexisted, and some companies offered both on one site.
The turn toward ranking
By the late 1990s, the quality of results had become a central concern. Researchers explored using the web’s link structure as a clue to importance, an idea that led to link-based ranking. This did not replace the crawl-and-index design; it improved the ordering of results. A service that used links alongside text matching became very widely used in the following years.
Why this period matters
The early tools show how search was shaped by practical limits. Slow connections, small disks and limited computing power meant indexes covered only part of what existed. Their designers made decisions, such as how often to revisit pages or whether to index full text, that influenced what could be found.
They also set patterns still visible today: a simple box for typing, a list of results, and a constant contest between those who build indexes and those who try to manipulate them. For readers interested in the networks these tools ran on, the story of the ARPANET and the early internet gives useful background, and the arrival of the World Wide Web explains why crawling pages became the main task.