Information online before the web
Networked information existed well before the web. In the 1960s and 1970s, researchers on shared computers and early research networks exchanged messages and files, usually in small, specialized groups. Finding something meant knowing who had it, or asking someone who did.
Two kinds of tools from this period matter for what came later. One was messaging between users, which grew into electronic mail. The other was the idea of linked documents, developed by a handful of thinkers who imagined text that readers could navigate by following connections rather than reading in order. Neither was a search tool, but both set expectations about what a network of information could be.
Dial-up communities
In the late 1970s and 1980s, home computers and inexpensive modems let ordinary people connect to each other over telephone lines. Bulletin board systems were typically run by hobbyists from a spare computer. Callers dialed in, one at a time, to read messages and download files. Larger commercial online services appeared alongside them, and discussion networks such as Usenet carried topic-based conversations between sites.
Discovery on these systems was social. You learned about a good board from a friend, a printed list, or another board’s menu. There was no general way to ask a question of “everything online”, because there was no single “everything”.
Indexing files on the early internet
By the late 1980s and early 1990s, universities and research centers were connected through the growing internet, and public file archives held software, documents and data. The problem was practical: files were scattered across many servers, and nobody had a full list.
Several groups built tools to address this. Archie, developed around 1990, gathered file listings from public archives so that users could search by filename. Gopher, from around 1991, organized information into menus that could be browsed across servers, and companion tools were built to search those menus. These were among the early search engines, and they worked on names and titles, not on the full text of pages.
The arrival of the web
In the same period, Tim Berners-Lee at CERN proposed a system for sharing linked documents among researchers. The key pieces were an address scheme for locating documents, a protocol for fetching them, and a page format built on HTML markup. The first web software appeared around 1990 and 1991, and the system spread beyond physics laboratories in the following years. Graphical browsers, notably Mosaic in 1993, made pages with text and images easy for non-specialists to use, and they were an important part of the story of web browsers.
A point worth keeping clear is that the web is one service that runs on the internet, not the internet itself, a distinction covered in more detail in the discussion of web versus internet. Email, file transfer and many other services existed before the web and continue alongside it.
Hypertext ideas had been explored before, including Vannevar Bush’s 1945 essay on a desk-sized “memex” and later working systems at universities and companies. What the web added was openness. Anyone with a server could publish, and a link could point to any other page without permission from a central owner.
Directories and the first web search tools
Once pages multiplied, the same discovery problem returned. Early answers took two forms.
The first was the human-curated list. Small collections of links turned into large web directories, where editors sorted sites into categories. The best-known example from the mid-1990s began as a personal list of links before growing into a company, and a volunteer-edited directory followed later in the decade.
The second was the automated crawler. Programs followed links from page to page, stored what they found, and let users search the stored text. Several such services appeared in the mid-1990s, and some could search the full text of pages instead of only titles. Early results were often ordered mainly by how well the words on a page matched the query, which made them easy to manipulate by repeating words.
Ranking by links
Late in the 1990s, researchers began treating links themselves as information. If many pages link to a page, that suggests it may be useful, and links from pages that are themselves well linked may count for more. Link-based ranking was developed independently by several researchers; one well-known version came from two Stanford students who later founded Google. Combined with text matching, this approach often produced more useful result lists for general queries, though it did not remove the need to deal with spam and manipulation, which continued to evolve.
Over time, directories lost ground because the web grew faster than editors could classify it, and a short search query was usually quicker than browsing categories.
Collaboration and reference
Another thread was how people built shared knowledge on the web. The wiki, a page anyone can edit, appeared in the mid-1990s. In 2001 a volunteer-written encyclopedia built on a wiki launched and grew steadily. Its rise coincided with the decline of large printed sets and the move of established publishers toward digital editions. Online encyclopedias and wikis brought benefits in reach and speed, and trade-offs in consistency and reliability that readers still weigh today.
Search after the early web
From the 2000s on, search became a routine part of daily life. Results expanded beyond pages to images, news, maps and local information. Mobile phones and voice assistants changed how people ask questions. Social media created huge amounts of content that is shared rather than searched in the traditional sense, and recommendation feeds now compete with search as ways of finding things.
The details of how modern engines work are largely proprietary, and they change often. Public statements about specifics should be treated cautiously.
Recurring themes
A few patterns repeat across this history:
- Scale breaks every method. Boards, file indexes, directories and early crawlers each worked until the amount of material outgrew them.
- Openness cuts both ways. Anyone can publish, which creates variety and also noise, spam and misinformation.
- Credit is shared. Many of the key tools came from groups working in parallel, so “first” claims in this area are usually disputed or qualified.
- Tools reflect their era’s hardware. Dial-up speeds, storage costs and available computing power shaped what each generation of tools could do.
The articles in this cluster take these developments one at a time, from boards and email to markup, directories, ranking and encyclopedias.