Web Directories and Human-Curated Lists

A list as a first answer

When the World Wide Web began to grow in the early 1990s, there were few ways to find anything on it. The simplest solution was also the oldest: write a list. Researchers and enthusiasts kept pages of links to interesting sites and shared them. Some of these “what’s new” pages and subject lists were maintained by individuals, and some by institutions such as libraries and universities.

Lists had a clear advantage. A person who had visited a site could describe it and place it near similar ones. Readers could browse by topic without knowing what to type.

From personal lists to directories

As more sites appeared, a single person could no longer keep up, and the lists became organized into categories and subcategories. The result was a web directory: a hierarchy of topics such as arts, business, computers, science and recreation, with links beneath each.

The best-known example from the mid-1990s began as a hobby list by two graduate students at Stanford University and soon became a company that offered a directory, a search box, news and other services. Other directories were launched by existing companies and by libraries, and some universities maintained virtual libraries organized by subject experts.

A different model arrived in 1998 with a directory built by volunteers. Instead of a company’s paid staff, thousands of volunteer editors each looked after a category. The project was available to anyone, and its data was reused by other websites. It was known by the name DMOZ, and it continued for many years before the project closed in 2017.

How a directory worked

The process was fairly consistent:

1. A site owner submitted a page along with a short title and description.
2. An editor reviewed the submission and decided whether it fit a category.
3. If accepted, it was placed in the hierarchy, sometimes alongside a brief editorial note.
4. Editors might later check whether the link still worked.

Some services charged a fee for faster review, which created criticism about whether the listings were truly independent. Others accepted only free submissions and could take weeks or months to review them.

Strengths

Directories had real virtues, particularly for broad or unfamiliar topics:

• Quality control. Editors could exclude spam and low-effort pages.
• Context. Categories showed related sites, which helped a beginner learn the shape of a subject.
• Predictability. The same path led to the same list each time.
• No query skills required. Browsing is easier than choosing the right words.

Libraries and educators valued them because they resembled catalogs, a familiar way of organizing knowledge.

Weaknesses

The same features limited them.

• Scale. The web grew much faster than any group of editors could review. A directory could list a fraction of existing sites, and categories became crowded.
• Staleness. Sites moved or vanished, leaving dead links, and keeping every entry current was labor-intensive.
• Subjectivity. Deciding where something belongs, and whether it deserves to be listed at all, involves judgment. Sites that fit several categories or none were awkward.
• Specific questions. A directory suited “find sites about gardening” but not “find the page that explains this one detail”.
• Commercial pressure. Paid placements and the difficulty of getting listed led to complaints about fairness.

Why search overtook them

Crawler-based search engines did not need submissions and could index far more pages. Once result ordering improved through link-based ranking, a short query often reached a useful page faster than clicking through three levels of categories. Many portals began to put the search box first and the directory second, and in time some dropped directory listings altogether, as the larger services shifted toward search and advertising.

Users’ habits changed as well. Instead of asking “which category is this in”, people started asking “what words would this page contain”. This is one of the quiet but important shifts in how the web is used.

What survived

Human curation did not disappear. It moved. Several forms of it are still common:

• Curated collections. Librarians, universities and subject specialists publish selected resource lists, which are valued where reliability matters.
• Social bookmarking and link-sharing sites. Users submit and vote on links, so a crowd acts as editor, though with its own biases.
• Reviews and “best of” lists. Many publishers rely on editors to select and recommend.
• Online encyclopedias and reference sites. Volunteer communities organize knowledge into categories and cross-links, a theme explored in the history of wikis.

Modern search engines also use human input in the background, for example in testing result quality, although the details are not public.

A useful comparison

Directories and search represent two ways of organizing information. One begins with a classification system and files things into it. The other begins with the raw material and lets users ask for what they need. Directories are easier to trust but hard to scale, and search scales well but needs good ranking and constant defense against spam. The history of the web shows each approach compensating for the other’s weaknesses, and the mix of both is still visible in how people find things today.