Two ways to build a thinking machine
From the 1950s onward, researchers pursuing artificial intelligence tended to follow one of two broad paths. One path was to write down knowledge and reasoning rules explicitly and have the computer manipulate symbols according to them. The other was to build systems that adjust themselves based on examples, so that the behavior is learned rather than programmed. The second path is the subject here. Its history is a useful illustration of how the field has swung between optimism and disappointment, and it is worth remembering that both paths have been influential and often combined.
Early models of neurons
The idea of drawing on the brain for inspiration is old. In 1943, Warren McCulloch and Walter Pitts described a simplified mathematical neuron that adds up inputs and produces an output depending on whether a threshold is reached. They argued that networks of such units could in principle carry out logical operations. In 1949, the psychologist Donald Hebb proposed that connections between neurons strengthen when the neurons are active together, an idea that influenced later learning rules, though real brains are far more complicated than these models.
The perceptron
The most famous early learning machine was the perceptron, developed by Frank Rosenblatt in the late 1950s. It took numerical inputs, multiplied them by adjustable weights, and produced a decision, adjusting the weights when it made mistakes. A hardware version built for image recognition used motors and adjustable resistors, which makes it a curious mix of analog computers and digital ideas. Newspaper coverage of the time predicted much more than the device could deliver, a pattern that would recur.
The perceptron could learn to separate some kinds of patterns, but a single layer of weights has limits. In 1969, Marvin Minsky and Seymour Papert published a mathematical analysis showing certain simple problems that such a device could not solve. The book is widely credited with reducing enthusiasm and funding for this approach through the 1970s. Historians note that the book did not rule out multi-layer networks, and that other factors, such as limited computing power, also played a role.
Quiet years and a revival
Work did not stop. Researchers in several countries continued developing ideas about learning in layered networks. The key practical question was how to adjust the weights of the inner layers when only the final output can be checked against a target. A procedure known as backpropagation provided an answer. Variants had been described by several people earlier, and it became widely known in the mid-1980s after a paper by David Rumelhart, Geoffrey Hinton and Ronald Williams, which demonstrated that multi-layer networks could learn useful internal representations.
This produced a wave of excitement under names such as connectionism. Networks were applied to tasks like recognizing handwritten digits, and early systems of that kind were tested on things like reading numbers on mail and bank checks. Interest also touched speech recognition and early work on automatic translation. Results were promising but limited. Training was slow, data sets were small, and in many tasks other statistical methods performed as well or better. During the 1990s, techniques such as support vector machines attracted much attention, and neural network research became a smaller area for a while.
Data, hardware and deep learning
Several developments combined in the late 2000s and 2010s. Large labeled data sets became available, largely through the web. Graphics processors, designed for video games, turned out to be well suited to the repeated arithmetic that network training requires. And researchers refined training methods so that networks with many layers, the “deep” in deep learning, could be trained more reliably.
A widely cited turning point came in 2012, when a deep convolutional network achieved a notably better result in a large image recognition competition than the methods then in use. Within a few years, similar approaches advanced voice transcription, machine translation and game-playing. In 2017, a new network design called the transformer was introduced, and it became the basis for many of the large language models that attracted wide attention in the 2020s.
What the cycles suggest
Looking at this history, a few patterns stand out.
- Ideas return. Many central techniques existed decades before they became practical. What changed was the supply of data and computing power, along with refinements to the methods.
- Early claims outran evidence. Each wave began with bold forecasts, and each was followed by a period of reduced funding and interest.
- Learned systems have trade-offs. They can handle messy inputs such as images and speech that rules struggle with, but they need large amounts of data, can make confident mistakes, and are often hard to explain.
- Progress is uneven. Success in one narrow task has often been mistaken for a general advance.
It is tempting to see the current period as the final victory of learning-based methods, but the history counsels caution. Whether today’s approaches will continue to improve, hit new limits, or be combined with other methods is uncertain, and past experience suggests that confident predictions deserve some skepticism.