From hand-written rules to machine learning
Artificial intelligence has developed through several different approaches. The earliest major school was symbolism, also known as logicism, psychologism, or computerism. Its basic idea was straightforward: knowledge could be written as a collection of logical rules. If A happens, then B follows.
This approach had a low barrier to entry and could be used to build systems such as expert systems and stock-prediction programs. Its limitation was equally clear: because people supplied the rules, the system's upper limit was tied to the knowledge and foresight of its designers. It could not gradually accumulate experience and improve in the flexible way a human can. Symbolic AI is therefore largely regarded as an abandoned direction of AI development.
The mainstream approach today is machine learning, which developed from connectionism and attempts to imitate, in a simplified way, how the human brain learns. A useful analogy is training a dog: reward the desired behavior, penalize the wrong behavior, and over time the dog becomes better at responding.
In machine learning, three questions correspond to three central components:
- Where does the dog come from? The model structure.
- How is the dog rewarded or punished? The loss function.
- How does the dog become better? The training process.
The perceptron: adding weights to rules
The perceptron can be understood as a development built on symbolic reasoning. Instead of treating every rule as simply true or false, it assigns numerical weights to different features and combines them through calculation.
Suppose we want to identify an apple. We might assign scores like these:
- Red: +1
- Yellow: +0.5
- Black: -1
- Small shape: +1
- Large shape: -1
- Sweet taste: +1
- Sour taste: -1
After adding the scores of the observed features, the system classifies the object as an apple whenever the total is at least 3. For example:
red + small shape + sweet taste >= 3
An apple with both red and yellow coloring could also qualify:
red + yellow + small shape + sweet taste >= 3
This kind of weighted threshold calculation is a perceptron.
However, a single perceptron cannot perform every type of classification. A classic example is the XOR operation, in which equal inputs produce 0 and different inputs produce 1:
<table> <thead> <tr> <th>x</th> <th>y</th> <th>x⊕y</th> </tr> </thead> <tbody> <tr> <td>0</td> <td>0</td> <td>0</td> </tr> <tr> <td>0</td> <td>1</td> <td>1</td> </tr> <tr> <td>1</td> <td>0</td> <td>1</td> </tr> <tr> <td>1</td> <td>1</td> <td>0</td> </tr> </tbody> </table>
A formula such as
0.7x + 0.7y - 0.5 > 0
represents a two-variable linear equation. Geometrically, it produces a straight line. One straight line cannot separate the two regions required by XOR, so a single perceptron is unable to recognize the pattern.
The solution is to feed the result through another layer. This produces a multilayer perceptron. In the following example, two intermediate units first identify whether one input is greater than the other, and a final unit combines those results:
1x + (-1y) - 0 > 0 activates X1 as 1; otherwise it is 0
1y + (-1x) - 0 > 0 activates Y1 as 1; otherwise it is 0
1X1 + 1Y1 - 0 > 0 produces 1; otherwise it is 0
The -0 serves as a threshold and can be ignored for the moment. With this extra layer, XOR can be recognized easily.
Why depth and convolution matter
There is no requirement that a neural network contain only two intermediate units or two layers. Even if a network starts with just two inputs, increasing its width and depth gives it much greater expressive power. With sufficient size, a multilayer perceptron can, in theory, approximate any function.
Consider recognizing handwritten digits in images. The neurons in the first layer could represent individual pixels. Later neurons could combine those pixels to detect basic elements such as horizontal lines, vertical lines, diagonal strokes, and dots. Further layers could combine those elements into complete digit patterns.
A problem soon appears: if every neuron in one layer is connected to every neuron in the previous layer, the number of parameters and calculations grows rapidly. Researchers noticed that neurons in the visual systems of animals do not need to connect to every neuron in the preceding layer. They respond mainly to local regions, and their connections follow a similar repeated structure.
This observation led to the convolutional neural network, or CNN. By focusing on local connections, a CNN can reduce the number of parameters and the amount of computation while improving performance on tasks such as image recognition.
Residual connections
Making a convolutional network deeper can improve its representational ability, but very deep networks are difficult to train. One response was to introduce shortcut connections that allow information to skip one or more layers. This design is known as a residual network, or ResNet.
A more general arrangement can connect layers through many such shortcuts, producing a DenseNet-style structure. These connections help information and gradients move through a deep network more easily.
Residual network
DenseNet
Fitting data with a loss function
At a broad level, intelligence in machine learning means being given a collection of points and finding a function that captures the relationship among them—in other words, finding a pattern.
Real-world outputs are often not simply right or wrong. Imagine giving a model a picture of a young tree and asking it to estimate the tree's age. The answer may be completely correct, nearly correct, roughly correct, or far off. Each prediction has a measurable distance from the correct answer.
The data points representing the desired answers are the model's training targets. Finding a rule that follows those points means constructing a fitting function.
A loss function measures how good the fitting function is. It compares the predicted and correct values for all training examples, calculates their deviations, and combines them—one common method is to square the deviations and add them together. The lower the loss, the more accurately the function fits the data.
The difficulty is that a modern model may contain an enormous number of adjustable parameters. In the XOR example above, there are nine parameters—the constants appearing in the formulas—that can be changed. GPT-3, by comparison, has 175 billion parameters. In mathematics, finding good values in such a complicated parameter space is a non-convex optimization problem, which is extremely difficult to solve directly.
A larger and more complex model may appear capable of powerful behavior, but that does not automatically reveal a set of parameters that produces the desired behavior. The practical strategy is to adjust the parameters step by step. We can temporarily assume that all other parameters are already suitable, vary one parameter, observe how the loss changes, and try to move toward the lowest point.
The training data is often discrete rather than continuous. There may be gaps between known points, making the complete shape of the loss function impossible to see. Even so, we can estimate the local direction: is the function increasing or decreasing at a particular point? This local slope is the derivative.
At each step, the derivative tells us which way to move the parameter and how the loss is changing. Repeating the process should eventually bring the parameter toward a low point.
The same idea extends from one parameter to two. If two parameters are inputs and the loss is the output, the result is a two-variable function. It can be visualized as a three-dimensional surface called a loss surface.
To find the effect of parameter K1, we can hold K2 fixed and change only K1. This gives the partial derivative of the loss with respect to K1. Geometrically, taking a vertical cross-section through the surface produces a curve, whose slope describes that partial derivative.
The same process can be used for K2 while holding K1 constant.
Combining the two partial derivatives gives the gradient, which is the multi-dimensional equivalent of a derivative. At each point on the surface, the gradient indicates the direction of steepest increase. Since an increase means a larger loss, training moves in the opposite direction. That is gradient descent.
If this method can adjust two parameters at once, it can also be extended to any number of parameters. This is the basic mechanism used to tune the vast parameter spaces of neural networks.
Backpropagation: calculating the gradient efficiently
The remaining question is how to calculate gradients efficiently in a complex neural network. The answer is the chain rule, which handles nested functions.
Imagine three gears whose rotation angles are represented by x, g(x), and f(g(x)). A derivative describes the ratio at which motion is transmitted between two gears.
If we want to know how quickly a smaller gear changes when a particular gear moves, we can multiply the two transmission ratios together. The same principle applies to a sequence of mathematical functions: the derivatives along the chain are multiplied.
Every parameter in a neural network affects the final loss through a series of composed functions. If we want to know how changing the output should alter a particular parameter, we can work backward through the network and multiply the derivatives at each stage. The result is the gradient of the loss with respect to that parameter.
This process is backpropagation: an algorithm for efficiently calculating gradients in complex neural networks. Once those gradients are available, gradient descent can use them to update the parameters and gradually reduce the loss.