The linearity assumption just says that the target (price) can be expressed as a weighted sum of the features (area and age):

\begin{equation} \label{orgd72036a} \mathrm{price} = w_{\mathrm{area}} \cdot \mathrm{area} + w_{\mathrm{age}} \cdot \mathrm{age} + b. \end{equation}

In equation \eqref{orgd72036a}, \(w_{\mathrm{area}}\) and \(w_{\mathrm{age}}\) are called weights, and \(b\) is called a bias (also called an offset or intercept). The weights determine the influence of each feature on our prediction and the bias just says what value the predicted price should take when all of the features take value 0. Even if we will never see any homes with zero area, or that are precisely zero years old, we still need the bias or else we will limit the expressivity of our model. Strictly speaking, equation \eqref{orgd72036a} is an affine transformation of input features, which is characterized by a linear transformation of features via weighted sum, combined with a translation via the added bias.

Given a dataset, our goal is to choose the weights \(\mathbf{w}\) and the bias \(b\) such that on average, the predictions made according to our model best fit the true prices observed in the data. Models whose output prediction is determined by the affine transformation of input features are linear models, where the affine transformation is specified by the chosen weights and bias.

In disciplines where it is common to focus on datasets with just a few features, explicitly expressing models long-form like this is common. In machine learning, we usually work with high-dimensional datasets, so it is more convenient to employ linear algebra notation. When our inputs consist of \(d\) features, we express our prediction \(\hat{y}\) (in general the “hat” symbol denotes estimates) as

\begin{equation} \hat{y} = w_1 x_1 + ... + w_d x_d + b. \end{equation}

Collecting all features into a vector \(\mathbf{x} \in \mathbb{R}^d\) and all weights into a vector \(\mathbf{w} \in \mathbb{R}^d\), we can express our model compactly using a dot product:

\begin{equation} \label{org610e8df} \hat{y} = \mathbf{w}^\top \mathbf{x} + b. \end{equation}

In equation \eqref{org610e8df}, the vector \(\mathbf{x}\) corresponds to features of a single data example. We will often find it convenient to refer to features of our entire dataset of \(n\) examples via the design matrix \(\mathbf{X} \in \mathbb{R}^{n \times d}\). Here, \(\mathbf{X}\) contains one row for every example and one column for every feature.

For a collection of features \(\mathbf{X}\), the predictions \(\hat{\mathbf{y}} \in \mathbb{R}^n\) can be expressed via the matrix-vector product:

\begin{equation} {\hat{\mathbf{y}}} = \mathbf{X} \mathbf{w} + b, \end{equation}

where broadcasting (see 2.1-data-manipulation.html#9dcbe412-db7e-485a-bb3c-d7181f2f7f05) is applied during the summation. Given features of a training dataset \(\mathbf{X}\) and corresponding (known) labels \(\mathbf{y}\), the goal of linear regression is to find the weight vector \(\mathbf{w}\) and the bias term \(b\) that given features of a new data example sampled from the same distribution as \(\mathbf{X}\), the new example’s label will (in expectation) be predicted with the lowest error.

Even if we believe that the best model for predicting \(y\) given \(\mathbf{x}\) is linear, we would not expect to find a real-world dataset of \(n\) examples where \(y^{(i)}\) exactly equals \(\mathbf{w}^\top \mathbf{x}^{(i)}+b\) for all \(1 \leq i \leq n\). For example, whatever instruments we use to observe the features \(\mathbf{X}\) and labels \(\mathbf{y}\) might suffer small amount of measurement error. Thus, even when we are confident that the underlying relationship is linear, we will incorporate a noise term to account for such errors.

Before we can go about searching for the best parameters (or model parameters) \(\mathbf{w}\) and \(b\), we will need two more things: (i) a quality measure for some given model; and (ii) a procedure for updating the model to improve its quality.

Before we start thinking about how to fit data with our model, we need to determine a measure of fitness. The loss function quantifies the distance between the real and predicted value of the target. The loss will usually be a non-negative number where smaller values are better and perfect predictions incur a loss of 0. The most popular loss function in regression problems is the squared error. When our prediction for an example \(i\) is \(\hat{y}^{(i)}\) and the corresponding true label is \(y^{(i)}\), the squared error is given by:

\begin{equation} l^{(i)}(\mathbf{w}, b) = \frac{1}{2} \left(\hat{y}^{(i)} - y^{(i)}\right)^2. \end{equation}

The constant \(\frac{1}{2}\) makes no real difference but will prove notationally convenient, canceling out when we take the derivative of the loss. Since the training dataset is given to us, and thus out of our control, the empirical error is only a function of the model parameters. To make things more concrete, consider the example below where we plot a regression problem for a one-dimensional case as shown in Fig 1.

Note that large differences between estimates \(\hat{y}^{(i)}\) and observations \(y^{(i)}\) lead to even larger contributions to the loss, due to the quadratic dependence. To measure the quality of a model on the entire dataset of \(n\) examples, we simply average (or equivalently, sum) the losses on the training set.

\begin{equation} L(\mathbf{w}, b) =\frac{1}{n}\sum_{i=1}^n l^{(i)}(\mathbf{w}, b) =\frac{1}{n} \sum_{i=1}^n \frac{1}{2}\left(\mathbf{w}^\top \mathbf{x}^{(i)} + b - y^{(i)}\right)^2. \end{equation}

When training the model, we want to find parameters (\(\mathbf{w}^*, b^*\)) that minimize the total loss across all training examples:

\begin{equation} \mathbf{w}^*, b^* = \operatorname*{argmin}_{\mathbf{w}, b}\ L(\mathbf{w}, b). \end{equation}

Linear regression happens to be an unusually simple optimization problem. Unlike most other models that we will encounter in this book, linear regression can be solved analytically by applying a simple formula. To start, we can subsume the bias \(b\) into the parameter \(\mathbf{w}\) by appending a column to the design matrix consisting of all ones. Then our prediction problem is to minimize \(\|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2\). There is just one critical point on the loss surface and it corresponds to the minimum of the loss over the entire domain. Taking the derivative of the loss with respect to \(\mathbf{w}\) and setting it equal to zero yields the analytic (closed-form) solution:

\begin{equation} \mathbf{w}^* = (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf{y}. \end{equation}

While simple problems like linear regression may admit analytic solutions, you should not get used to such good fortune. Although analytic solutions allow for nice mathematical analysis, the requirement of an analytic solution is so restrictive that it would exclude all of deep learning.

Even in cases where we cannot solve the models analytically, it turns out that we can still train models effectively in practice. Moreover, for many tasks, those difficult-to-optimize models turn out to be so much better that figuring out how to train them ends up being well worth the trouble.

The key technique for optimizing nearly any deep learning model, and which we will call upon throughout this book, consists of iteratively reducing the error by updating the parameters in the direction that incrementally lowers the loss function. This algorithm is called gradient descent.

The most naive application of gradient descent consists of taking the derivative of the loss function, which is an average of the losses computed on every single example in the dataset. In practice, this can be extremely slow: we must pass over the entire dataset before making a single update. Thus, we will often settle for sampling a random minibatch of examples every time we need to compute the update, a variant called minibatch stochastic gradient descent.

In each iteration, we first randomly sample a minibatch \(\mathcal{B}\) consisting of a fixed number of training examples. We then compute the derivative (gradient) of the average loss on the minibatch with regard to the model parameters. Finally, we multiply the gradient by a predetermined positive value \(\eta\) and subtract the resulting term from the current parameter values.

We can express the update mathematically as follows (\(\partial\) denotes the partial derivative):

\begin{equation} (\mathbf{w},b) \leftarrow (\mathbf{w},b) - \frac{\eta}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \partial_{(\mathbf{w},b)} l^{(i)}(\mathbf{w},b). \end{equation}

To summarize, steps of the algorithm are the following: (i) we initialize the values of the model parameters, typically at random; (ii) we iteratively sample random minibatches from the data, updating the parameters in the direction of the negative gradient. For quadratic losses and affine transformations, we can write this out explicitly as follows:

\begin{equation} \label{orgad28d27} \begin{aligned} \mathbf{w} &\leftarrow \mathbf{w} - \frac{\eta}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \partial_{\mathbf{w}} l^{(i)}(\mathbf{w}, b) = \mathbf{w} - \frac{\eta}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \mathbf{x}^{(i)} \left(\mathbf{w}^\top \mathbf{x}^{(i)} + b - y^{(i)}\right),\\ b &\leftarrow b - \frac{\eta}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \partial_b l^{(i)}(\mathbf{w}, b) = b - \frac{\eta}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \left(\mathbf{w}^\top \mathbf{x}^{(i)} + b - y^{(i)}\right). \end{aligned} \end{equation}

Note that \(\mathbf{w}\) and \(\mathbf{x}\) are vectors in equation \eqref{orgad28d27}. Here, the more elegant vector notation makes the math much more readable than expressing things in terms of coefficients, say \(w_1, w_2, \ldots, w_d\). The set cardinality \(|\mathcal{B}|\) represents the number of examples in each minibatch (the batch size) and \(\eta\) denotes the learning rate. We emphasize that the values of the batch size and learning rate are manually pre-specified and not typically learned through model training. These parameters that are tunable but not updated in the training loop are called hyperparameters. Hyperparameter tuning is the process by which hyperparameters are chosen, and typically requires that we adjust them based on the results of the training loop as assessed on a separate validation dataset (or validation set).

After training for some predetermined number of iterations (or until some other stopping criteria are met), we record the estimated model parameters, denoted \(\hat{\mathbf{w}}, \hat{b}\). Note that even if our function is truly linear and noiseless, these parameters will not be the exact minimizers of the loss because, although the algorithm converges slowly towards the minimizers it cannot achieve it exactly in a finite number of steps.

Linear regression happens to be a learning problem where there is only one minimum over the entire domain. However, for more complicated models, like deep networks, the loss surfaces contain many minima. Fortunately, for reasons that are not yet fully understood, deep learning practitioners seldom struggle to find parameters that minimize the loss on training sets. The more formidable task is to find parameters that will achieve low loss on data that we have not seen before, a challenge called generalization. We return to these topics throughout the book.

Given the learned linear regression model \(\hat{\mathbf{w}}^\top \mathbf{x} + \hat{b}\), we can now estimate the price of a new house (not contained in the training data) given its area \(x_1\) and age \(x_2\). Estimating targets given features is commonly called prediction or inference.

We will try to stick with prediction because calling this step inference, despite emerging as standard jargon in deep learning, is somewhat of a misnomer. In statistics, inference more often denotes estimating parameters based on a dataset. This misuse of terminology is a common source of confusion when deep learning practitioners talk to statisticians.

Deep learning practitioners like to draw diagrams to visualize what is happening in their models. In fig 2, we depict our linear regression model as a neural network. Note that these diagrams highlight the connectivity pattern such as how each input is connected to the output, but not the values taken by the weights or biases.

For the neural network shown in fig 2, the inputs are \(x_1, \ldots, x_d\), so the number of inputs (or feature dimensionality) in the input layer is \(d\). The output of the network in fig 2 is \(o_1\), so the number of outputs in the output layer is 1. Note that the input values are all given and there is just a single computed neuron. Focusing on where computation takes place, conventionally we do not consider the input layer when counting layers. That is to say, the number of layers for the neural network in fig 2 is 1. We can think of linear regression models As neural networks consisting of just a single artificial neuron, or as single-layer neural networks.

Since for linear regression, every input is connected to every output (in this case there is only one output), we can regard this transformation (the output layer in fig 2 as a fully-connected layer or dense layer. We will talk a lot more about networks composed of such layers in the next chapter.

Since linear regression (invented in 1795) predates computational neuroscience, it might seem anachronistic to describe linear regression as a neural network. To see why linear models were a natural place to begin when the cyberneticists/neurophysiologists Warren McCulloch and Walter Pitts began to develop models of artificial neurons, consider the cartoonish picture of a biological neuron in fig 3, consisting of dendrites (input terminals), the nucleus (CPU), the axon (output wire), and the axon terminals (output terminals), enabling connections to other neurons via synapses.

Information \(x_i\) arriving from other neurons (or environmental sensors such as the retina) is received in the dendrites. In particular, that information is weighted by synaptic weights \(w_i\) determining the effect of the inputs (e.g., activation or inhibition via the product \(x_i w_i\)). The weighted inputs arriving from multiple sources are aggregated in the nucleus as a weighted sum \(y = \sum_i x_i w_i + b\), and this information is then sent for further processing in the axon \(y\), typically after some nonlinear processing via \(\sigma(y)\). From there it either reaches its destination (e.g., a muscle) or is fed into another neuron via its dendrites.

Certainly, the high-level idea that many such units could be cobbled together with the right connectivity and right learning algorithm, to produce far more interesting and complex behavior than any one neuron alone could express owes to our study of real biological neural systems.

At the same time, most research in deep learning today draws little direct inspiration in neuroscience. We invoke Stuart Russell and Peter Norvig who, in their classic AI text book Artificial Intelligence: A Modern Approach [Russell & Norvig, 2016], pointed out that although airplanes might have been inspired by birds, ornithology has not been the primary driver of aeronautics innovation for some centuries. Likewise, inspiration in deep learning these days comes in equal or greater measure from mathematics, statistics, and computer science.