We begin by addressing the calculation of derivatives, a crucial step in nearly all deep learning optimization algorithms. In deep learning, we typically choose loss functions that are differentiable with respect to our model’s parameters. Put simply, this means that for each parameter, we can determine how rapidly the loss would increase or decrease, were we to increase or decrease that parameter by an infinitesimally small amount.

Suppose that we have a function \(f: \mathbb{R} \rightarrow \mathbb{R}\), whose input and output are both scalars. The derivative of \(f\) is defined as

if this limit exists. If \(f'(a)\) exists, \(f\) is said to be differentiable at \(a\). If \(f\) is differentiable at every number of an interval, then this function is differentiable on this interval. We can interpret the derivative :\(f'(x)\) in \eqref{org56761fc} as the instantaneous rate of change of \(f(x)\) with respect to \(x\). The so-called instantaneous rate of change is based on the variation \(h\) in \(x\), which approaches \(0\).

To illustrate derivatives, let us experiment with an example. Define \(u = f(x) = 3x^2-4x\).

*Note: We will be using Double in this section to avoid incorrect results since Double provides more decimal precision. Generally though, we would use Float as deep learning frameworks by default use Fault.*

(ns clj-d2l.calculus
  (:require [clj-djl.ndarray :as nd]
            [clj-chart.chart :as chart]
            [clj-chart.plot :as plot]
            [clj-d2l.core :as d2l]
            [clojure.java.io :as io]))

(defn f [x]
  (- (* 3 (Math/pow x 2)) (* 4 x)))

By setting \(x=1\) and letting \(h\) approach \(0\), the numerical result of \(\frac{f(x+h) - f(x)}{h}\) in \eqref{org56761fc} approaches \(2\). Though this experiment is not a mathematical proof, we will see later that the derivative \(u'\) is \(2\) when \(x=1\).

(defn numerical-lim [f x h]
  (/ (- (f (+ x h)) (f x)) h))

(->> (map #(/ 0.1 (Math/pow 10 %)) (range 5))
     (map (fn [h] [h (numerical-lim f 1 h)]))
     (map #(println "h = " (% 0) ", numerical limit = " (% 1)))
     (dorun))

h =  0.1 , numerical limit =  2.3000000000000043
h =  0.01 , numerical limit =  2.0299999999999763
h =  0.001 , numerical limit =  2.002999999999311
h =  1.0E-4 , numerical limit =  2.0002999999979565
h =  1.0E-5 , numerical limit =  2.0000300000155846

Let us familiarize ourselves with a few equivalent notations for derivatives. Given \(y = f(x)\), where \(x\) and \(y\) are the independent variable and the dependent variable of the function \(f\), respectively. The following expressions are equivalent:

where symbols \(\frac{d}{dx}\) and \(D\) are differentiation operators that indicate operation of differentiation. We can use the following rules to differentiate common functions:

• \(DC = 0\) (\(C\) is a constant),
• \(Dx^n = nx^{n-1}\) (the power rule, \(n\) is any real number),
• \(De^x = e^x\),
• \(D\ln(x) = 1/x.\)

To differentiate a function that is formed from a few simpler functions such as the above common functions, the following rules can be handy for us. Suppose that functions \(f\) and \(g\) are both differentiable and \(C\) is a constant, we have the constant multiple rule

the sum rule

and the quotient rule

Now we can apply a few of the above rules to find \(u' = f'(x) = 3 \frac{d}{dx} x^2-4\frac{d}{dx}x = 6x-4\). Thus, by setting \(x = 1\), we have \(u' = 2\): this is supported by our earlier experiment in this section where the numerical result approaches \(2\). This derivative is also the slope of the tangent line to the curve \(u = f(x)\) when \(x = 1\).

To visualize such an interpretation of derivatives, we will use xchart. a simple plotting library.

We define plot-lines which will take as input three arrays. The first array will be the data in the x axis and the next two arrays will contain the two functions that we want to plot in the y axis. In addition to this data, the function requires us to specify the name of the two lines we will be plotting, the label of both axes, and the width and height of the figure. This function or a modified version of it, will allow us to plot multiple curves succinctly since we will need to visualize many curves throughout the book.

Now we can plot the function \(u = f(x)\) and its tangent line \(y = 2x - 3\) at \(x=1\), where the coefficient \(2\) is the slope of the tangent line.

(let [x (range 0 3 1/32)
      y1 (map f x)
      y2 (map #(- (* 2 %) 3) x)
      chart (chart/line {:title "tangent line (x=1)"
                         :series [{:name "y1"
                                   :xs x
                                   :ys y1}
                                  {:name "y2"
                                   :xs x
                                   :ys y2}]})]
  (plot/store! chart nil "notes/figures/tangent_line.svg"))

So far we have dealt with the differentiation of functions of just one variable. In deep learning, functions often depend on many variables. Thus, we need to extend the ideas of differentiation to these multivariate functions.

Let \(y = f(x_1, x_2, \ldots, x_n)\) be a function with \(n\) variables. The partial derivative of \(y\) with respect to its \(i^\mathrm{th}\) parameter \(x_i\) is

To calculate \(\frac{\partial y}{\partial x_i}\), we can simply treat \(x_1, \ldots, x_{i-1}, x_{i+1}, \ldots, x_n\) as constants and calculate the derivative of \(y\) with respect to \(x_i\). For notation of partial derivatives, the following are equivalent:

We can concatenate partial derivatives of a multivariate function with respect to all its variables to obtain the gradient vector of the function. Suppose that the input of function \(f: \mathbb{R}^n \rightarrow \mathbb{R}\) is an \(n\)-dimensional vector \(\mathbf{x} = [x_1, x_2, \ldots, x_n]^\top\) and the output is a scalar. The gradient of the function \(f(\mathbf{x})\) with respect to \(\mathbf{x}\) is a vector of \(n\) partial derivatives:

where \(\nabla_{\mathbf{x}} f(\mathbf{x})\) is often replaced by \(\nabla f(\mathbf{x})\) when there is no ambiguity.

Let \(\mathbf{x}\) be an \(n\)-dimensional vector, the following rules are often used when differentiating multivariate functions:

• For all \(\mathbf{A} \in \mathbb{R}^{m \times n}\), \(\nabla_{\mathbf{x}} \mathbf{A} \mathbf{x} = \mathbf{A}^\top\),
• For all \(\mathbf{A} \in \mathbb{R}^{n \times m}\), \(\nabla_{\mathbf{x}} \mathbf{x}^\top \mathbf{A} = \mathbf{A}\),
• For all \(\mathbf{A} \in \mathbb{R}^{n \times n}\), \(\nabla_{\mathbf{x}} \mathbf{x}^\top \mathbf{A} \mathbf{x} = (\mathbf{A} + \mathbf{A}^\top)\mathbf{x}\),
• \(\nabla_{\mathbf{x}} \|\mathbf{x} \|^2 = \nabla_{\mathbf{x}} \mathbf{x}^\top \mathbf{x} = 2\mathbf{x}\).

Similarly, for any matrix \(\mathbf{X}\), we have \(\nabla_{\mathbf{X}} \|\mathbf{X} \|_F^2 = 2\mathbf{X}\). As we will see later, gradients are useful for designing optimization algorithms in deep learning.

However, such gradients can be hard to find. This is because multivariate functions in deep learning are often composite, so we may not apply any of the aforementioned rules to differentiate these functions. Fortunately, the chain rule enables us to differentiate composite functions.

Let us first consider functions of a single variable. Suppose that functions \(y=f(u)\) and \(u=g(x)\) are both differentiable, then the chain rule states that

Now let us turn our attention to a more general scenario where functions have an arbitrary number of variables. Suppose that the differentiable function \(y\) has variables \(u_1, u_2, \ldots, u_m\), where each differentiable function \(u_i\) has variables \(x_1, x_2, \ldots, x_n\). Note that \(y\) is a function of \(x_1, x_2, \ldots, x_n\). Then the chain rule gives

for any \(i = 1, 2, \ldots, n\).

• Differential calculus and integral calculus are two branches of calculus, where the former can be applied to the ubiquitous optimization problems in deep learning.
• A derivative can be interpreted as the instantaneous rate of change of a function with respect to its variable. It is also the slope of the tangent line to the curve of the function.
• A gradient is a vector whose components are the partial derivatives of a multivariate function with respect to all its variables.
• The chain rule enables us to differentiate composite functions.

1. Plot the function \(y = f(x) = x^3 - \frac{1}{x}\) and its tangent line when \(x = 1\).

   (defn f2 [x]
     (- (Math/pow x 3) (/ 1 x)))
   

   (numerical-lim f2 1 0.000001)
   

   4.000001999737712
   

   (- (* 4 1) (f2 1))
   

   4.0
   

   (let [x (map #(* 1/100 %) (range 10 300))
         y1 (map f2 x)
         y2 (map #(- (* 4 %) 4) x)
         chart (chart/line {:title "tangent line"
                            :series [{:name "y1"
                                      :xs x
                                      :ys y1}
                                     {:name "y2"
                                      :xs x
                                      :ys y2}]})]
     (plot/store! chart nil "notes/figures/exercise-2.4-1.svg"))
   

   

2. Find the gradient of the function \(f(\mathbf{x}) = 3x_1^2 + 5e^{x_2}\).

   \begin{equation} \label{orge7b9b0d} \nabla_\mathbf{x} f(\mathbf{x}) = \bigg[\frac{\partial f(\mathbf{x})}{\partial x_1}, \frac{\partial f(\mathbf{x})}{\partial x_2}\bigg]^\top = \bigg[6x_1, 5e^{x_2}\bigg]^\top \end{equation}
3. What is the gradient of the function \(f(\mathbf{x}) = \|\mathbf{x}\|_2\)?
4. Can you write out the chain rule for the case where \(u = f(x, y, z)\) and \(x = x(a, b)\), \(y = y(a, b)\), and \(z = z(a, b)\)?