Houjun Liu

Automatic Differentiation

Forward Accumulation

First, make a computation graph.

Consider \(\ln (ab + \max (a,2))\)

Say we want \(\pdv{f}{a}(3,2)\).

Let’s begin by tracking, left to right, both the value of each node and its derivative.

Layer 1:

  • \(b = 2, \pdv{b}{a} = 0\)
  • \(a = 3, \pdv{a}{a} = 1\)

Layer 2:

  • \(c_1 = a\times b = 6, \pdv{c_1}{a} = b\pdv{a}{a} + a \pdv{a}{b} = 2\)

and so on; until we get to \(c_4\)

Dual Number Method

Dual Number


\begin{equation} a+b \epsilon \end{equation}

Let’s declare:

\begin{equation} \epsilon^{2} = 0 \end{equation}

The standard field operations still apply:

\begin{equation} (a+b\epsilon) + (c+d\epsilon) = (a+c) + (b+d) \epsilon \end{equation}

The Method

We can write down a usual Taylor expansion:

\begin{equation} f(a+b\epsilon) = \sum_{k=0}^{\infty} \frac{f^{(k)}}{k!} (a+b \epsilon - a)^{k} \end{equation}


\begin{equation} f(a+1\epsilon) = f(a) + f’(a) \epsilon \end{equation}

This means that we can use Dual Numbers to directly compute derivatives.