Forward Accumulation
First, make a computation graph.
Consider \(\ln (ab + \max (a,2))\)
Say we want \(\pdv{f}{a}(3,2)\).
Let’s begin by tracking, left to right, both the value of each node and its derivative.
Layer 1:
- \(b = 2, \pdv{b}{a} = 0\)
- \(a = 3, \pdv{a}{a} = 1\)
Layer 2:
- \(c_1 = a\times b = 6, \pdv{c_1}{a} = b\pdv{a}{a} + a \pdv{a}{b} = 2\)
and so on; until we get to \(c_4\)
Dual Number Method
Dual Number
Consider:
\begin{equation} a+b \epsilon \end{equation}
Let’s declare:
\begin{equation} \epsilon^{2} = 0 \end{equation}
The standard field operations still apply:
\begin{equation} (a+b\epsilon) + (c+d\epsilon) = (a+c) + (b+d) \epsilon \end{equation}
The Method
We can write down a usual Taylor expansion:
\begin{equation} f(a+b\epsilon) = \sum_{k=0}^{\infty} \frac{f^{(k)}}{k!} (a+b \epsilon - a)^{k} \end{equation}
IMPORTANTLY:
\begin{equation} f(a+1\epsilon) = f(a) + f’(a) \epsilon \end{equation}
This means that we can use Dual Numbers to directly compute derivatives.