\begin{equation} f(x) \approx f(x_{t-1}) + (x-x_{t-1}) f’(x_{t-1}) + \frac{(x-x_{t-1})^{2}}{2} f’’(x_{t-1}) \end{equation}

Taking a derivative with respect to this, we obtain:

\begin{equation} f’(x_{t-1}) + (x-x_{t-1}) f’’(x_{t-1}) \end{equation}

Solving the update equation for zero, we obtain that:

\begin{equation} x = x_{t-1} - \frac{f’(x_{t-1})}{f’’(x_{t-1})} \end{equation}

This converges quadratically!! to solve for minima/maxima. For solving minima/maxima for gardients:

\begin{equation} x_{t} = x_{t-1} - \qty(\bold{H}_{g})^{-1}\nabla J \end{equation}

for Hessian \(H\) and gradient \(g\).

For finding ZEROS instead of minima:

\begin{equation} x = x_{t-1} - \frac{f(x_{t-1})}{f’(x_{t-1})} \end{equation}

graphical intuition

We want to drop down a tangent line from the standing point until we hit \(0\), and then go to that point on the function and draw a tangent line again.

Namely, we write:

\begin{equation} f’\qty(x^{(j)}) = \frac{f\qty(\qty(x^{(j)}))}{\delta} \end{equation}

for small \(\delta\), Meaning, \(\delta = \frac{f\qty(x^{(j)})}{f’\qty(x^{(j)})}\), which is the gap between each step. Then we update just by this ratio \(x = x - \delta\)

Failure Case

If the function is near an inflection point (i.e. with bad quadratic approximation), you may converge at a point which doesn’t satisfy SONC (i.e. you will get an inflection but not a minima).