kernel smoothing is a way of smoothing a utility function over continuous state space despite only sampling a discrete set of the states.

\begin{equation} U_{\theta}(s) = \theta^{T} \beta(s) \end{equation}

We multiply a vector \(\theta_{j}\), the utility of being in each state \(s_{j}\) a basis function, which smears, generated for each \(i\) of known discrete state we have:

\begin{equation} \beta_{i}(s) = \frac{k(s, s_{i})}{\sum_{j}^{} k(s, s_{j})} \end{equation}

where, \(k\) is the kernel function, a function inversely proportional to how close the two states are:

** k(s,sj)** is a normalization factor and doesn’t need to be computed at every call.

\begin{equation} k(s, s’) = \max \qty(d(s,s’), \epsilon)^{-1} \end{equation}

where \(d\) is a measure of distance. We clip this function at \(\epsilon\) to prevent inverting \(0\).

## gaussian kernel

There is an alternate state smoothing function which is called gaussian kernel, which allows you to control the degree of smoothing between two states through a parameter \(\sigma\):

\begin{equation} k(s,s’) = \exp \qty( - \frac{d(s,s’)^{2}}{2 \sigma^{2}}) \end{equation}