kernel smoothing is a way of smoothing a utility function over continuous state space despite only sampling a discrete set of the states.
\begin{equation} U_{\theta}(s) = \theta^{T} \beta(s) \end{equation}
We multiply a vector \(\theta_{j}\), the utility of being in each state \(s_{j}\) a basis function, which smears, generated for each \(i\) of known discrete state we have:
\begin{equation} \beta_{i}(s) = \frac{k(s, s_{i})}{\sum_{j}^{} k(s, s_{j})} \end{equation}
where, \(k\) is the kernel function, a function inversely proportional to how close the two states are:
k(s,sj) is a normalization factor and doesn’t need to be computed at every call.
\begin{equation} k(s, s’) = \max \qty(d(s,s’), \epsilon)^{-1} \end{equation}
where \(d\) is a measure of distance. We clip this function at \(\epsilon\) to prevent inverting \(0\).
gaussian kernel
There is an alternate state smoothing function which is called gaussian kernel, which allows you to control the degree of smoothing between two states through a parameter \(\sigma\):
\begin{equation} k(s,s’) = \exp \qty( - \frac{d(s,s’)^{2}}{2 \sigma^{2}}) \end{equation}