See also Roll-out utility if you don’t want to get a vector utility over all states.

solving for the utility of a policy

We can solve for the utility of the policy given the transitions \(T\) and reward \(R\) by solving the following equation

\begin{equation} \bold{U}^{\pi} = (I - \gamma T^{\pi})^{-1} \bold{R}^{\pi} \end{equation}

where \(T\) is an \(|S| \times |S|\) square matrix where each horizontal row is supposed to add up to \(1\) which encodes the probability of transitioning from each horizontal row to the column next rows.

“How quickly can be prototype quickly?” 2-7 days.

Key Sequence

Notation

New Concepts

• Markov Decision Process
• Bellman Equation
  • optimal policy
• value iteration

Important Results / Claims

Questions

Interesting Factoids

229 MDP notation

\(S\) (state), \(A\) (actions), \(P_{(s,a)}\qty(s’) = T\qty(s’ | s,a)\) , \(\gamma\) (discount), \(R\qty(s,a)\).

FUN FACT: discount factors \(< 1\) makes value iteration converge.

\begin{equation} V^{\pi}\qty(s) = \mathbb{E}\qty [R\qty(s_{0},a_{0}) + \gamma R\qty(s_{1}, a_{1}) + \gamma^{2} \dots] \end{equation}

\begin{equation} V^{\pi} \qty(s) = R\qty(s) + \gamma \sum_{s’}^{} P_{s,\pi\qty(s)}\qty(s’) V^{\pi}\qty(s’) \end{equation}

One-Liner

cluster the input, activate a seperate expert group for cluster target.

Motivation

• heterogeneity of input instruction tuning data poses difficulty for MoE
• routing only operates at token level, so can’t deal with sequence level generalization

Novelty

Architecture to enable hierarchical expert routing.

Notable Methods

Mixure of Clustered Experts

Mixture of Clustered Experts

Dual-stage routing mechanism.

1. group the \(M\) experts into groups of \(N\) expert (i.e. \(M = \qty(N, \dots, N)\)
2. k-means clustering the sequence embedding at input
3. given the assigned cluster, only route to the assigned subgroup

Results

• outperforms MoE baselines
• demonstrate expert group specialization