constituents

• \(z \in \mathbb{R}^{n}\) which encodes \(n\) distinct concepts, which is “sparse” \(\norm{z}_{1} < \epsilon_{1}\)
• residual steam \(x \in \mathbb{R}^{m}\) such that \(m \ll n\)

requirements

The Linear Representation Hypothesis states that representation in neural networks can be determined by some:

\begin{equation} \exists F \in \mathbb{R}^{m \times n} \end{equation}

such that \(Fz=x\) for any choice of \(x, z\). That is, neural networks encodes stream-concept mapping linearly.

Importantly, this representation \(F\) also admits a reverse mapping \(G\) which closely recovers the concept, that is:

\begin{equation} \norm{GFz - z} < \epsilon_{2} \end{equation}