constituents

• \(z \in \mathbb{R}^{n}\) which encodes \(n\) distinct concepts, which is “sparse” \(\norm{z}_{1} < \epsilon_{1}\)
• residual steam \(x \in \mathbb{R}^{m}\) such that \(m \ll n\)

requirements

The Linear Representation Hypothesis states that representation in neural networks can be determined by some:

\begin{equation} \exists F \in \mathbb{R}^{m \times n} \end{equation}

such that \(Fz=x\) for any choice of \(x, z\). That is, neural networks encodes stream-concept mapping linearly.

Importantly, this representation \(F\) also admits a reverse mapping \(G\) which closely recovers the concept, that is:

Hidden

no

Random Variable

random variables takes on different values with different probabilities. Each value a random variable take on is an event.

For instance, here’s a random variable representing a die: \(X\). It can takes on the following values, with the following probabilities:

\begin{align} P(X=1) = \frac{1}{6}\\ P(X=2) = \frac{1}{6}\\ \dots \\ P(X=6) = \frac{1}{6} \end{align}

where each assignment \(X=k\) is what we refer to above as an event.

The set of assignments of a random variable and their associated probability is called a distribution: distributions “assigns probabilities to outcomes.” When we say a certain random variable \(X\) is “distributed” following a distribution \(D\), we say \(X \sim D\). Semantically, we say \(X\) is a \(D\) random variable.

After a conversation with an LM https://chatgpt.com/share/697143db-c3e0-8000-b56c-07cf7ca43795 the following proposal was generated.