Like a sound you hear That lingers in your ear But you can’t forget From sundown to sunset

It’s all in the air You hear it everywhere No matter what you do It’s gonna grab a hold on you California soul

\begin{equation} x_1^{(j)} = x_1^{(j-1)} + Attn\qty(x_{k}^{(j-1)}, \forall k) \end{equation}

\begin{equation} At_{x_{1}^{(j-1)}} = \text{softmax}\qty(\frac{q_{1} k_{j}, \forall j}{\sqrt{d_{\ \text{model}}}}) v_{j} \end{equation}

\begin{equation} At_{x_{1}^{(j-1)}} = \text{softmax}_{\text{top-k cliff}}\qty(\frac{q_{1} k_{j}, \forall j}{\sqrt{d_{\ \text{model}}}}) v_{j} \end{equation}

a quasiconvex function \(f: \mathbb{R}^{n} \to \mathbb{R}\) is quasiconvex if \(\text{dom } f\) is a convex set and the sublevel sets:

\begin{equation} S_{\alpha} = \qty {x \in \text{dom } f \mid f\qty(x) \leq \alpha } \end{equation}

are convex for all \(\alpha\). These functions are also called unimodal functions.

properties of quasiconvex functions

modified Jensen’s Inequality

\begin{equation} 0 \leq \theta \leq 1 \implies f\qty(\theta x + \qty(1-\theta)y) \leq\max\qty{f\qty(y), f\qty(x)} \end{equation}

l

first-order condition

differential \(f\) with convex domain is quasiconvex IFF

\begin{align} \min_{x}\quad & f_{0}\qty(x) \\ \textrm{s.t.} \quad & f_{i}\qty(x) \leq 0, i = 1 \dots m\\ &Ax = b \end{align}

Where \(f_{0}\) is quasiconvex, and \(f_{i\geq 1}\) is convex.

solution methods

convex representation of sublevel sets

if \(f_{0}\) is quasiconvex, then there exists a family of functions \(\phi_{t}\) for which:

\begin{equation} f_{0} \leq t \iff \phi_{t}\qty(x) \leq 0 \end{equation}

where \(\phi_{t}\) is convex for fixed \(t\).

bisection method for quasiconvex optimization

We can cast all quasiconvex problems into a binary search over \(t\). Solve the following convex feasibility problem:

Key Sequence

Notation

New Concepts

• Strong and Weak Duality
• Geometric Interperattion of the Dual
  • Complementary Slackness
  • KKT Conditions
• preturbation analysis
• dual transformation
• theorem of alternatives

Important Results / Claims

Questions

Interesting Factoids

Key Sequence

Notation

New Concepts

• Duality for Feasible Problems
• Approximation and Fitting
  • Huber Penalty Function

Important Results / Claims

• Farkas’ Lemma

Questions

Interesting Factoids