TPU Topo

HBM renders into two memories: vmem, where matricies go, and smem, where scalars and conditional logic goes.

Recall that math happens in vmem.

OG Pallas

1. memory pipelining: HBM and VMEM mapping between
2. custom prefetch: PrefetchScalarGridSpec—enable runtime dependent scalar transform
3. input-output aliasing: recycle input and output buffers for in-place updates
4. passing HBM buffers: for collectives where there’s no real math etc., you can just pass references to HBM

pl.kernel

A brand new API. Also need multi-core support like Megacore stuff.

Introduction

Recent advances of language models (LMs) introduced the possibility of in-context, few or zero-shot reasoning (Brown et al. 2020) using LMs without much or any fine tuning.

Yet, classically, LM decoding takes place in a left-to-right fashion, auto-regressively resolving one token at a time by sampling from the output distribution of possible next words without multi-step planning.

Work in LM agents have taken steps to solve more complex problems that would typically require multi-step reasoning even while using this direct decoding approach. The simplest idea, named “chain-of-thoughts” (CoT), involves forcing the LM at decode time to begin the decoding process with natural language reasoning about its actions (Wei et al. 2022). The method has contributed to the creation of powerful language agents (Yao, Zhao, et al. 2023) that can reason about complex actions.

Projects Index is a index that contains a list of almost all projects for which I have ever worked on. Major categories are highlighted from chapter titles.

Research Projects

I end up doing a lot of research these days, and so have isolated that to a different, academic homepage.

For a list of my recent research, please head to my academic homepage. For concision they are NOT repeated here.

Large-Scale Endeavors

Condution

An open-source task management app. Website.

Dual influence framework:

1. requires political involvement
2. requires diverse media diet

Proposal: based on feedback on TIC-focus on one case study and isolate it well

Quotes

Social media as a means of exposure to the modern world

“Daniel Lerner (1958) saw mass media as the main catalyst for social change. Lerner argued that media exposed people who possess traditional values to the “modern” world, and that exposure in turn produced a desire to live in it.” [⁨Díez⁩ and ⁨Dion⁩, 2022, p. 467]

Here’s a general form:

\begin{equation} a\dv[2]{x}{t} + b \dv{x}{t} + cx = f(t) \end{equation}

see:

• solving homogeneous constant coefficient higher-order differential equations
• and more generally, using matrix exponentiation, solving homogeneous higher-order differential equations

solving homogeneous higher-order differential equations

This problem because easier if the right side is \(0\).

\begin{equation} a\dv[2]{x}{t} + b \dv{x}{t} + cx = 0 \end{equation}

The general goal to solve in this case is to make this a system of First-Order Differential Equations.