buffer overflow happens when operations like stcpy runs beyond the edge of the allocated buffer. We need to find and fix buffer overflows, which causes people who use o

buffer overflow horror stories

• AOL messanger

identifying buffer overflows

Think about whether or not what you are going to do will cause buffer overflows. There are stuff which you shouldn’t do:

• strcpy: which keeps copying
• strcat:
• gets: it keeps taking input forever and forever

https://www.acm.org/code-of-ethics “Design and implement systems that are robustly and usably secure.”

“How do we build well developed AI systems without a bangin’ company”

Two main paradigms

• transfer learning: (pretrain a model, and) faster convergence, better performance
• *monolithic models: (pretrain a model, and) just use the pretrained model

Problems with monolythic models

• Continual development of large language models mostly don’t exist: no incremental updates
• To get better improvements, we throw out the old monolythic model
• Most of the research community can’t participate in their development

New Alternative Paradigm

• A very simple routing layer
• A very large collection of specialist models all from a base model
• Collaborative model development means that a large amount of contributors can band together to contribute to the development of the models

Why

• Specialist models are cheaper and better to train
  • few shot parameter efficient fine tuning is better liu et al
  • few shot fine-tuning is better than few-shot in-context learning
• Specialist models can be communicable, incremental updates to a base model
  • think: PEFT
  • each of the specialist models can only need to update a small percent of the weights
  • think “adapters”: parameter efficient updates

Routing

• task2vec: task embedding for meta learning Achille et al
• efficiently tuned parameters are task embeddings Zhou et al

distinction between MoE

• instead of routing in sub-layer level routing, we are routing at the input level
• we look at the

Novel Tasks (Model Merging)

Tasks can be considered as a composition of skills.

BPE is a common Subword Tokenization scheme.

Training

1. choose two symbols that are most frequency adjacent
2. merge those two symbols as one symbol throughout the text
3. repeat to step \(1\) until we merge \(k\) times

v = set(corpus.characters())
for i in range(k):
    tl, tr = get_most_common_bigram(v)
    tnew = f"{tl}{tr}"
    v.push(tnew)
    corpus.replace((tl,tr), tnew)
return v

Most commonly, BPE is not ran alone: it usually run inside space separation systems. Hence, after each word we usually put a special _ token which delineates end of word.

C was created around 1970s to make Unix tool writing easier. It was meant to the be the simplest, lightest, human readable code on top of assembly.

• C is procedural: you write functions and compose them, instead of defining types
• C++ is procedural (mostly), but you can have objects: you still write functions, and can build objects and call methods
• Java is actual oop

We use C because its fast, highly efficient. C is popular for systems programming, OSes, networking, etc. Lets you work at a lower level to understand/manipulate the underlying systems.

All caches, uses similar memory addressing scheme (i.e. each address can map to one of each device); each “line” of cache is usually 64 bytes. This means that each time you grab something from memory, the 64 bytes surrounding that area of memory will be cached. You are most likely to do this because of iterating through an array.

• L1-D cache: on the CPU, used for staging for registers
• L1-I cache: on the CPU, used for staging for assembly instructions
• L2 cache: L1 staging
• L3 cache: L2 staging
• Main memory: memory

cache locality

temporal locality

“Things I have used recently, I’m likely to use again soon”