Big Idea

Motivation: Touring Test

Point of the Turing test: we can use language to get the underlying thought and the underlying cognition. However, language IS NOT thought.

Language is not thought

Good at language != good at thought => public speaking settings

Bad at language != bad at thought => language models?

LLM eval should separate language and thought

1. formal vs functional linguistic confidence (good at speaking and speaking is useful)
2. generalized world knowledge

Detour and motivation: cognitive science

• language centre in brain is specific to a location, and changes in language doesn’t change what region gets activated
• language shows little/no response when we are thinking of cognitively challenging tasks lik emaffs

Key examples: aphasics can still think. So each skill is in a separate brain-place.

distributed algorithm is a type of algorithm that can be distributed across many modules.

There are a few core areas of research:

failure-proofing nodes is a distributed algorithm

• What if one processor fails?

communication in a distributed algorithm

• What if communication between processors fails?
• What if timing fails?

atomicity

atomicity is a property of distributed algorithm where, for a set of steps, a processor can only do one or all of the steps. i.e.: if you are asking a node to do something, it can either do all of the thing or be able to roll back as if the entire thing didn’t happen.

distributive harm is a harm where a system extends, withhold opportunities given a specific group of people

St. George’s hospital: overweighting

1. observing the weights
2. observing the input, consider whether or not input has sensitive features