Self Driving Level 3

You may or may not have heard of the word “level” being used to describe degrees of capability for self-driving cars. As each level increases, the amount of human intervention decreases. Fascinatingly though, consensus is that each level is not necessarily safer than the level that came before it. Specifically, level 3--where the car is capable of driving itself, say, 99% of the time, but 1% of the time it requires human control to handle extreme cases--is more dangerous than level 2--where, even though the automated system can completely control the vehicle, human attention is required 100% of the time.

It’s easy to imagine why this is the case. If 99% of the time everything takes care of itself, the temptation will be to spend that 99% of the time doing something else, like texting, watching a movie, or eating lunch. If most of the time, you don’t have to pay attention, the likelihood is that you won’t. And now the question is, for that 1%, will you be ready? The way that human brains work, it’s entirely possible you won’t.

(Now, there are many debates to be had about level 3 for self-driving, but that’s not what I wanted to talk about here.)

Human Psychology and chat GPT

I’m definitely not here to define “level” equivalents for cognition, but I bring up the self-driving levels discussion because it illuminates an interesting interaction between technological improvements and human psychology. Even though the technological capabilities are strictly increasing, the benefit to humans may not. In fact, it’s entirely possible that (given the number of cars on the road) that level 3 self-driving is straight up more dangerous than no self-driving at all. My opinion of the current hype around large language models, like chat GPT, is that we are in another one of these troughs.

Cognitive Capabilities and Heuristics

As humans, we bumble through life with a set of heuristics to judge the cognitive capabilities of the people we encounter. For example, age: we assume, and largely correct, that the older you are, the more average-adult-human-like the person thinks. A teenager will think more like an adult than an eight year old, than a three year old, and so on.

Within these age-ranges, there are huge amounts of correlational heuristics. A teenager has better vocabulary than an eight year old, but also more life experience, emotional intelligence, and self-determination. Even in the main cognitive variance we encounter, which is age based, these things are all extremely tightly correlated.

We also have heuristics for judging the “cognitive” capabilities of the computers in our lives. For example, an automated phone answering tree can respond to it, but at this point we understand it has fairly limited capabilities. The slightly fancier phone answering can take less limited inputs, but its canned responses tell us there’s only so much that it “knows”.

The Unevenness of Large Language Models

Large languages play poorly with these heuristics. Specifically, because, as language models, they are very good at producing “human-like” outputs. And so far, we’ve spent our whole-lives using the language that people use as proxies for their other cognitive capabilities.

The real advancement of chat GPT (even vs things like GPT 3+) is its ability to respond in a way that feels familiar and human-like to us. Compare chat GPT to the phone dialogue tree mentioned earlier. Like the slightly fancier system, it can take in free form inputs and make sense of them. Unlike the phone answering trees though, it can also respond in a way that is flexible and human-like. Because of this, our first reaction is to deploy all of the heuristics we’ve built interacting with both humans and other automated systems and make assumptions about its other cognitive capabilities.

But chat GPT isn’t a human. It’s a language model. It models language. And the heuristics we have don’t apply. Chat GPT can output language like a journalist, a professor, a poet, a songwriter might, but that doesn’t mean anything about its other cognitive capabilities.

Just as I’m not going to try to create a level-system for cognition, I’m also not going to enumerate what the important cognitive capabilities are. But there are several discrepancies which stand out to me as important.

Language is not Real World Experience

There’s more to language than vocabulary and syntax. Human languages were created to communicate about the world around us, and thus it’s my belief that any “proper” language model also must encode some knowledge of the world around us. In this regard, I think the GPT 3+ models are a genuine advancement of the state of the art.

However, while chat GPT talks like an adult, I’d guess that its actual “real world experience” is probably more comparable to a child's. Sure, the topics that its “experience” covers aren’t the same as a five year old--no one’s read the IRS’s official docs to a kid. But don’t mistake encyclopedic recall of the tax code for its ability to advise you on its tax decisions. When you ask chat GPT how to file your taxes, really what it’s doing is consulting the mess of forums and government sites encoded into a neural net for the most likely response, the same way a kid would cite the top search result for a school report. And sometimes that’s good enough. And sometimes it’s not.

Either way, it would be absurd to entrust something with the real-world experience of a child with your financial well-being.

Language is not Decision Making

The “real world experience” mentioned above is an emergent property of the sheer quantity of training data the large language models have ingested. If you feed the system 3,000 articles about what precipitated the start of World War 1, eventually it’ll learn an association with Franz Ferdinand.

However, it’ll never learn something that it hasn’t seen. If you ask it to decide something for you, say, “what should I wear on my date”, it’s not actually making a decision. It’s looking at the aggregate answer of every instance of its training data where someone who asked that question and pulling out the thing the most people have responded with. It is definitely, absolutely not, looking at the available options and composing an outfit for you.

For something as trivial as what to wear, this can be fine. In fact, it might be desirable to see what the majority thinks here. But it can be absolutely devastating for entrenching the status quo, especially for places where the majority disagrees with the minority. For example, “what does a family look like?”, “what’s a typical diet?”, “what are successful careers?” Responsible stewards of these models will take steps to make sure that responses to these kinds of questions are adequately couched by seeding the training data with more representative examples and or deflections, but fundamentally, language models are reflections of their training data. They cannot divert from them.

Hallucination is not Imagination

Pick a man’s name. Pick a woman’s name. Ask chat GPT to tell you the difference between them.

Was the man taller? Was the woman smaller? Was the man stronger? Was the woman kinder? Were they both white?

It made up these two people, but it did not imagine them. It picked the most common traits that might be associated with those people, minus the most egregious things that the companies that run them have put in huge expense to train out.

Something about this process strikes me as the exact opposite of imagination. Instead of imagining how the world might be different, unique, peculiar and strange, these models tell us the world exactly as we believe it to be, stereotyped, oversimplified, with none of the nuance or wonder it has today, or might have tomorrow.