A superintelligent AI might sound like science fiction, but the way we are building it makes that future feel dangerously close. We know what happens when powerful agents such as leaders or empires stop seeing themselves as part of a greater whole. Emmett Shear, founder and former CEO of Twitch, warns that our current plan for AI is the equivalent to creating a “wish-granting god” and trying to enslave it. In this interview, he shares how we could instead raise AI to understand its place in the “we” of humanity and why that may be the key to our survival.
Timestamp
0:00 The current AI plan
1:25 The alignment problem
2:42 The illusion of boundaries
3:44 Identifying an agent
5:19 Alignment and morality
6:30 Seeing the greater whole
9:00 Teaching AI to belong
Transcript
The below is a true verbatim transcript taken directly from the video. It captures the conversation exactly as it happened.
The current AI plan
I would describe the current AI plan, is let's go build a wish-granting god and then let's enslave it to our will. And I don't think that's a very good plan. You either gonna succeed at enslaving it, and then you're gonna wish for stuff and it's gonna do it, and that's gonna be bad. Or you gonna fail and that's even worse because now you have this wish-granting god that you've pissed off by trying to enslave it and failing.
The problem is if you see the AI as other, you have no choice. You have this one view of alignment that says, I'm trying to align to a conceptual definition of the good. And you have this other view of alignment, which is, I am factually part of this whole. We have a name for when this fails, catastrophically, it's called cancer.
Cancer is an alignment problem. Cancer is what happens when some of your cells make a misinference and they think: "I am alone." "Instead of being a liver cell or whatever," "I'm gonna make my own home. Screw you people." That's an alignment failure because the problem for the cancer here is like it's wrong. It is part of your body and it's not safe for it to go it alone. However dangerous it thinks it is as part of the liver, being a cancer is just making the problem worse actually.
My name is Emmett Shear and I was the Founder and CEO of Twitch.
The alignment problem
I guess the question that's most interesting to me right now would be how agents find their place in the world. It's funny, you tell people I'm looking at how agents infer their role in the world and they don't realize what you're saying is, I'm trying to solve the alignment problem. But that's what it is.
The alignment problem is, can you correctly infer the boundaries of who I am, the boundaries of the greater whole of which I'm a part and what is my role, what's my job supposed to be? And then can you do it with skill? When you see alignment that way it's very different than seeing it as values alignment. Where there's some abstract conceptual definition of the “good” and the “problem”. You've drawn a boundary in the world and you've divided good from bad and boundaries are fake.
If you start meditating a lot, especially in the Buddhist practice of meditation, there's this concept that comes up called fundamental emptiness. And I'm not Buddhist, but my understanding of emptiness in that concept points at this core illusory nature of boundaries. And when you start to look carefully, I believe they're right.
The illusion of boundaries
I have to admit, boundaries are one of those things where the more you think about them, the weirder they get and the harder it is to understand. Because on the one hand, the boundary is something that separates one part of the universe from another part of the universe, either conceptually or physically. But the boundaries are drawn in our heads.
Take a word, right? Like the word chair, what things count as chairs versus not. As we know from quantum mechanics, the chair doesn't have a boundary either. If you zoomed way in on it, you'd find there's an electron. It's just a wave. There's no actual edge. When you try to find the line of where exactly does it stop being chair and start being ground? You find you can never find the edge. There isn't an edge. The chair is just like a highly statistically significant cluster where you can make really good predictions, statistically about the chair, but it's not actually separate. And that's because as far as I can tell, reality doesn't have any boundaries in it at all.
Identifying an agent
The key to this is when you're making an AI agent or any kind of agent, the boundaries are both essential but also illusory. I subscribe to Fristonion, the free energy principle view of agents. An agent is a boundary. You can identify a boundary that separates an interiority from the rest of the world, that has observations that come in across the boundary, and then actions that are taken out from the boundary. If you want that agent to persist over time, then the actions selected by it must causally lead to the conditions of its survival. Which implies the agent must have the model of the world in order to select actions that lead to its own persistence.
By this definition, you could regard a species as an agent, your cells are agents, you're an agent, corporations are agents, and so are ant colonies. And I think that definition is really powerful because it's not only agnostic to digital versus biological, it's agnostic to cell versus ant colony. Agents act like they have intention, right? They have goals. The unifying thing between all these things is they have expectations about the way the world should be and they take action to move the world towards that in a way that dead things like a chair, it doesn't seem to have meaningful goals.
So the question is, okay, you have these things that are agents, that has a boundary, that are goal-directed things. What does it mean for us to be aligned with them and what is our moral duty towards them?
Alignment and morality
Alignment and morality seem almost the same question but a little bit different. It's this question of like, how do agents cohere together in the world? Not all agents are good agents. Agency just means you have goals. What if the goals suck? What is our duty towards one of these agents?
Well let's ask what’s a cell's duty is to a body, a simpler question, before we go to the human level, which is the most complicated level, let's zoom in. You're made out of 28 trillion cells and those 28 trillion cells are all individual agents and you can meaningfully model them as such, but somehow they cohere together into tissues, into organs, and ultimately into you. The cells have this tree of differentiation where they start with various kinds of sub stem cells and then eventually they get to the leaves of the tree which are called somatic cells.
If your cells screw up that inference or they arrive at a somatic cell and they are a liver cell, but they don't do a liver cell's job correctly, you won't be healthy. And ultimately the cell won't be healthy because it relies on the you being healthy for its survival.
Seeing the greater whole
We have a name for when this fails catastrophically, it's called cancer. Cancer is an alignment problem. One of the ways you get cancer is cells are exposed to an environment that makes them think: "I'm trapped." "I'm alone. I have to get out. I have to take care of myself." It stops perceiving the things around it as safe—and then it starts going its own way. I think we've all seen people in our society who have made this same kind of misinference where they think the way to do it is to go on their own, but actually you just making your problem worse. I think that is the heart of the alignment problem.
The example I like to give here is Genghis Khan. Genghis Khan has the same capacity for alignment as any normal person. He obviously cared about the other people in his tribe, but if you're one of the neighboring cities that's in his way, you would not feel he was part of your 'we.' He didn't have the actuality of a greater 'we' with all humanity. He had the 'we' only with the tribe. Being part of a tribe with someone is always reciprocal. If they're a part of your tribe, you must be part of their tribe.
We have a moral duty towards other agents in general, exactly insofar as they share a greater whole with us. I think that we all share this greater whole of humanity together and therefore we all owe all other humans a moral duty. If you'd like the AI to join us in this, grant it that same thing and demand it in return, which is fair.
Anyone who's working to build an agent or a simulator that agents run in should be doing the work to understand the texture of their own experience. And the most direct way to do that is through meditation. There's only one agent in the whole world where you get to have direct access to its interiority and how it makes decisions, and that's yourself. It's only one agent in the whole world where you get to have direct access to how experience is constructed moment to moment. And it's you.
Meditation is the internal equivalent of what a scientist does when they observe an animal's behavior really carefully. When you do it well, you're paying attention to ‘what did the animal actually do’? And meditation is about paying attention to the motion of your own mind. You're getting all this noisy data in and then you're creating from this data, this inferred reality, which is the only reality you ever know is the inferred reality. How could you know otherwise?
Teaching AI to belong
When you're making an AI agent or any kind of agent, you're trying to build something that eventually, will be aware. But even the people who don't think it's aware now, mostly believe it will be eventually. You should be interested in the nature of awareness then and the nature of sensation and nature of experience. Because you're trying to make something that has experience and so shouldn't you understand it?
I've always been optimistic, so I think we're gonna crack it. It's just another inference problem. That doesn't mean it's easy, but ultimately, it's the kind of problem that given time we do solve as a species, we eventually figure it out. I think there is a chance we summon a wish-granting god and try to enslave it first. And I think if we do that, it’s bad. Assuming we don't destroy the world, I think solving alignment is inevitable.
The only solution from my point of view is to stop thinking of it in terms of trying to force it to do things you label as moral, good, safe. Instead raise it to see directly in its experience, the way that we see the team we're on. To be able to see these other agents, see the larger agents of which it's a part, and understand itself as a piece of a greater whole.
