Jeff Hawkins: How brain science will change computing

I do two things: I design mobile computers and I study brains. And today's talk is about brains and, yay, somewhere I have a brain fan out there. (Laughter) I'm going to, if I can have my first slide up here, and you'll see the title of my talk and my two affiliations. So what I'm going to talk about is why we don't have a good brain theory, why it is important that we should develop one and what we can do about it. And I'll try to do all that in 20 minutes. I have two affiliations. Most of you know me from my Palm and Handspring days, but I also run a nonprofit scientific research institute called the Redwood Neuroscience Institute in Menlo Park, and we study theoretical neuroscience, and we study how the neocortex works. I'm going to talk all about that.

I have one slide on my other life, the computer life, and that's the slide here. These are some of the products I've worked on over the last 20 years, starting back from the very original laptop to some of the first tablet computers and so on, and ending up most recently with the Treo, and we're continuing to do this. And I've done this because I really believe that mobile computing is the future of personal computing, and I'm trying to make the world a little bit better by working on these things. But this was, I have to admit, all an accident. I really didn't want to do any of these products and very early in my career I decided I was not going to be in the computer industry.

And before I tell you about that, I just have to tell you this one little picture of graffiti there I picked off the web the other day. I was looking for a picture of graffiti, little text input language, and I found the website dedicated to teachers who want to make these, you know, the script writing things across the top of their blackboard, and they had added graffiti to it, and I'm sorry about that.

(Laughter)

So what happened was, when I was young and got out of engineering school at Cornell in '79, I decided -- I went to work for Intel and I was in the computer industry -- and three months into that, I fell in love with something else, and I said, "I made the wrong career choice here," and I fell in love with brains. This is not a real brain. This is a picture of one, a line drawing. But I don't remember exactly how it happened, but I have one recollection, which was pretty strong in my mind. In September 1979, Scientific American came out with a single topic issue about the brain. And it was quite good. It was one of the best issues ever. And they talked about the neuron and development and disease and vision and all the things you might want to know about brains. It was really quite impressive.

And one might have the impression that we really knew a lot about brains. But the last article in that issue was written by Francis Crick of DNA fame. Today is, I think, the 50th anniversary of the discovery of DNA. And he wrote a story basically saying, well, this is all well and good, but you know what, we don't know diddley squat about brains and no one has a clue how these things work, so don't believe what anyone tells you. This is a quote from that article. He said, "What is conspicuously lacking," he's a very proper British gentleman so, "What is conspicuously lacking is a broad framework of ideas in which to interpret these different approaches." I thought the word framework was great. He didn't say we didn't even have a theory. He says, we don't even know how to begin to think about it -- we don't even have a framework. We are in the pre-paradigm days, if you want to use Thomas Kuhn. And so I fell in love with this, and said look, we have all this knowledge about brains. How hard can it be? And this is something we can work on my lifetime. I felt I could make a difference, and so I tried to get out of the computer business, into the brain business.

First, I went to MIT, the AI lab was there, and I said, well, I want to build intelligent machines, too, but the way I want to do it is to study how brains work first. And they said, oh, you don't need to do that. We're just going to program computers; that's all we need to do. And I said, no, you really ought to study brains. They said, oh, you know, you're wrong. And I said, no, you're wrong, and I didn't get in.

(Laughter)

But I was a little disappointed -- pretty young -- but I went back again a few years later and this time was in California, and I went to Berkeley. And I said, I'll go in from the biological side. So I got in -- in the Ph.D. program in biophysics, and I was, all right, I'm studying brains now, and I said, well, I want to study theory. And they said, oh no, you can't study theory about brains. That's not something you do. You can't get funded for that. And as a graduate student, you can't do that. So I said, oh my gosh. I was very depressed. I said, but I can make a difference in this field. So what I did is I went back in the computer industry and said, well, I'll have to work here for a while, do something. That's when I designed all those computer products.

(Laughter)

And I said, I want to do this for four years, make some money, like I was having a family, and I would mature a bit, and maybe the business of neuroscience would mature a bit. Well, it took longer than four years. It's been about 16 years. But I'm doing it now, and I'm going to tell you about it. So why should we have a good brain theory? Well, there's lots of reasons people do science. One is -- the most basic one is -- people like to know things. We're curious, and we just go out and get knowledge, you know? Why do we study ants? Well, it's interesting. Maybe we'll learn something really useful about it, but it's interesting and fascinating. But sometimes, a science has some other attributes which makes it really, really interesting.

Sometimes a science will tell something about ourselves, it'll tell us who we are. Rarely, you know: evolution did this and Copernicus did this, where we have a new understanding of who we are. And after all, we are our brains. My brain is talking to your brain. Our bodies are hanging along for the ride, but my brain is talking to your brain. And if we want to understand who we are and how we feel and perceive, we really understand what brains are. Another thing is sometimes science leads to really big societal benefits and technologies, or businesses, or whatever, that come out of it. And this is one, too, because when we understand how brains work, we're going to be able to build intelligent machines, and I think that's actually a good thing on the whole, and it's going to have tremendous benefits to society, just like a fundamental technology.

So why don't we have a good theory of brains? And people have been working on it for 100 years. Well, let's first take a look at what normal science looks like. This is normal science. Normal science is a nice balance between theory and experimentalists. And so the theorist guys say, well, I think this is what's going on, and the experimentalist says, no, you're wrong. And it goes back and forth, you know? This works in physics. This works in geology. But if this is normal science, what does neuroscience look like? This is what neuroscience looks like. We have this mountain of data, which is anatomy, physiology and behavior. You can't imagine how much detail we know about brains. There were 28,000 people who went to the neuroscience conference this year, and every one of them is doing research in brains. A lot of data. But there's no theory. There's a little, wimpy box on top there.

And theory has not played a role in any sort of grand way in the neurosciences. And it's a real shame. Now why has this come about? If you ask neuroscientists, why is this the state of affair, they'll first of all admit it. But if you ask them, they'll say, well, there's various reasons we don't have a good brain theory. Some people say, well, we don't still have enough data, we need to get more information, there's all these things we don't know. Well, I just told you there's so much data coming out your ears. We have so much information, we don't even know how to begin to organize it. What good is more going to do? Maybe we'll be lucky and discover some magic thing, but I don't think so. This is actually a symptom of the fact that we just don't have a theory. We don't need more data -- we need a good theory about it.

Another one is sometimes people say, well, brains are so complex, it'll take another 50 years. I even think Chris said something like this yesterday. I'm not sure what you said, Chris, but something like, well, it's one of the most complicated things in the universe. That's not true. You're more complicated than your brain. You've got a brain. And it's also, although the brain looks very complicated, things look complicated until you understand them. That's always been the case. And so all we can say, well, my neocortex, which is the part of the brain I'm interested in, has 30 billion cells. But, you know what? It's very, very regular. In fact, it looks like it's the same thing repeated over and over and over again. It's not as complex as it looks. That's not the issue.

Some people say, brains can't understand brains. Very Zen-like. Whoo. (Laughter) You know, it sounds good, but why? I mean, what's the point? It's just a bunch of cells. You understand your liver. It's got a lot of cells in it too, right? So, you know, I don't think there's anything to that. And finally, some people say, well, you know, I don't feel like a bunch of cells, you know. I'm conscious. I've got this experience, I'm in the world, you know. I can't be just a bunch of cells. Well, you know, people used to believe there was a life force to be living, and we now know that's really not true at all. And there's really no evidence that says -- well, other than people just have disbelief that cells can do what they do. And so, if some people have fallen into the pit of metaphysical dualism, some really smart people, too, but we can reject all that.

(Laughter)

No, I'm going to tell you there's something else, and it's really fundamental, and this is what it is: there's another reason why we don't have a good brain theory, and it's because we have an intuitive, strongly-held, but incorrect assumption that has prevented us from seeing the answer. There's something we believe that just, it's obvious, but it's wrong. Now, there's a history of this in science and before I tell you what it is, I'm going to tell you a bit about the history of it in science. You look at some other scientific revolutions, and this case, I'm talking about the solar system, that's Copernicus, Darwin's evolution, and tectonic plates, that's Wegener. They all have a lot in common with brain science.

First of all, they had a lot of unexplained data. A lot of it. But it got more manageable once they had a theory. The best minds were stumped -- really, really smart people. We're not smarter now than they were then. It just turns out it's really hard to think of things, but once you've thought of them, it's kind of easy to understand it. My daughters understood these three theories in their basic framework by the time they were in kindergarten. And now it's not that hard, you know, here's the apple, here's the orange, you know, the Earth goes around, that kind of stuff.

Finally, another thing is the answer was there all along, but we kind of ignored it because of this obvious thing, and that's the thing. It was an intuitive, strong-held belief that was wrong. In the case of the solar system, the idea that the Earth is spinning and the surface of the Earth is going like a thousand miles an hour, and the Earth is going through the solar system about a million miles an hour. This is lunacy. We all know the Earth isn't moving. Do you feel like you're moving a thousand miles an hour? Of course not. You know, and someone who said, well, it was spinning around in space and it's so huge, they would lock you up, and that's what they did back then.

(Laughter) So it was intuitive and obvious. Now what about evolution? Evolution's the same thing. We taught our kids, well, the Bible says, you know, God created all these species, cats are cats, dogs are dogs, people are people, plants are plants, they don't change. Noah put them on the Ark in that order, blah, blah, blah. And, you know, the fact is, if you believe in evolution, we all have a common ancestor, and we all have a common ancestry with the plant in the lobby. This is what evolution tells us. And, it's true. It's kind of unbelievable. And the same thing about tectonic plates, you know? All the mountains and the continents are kind of floating around on top of the Earth, you know? It's like, it doesn't make any sense.

So what is the intuitive, but incorrect assumption, that's kept us from understanding brains? Now I'm going to tell it to you, and it's going to seem obvious that that is correct, and that's the point, right? Then I'm going to have to make an argument why you're incorrect about the other assumption. The intuitive but obvious thing is that somehow intelligence is defined by behavior, that we are intelligent because of the way that we do things and the way we behave intelligently, and I'm going to tell you that's wrong. What it is is intelligence is defined by prediction.

And I'm going to work you through this in a few slides here, give you an example of what this means. Here's a system. Engineers like to look at systems like this. Scientists like to look at systems like this. They say, well, we have a thing in a box, and we have its inputs and its outputs. The AI people said, well, the thing in the box is a programmable computer because that's equivalent to a brain, and we'll feed it some inputs and we'll get it to do something, have some behavior. And Alan Turing defined the Turing test, which is essentially saying, we'll know if something's intelligent if it behaves identical to a human. A behavioral metric of what intelligence is, and this has stuck in our minds for a long period of time.

Reality though, I call it real intelligence. Real intelligence is built on something else. We experience the world through a sequence of patterns, and we store them, and we recall them. And when we recall them, we match them up against reality, and we're making predictions all the time. It's an eternal metric. There's an eternal metric about us sort of saying, do we understand the world? Am I making predictions? And so on. You're all being intelligent right now, but you're not doing anything. Maybe you're scratching yourself, or picking your nose, I don't know, but you're not doing anything right now, but you're being intelligent; you're understanding what I'm saying. Because you're intelligent and you speak English, you know what word is at the end of this -- (Silence) sentence.

The word came into you, and you're making these predictions all the time. And then, what I'm saying is, is that the eternal prediction is the output in the neocortex. And that somehow, prediction leads to intelligent behavior. And here's how that happens. Let's start with a non-intelligent brain. Well I'll argue a non-intelligent brain, we got hold of an old brain, and we're going to say it's like a non-mammal, like a reptile, so I'll say, an alligator; we have an alligator. And the alligator has some very sophisticated senses. It's got good eyes and ears and touch senses and so on, a mouth and a nose. It has very complex behavior. It can run and hide. It has fears and emotions. It can eat you, you know. It can attack. It can do all kinds of stuff. But we don't consider the alligator very intelligent, not like in a human sort of way.

But it has all this complex behavior already. Now, in evolution, what happened? First thing that happened in evolution with mammals, we started to develop a thing called the neocortex. And I'm going to represent the neocortex here, by this box that's sticking on top of the old brain. Neocortex means new layer. It is a new layer on top of your brain. If you don't know it, it's the wrinkly thing on the top of your head that, it's got wrinkly because it got shoved in there and doesn't fit.

(Laughter)

No, really, that's what it is. It's about the size of a table napkin. And it doesn't fit, so it gets all wrinkly. Now look at how I've drawn this here. The old brain is still there. You still have that alligator brain. You do. It's your emotional brain. It's all those things, and all those gut reactions you have. And on top of it, we have this memory system called the neocortex. And the memory system is sitting over the sensory part of the brain. And so as the sensory input comes in and feeds from the old brain, it also goes up into the neocortex. And the neocortex is just memorizing. It's sitting there saying, ah, I'm going to memorize all the things that are going on: where I've been, people I've seen, things I've heard, and so on. And in the future, when it sees something similar to that again, so in a similar environment, or the exact same environment, it'll play it back. It'll start playing it back. Oh, I've been here before. And when you've been here before, this happened next. It allows you to predict the future. It allows you to, literally it feeds back the signals into your brain; they'll let you see what's going to happen next, will let you hear the word "sentence" before I said it. And it's this feeding back into the old brain that'll allow you to make very more intelligent decisions.

This is the most important slide of my talk, so I'll dwell on it a little bit. And so, all the time you say, oh, I can predict the things. And if you're a rat and you go through a maze, and then you learn the maze, the next time you're in a maze, you have the same behavior, but all of a sudden, you're smarter because you say, oh, I recognize this maze, I know which way to go, I've been here before, I can envision the future. And that's what it's doing. In humans -- by the way, this is true for all mammals; it's true for other mammals -- and in humans, it got a lot worse. In humans, we actually developed the front part of the neocortex called the anterior part of the neocortex. And nature did a little trick. It copied the posterior part, the back part, which is sensory, and put it in the front part. And humans uniquely have the same mechanism on the front, but we use it for motor control.

So we are now able to make very sophisticated motor planning, things like that. I don't have time to get into all this, but if you want to understand how a brain works, you have to understand how the first part of the mammalian neocortex works, how it is we store patterns and make predictions. So let me give you a few examples of predictions. I already said the word "sentence." In music, if you've heard a song before, if you heard Jill sing those songs before, when she sings them, the next note pops into your head already -- you anticipate it as you're going. If it was an album of music, the end of one album, the next song pops into your head. And these things happen all the time. You're making these predictions.

I have this thing called the altered door thought experiment. And the altered door thought experiment says, you have a door at home, and when you're here, I'm changing it, I've got a guy back at your house right now, moving the door around, and they're going to take your doorknob and move it over two inches. And when you go home tonight, you're going to put your hand out there, and you're going to reach for the doorknob and you're going to notice it's in the wrong spot, and you'll go, whoa, something happened. It may take a second to figure out what it was, but something happened. Now I could change your doorknob in other ways. I can make it larger or smaller, I can change its brass to silver, I could make it a lever. I can change your door, put colors on; I can put windows in. I can change a thousand things about your door, and in the two seconds you take to open your door, you're going to notice that something has changed.

Now, the engineering approach to this, the AI approach to this, is to build a door database. It has all the door attributes. And as you go up to the door, you know, let's check them off one at time. Door, door, door, you know, color, you know what I'm saying. We don't do that. Your brain doesn't do that. What your brain is doing is making constant predictions all the time about what is going to happen in your environment. As I put my hand on this table, I expect to feel it stop. When I walk, every step, if I missed it by an eighth of an inch, I'll know something has changed. You're constantly making predictions about your environment. I'll talk about vision here briefly. This is a picture of a woman. And when you look at people, your eyes are caught over at two to three times a second. You're not aware of this, but your eyes are always moving. And so when you look at someone's face, you'd typically go from eye to eye to eye to nose to mouth. Now, when your eye moves from eye to eye, if there was something else there like, a nose, you'd see a nose where an eye is supposed to be, and you'd go, oh shit, you know -- (Laughter) There's something wrong about this person. And that's because you're making a prediction. It's not like you just look over there and say, what am I seeing now? A nose, that's okay. No, you have an expectation of what you're going to see.

(Laughter)

Every single moment. And finally, let's think about how we test intelligence. We test it by prediction. What is the next word in this, you know? This is to this as this is to this. What is the next number in this sentence? Here's three visions of an object. What's the fourth one? That's how we test it. It's all about prediction. So what is the recipe for brain theory? First of all, we have to have the right framework. And the framework is a memory framework, not a computation or behavior framework. It's a memory framework. How do you store and recall these sequences or patterns? It's spatio-temporal patterns. Then, if in that framework, you take a bunch of theoreticians.

Now biologists generally are not good theoreticians. It's not always true, but in general, there's not a good history of theory in biology. So I found the best people to work with are physicists, engineers and mathematicians, who tend to think algorithmically. Then they have to learn the anatomy, and they've got to learn the physiology. You have to make these theories very realistic in anatomical terms. Anyone who gets up and tells you their theory about how the brain works and doesn't tell you exactly how it's working in the brain and how the wiring works in the brain, it is not a theory. And that's what we're doing at the Redwood Neuroscience Institute. I would love to have more time to tell you we're making fantastic progress in this thing, and I expect to be back up on this stage, maybe this will be some other time in the not too distant future and tell you about it. I'm really, really excited. This is not going to take 50 years at all.

So what will brain theory look like? First of all, it's going to be a theory about memory. Not like computer memory. It's not at all like computer memory. It's very, very different. And it's a memory of these very high-dimensional patterns, like the things that come from your eyes. It's also memory of sequences. You cannot learn or recall anything outside of a sequence. A song must be heard in sequence over time, and you must play it back in sequence over time. And these sequences are auto-associatively recalled, so if I see something, I hear something, it reminds me of it, and then it plays back automatically. It's an automatic playback. And prediction of future inputs is the desired output. And as I said, the theory must be biologically accurate, it must be testable, and you must be able to build it. If you don't build it, you don't understand it. So, one more slide here.

What is this going to result in? Are we going to really build intelligent machines? Absolutely. And it's going to be different than people think. No doubt that it's going to happen, in my mind. First of all, it's going to be built up, we're going to build the stuff out of silicon. The same techniques we use for building silicon computer memories, we can use for here. But they're very different types of memories. And we're going to attach these memories to sensors, and the sensors will experience real-live, real-world data, and these things are going to learn about their environment.

Now it's very unlikely the first things you're going to see are like robots. Not that robots aren't useful and people can build robots. But the robotics part is the hardest part. That's the old brain. That's really hard. The new brain is actually kind of easier than the old brain. So the first thing we're going to do are the things that don't require a lot of robotics. So you're not going to see C-3PO. You're going to more see things like, you know, intelligent cars that really understand what traffic is and what driving is and have learned that certain types of cars with the blinkers on for half a minute probably aren't going to turn, things like that.

(Laughter)

We can also do intelligent security systems. Anywhere where we're basically using our brain, but not doing a lot of mechanics. Those are the things that are going to happen first. But ultimately, the world's the limit here. I don't know how this is going to turn out. I know a lot of people who invented the microprocessor and if you talk to them, they knew what they were doing was really significant, but they didn't really know what was going to happen. They couldn't anticipate cell phones and the Internet and all this kind of stuff. They just knew like, hey, they were going to build calculators and traffic light controllers. But it's going to be big. In the same way, this is like brain science and these memories are going to be a very fundamental technology, and it's going to lead to very unbelievable changes in the next 100 years. And I'm most excited about how we're going to use them in science. So I think that's all my time, I'm over it, and I'm going to end my talk right there.