John Underkoffler: Pointing to the future of UI

We're 25, 26 years after the advent of the Macintosh, which was an astoundingly seminal event in the history of human-machine interface and in computation in general. It fundamentally changed the way that people thought about computation, thought about computers, how they used them and who and how many people were able to use them. It was such a radical change, in fact, that the early Macintosh development team in '82, '83, '84 had to write an entirely new operating system from the ground up. Now, this is an interesting little message, and it's a lesson that has since, I think, been forgotten or lost or something, and that is, namely, that the OS is the interface. The interface is the OS. It's like the land and the king (i.e. Arthur) they're inseparable, they are one. And to write a new operating system was not a capricious matter. It wasn't just a matter of tuning up some graphics routines. There were no graphics routines. There were no mouse drivers. So it was a necessity.

But in the quarter-century since then, we've seen all of the fundamental supporting technologies go berserk. So memory capacity and disk capacity have been multiplied by something between 10,000 and a million. Same thing for processor speeds. Networks, we didn't have networks at all at the time of the Macintosh's introduction, and that has become the single most salient aspect of how we live with computers. And, of course, graphics: Today 84 dollars and 97 cents at Best Buy buys you more graphics power than you could have gotten for a million bucks from SGI only a decade ago. So we've got that incredible ramp-up. Then, on the side, we've got the Web and, increasingly, the cloud, which is fantastic, but also -- in the regard in which an interface is fundamental -- kind of a distraction. So we've forgotten to invent new interfaces. Certainly we've seen in recent years a lot of change in that regard, and people are starting to wake up about that. So what happens next? Where do we go from there?

The problem, as we see it, has to do with a single, simple word: "space," or a single, simple phrase: "real world geometry." Computers and the programming languages that we talk to them in, that we teach them in, are hideously insensate when it comes to space. They don't understand real world space. It's a funny thing because the rest of us occupy it quite frequently and quite well. They also don't understand time, but that's a matter for a separate talk.

So what happens if you start to explain space to them? One thing you might get is something like the Luminous Room. The Luminous Room is a system in which it's considered that input and output spaces are co-located. That's a strangely simple, and yet unexplored idea, right? When you use a mouse, your hand is down here on the mouse pad. It's not even on the same plane as what you're talking about: The pixels are up on the display. So here was a room in which all the walls, floors, ceilings, pets, potted plants, whatever was in there, were capable, not only of display but of sensing as well. And that means input and output are in the same space enabling stuff like this. That's a digital storage in a physical container. The contract is the same as with real word objects in real world containers. Has to come back out, whatever you put in. This little design experiment that was a small office here knew a few other tricks as well. If you presented it with a chess board, it tried to figure out what you might mean by that. And if there was nothing for them to do, the chess pieces eventually got bored and hopped away.

The academics who were overseeing this work thought that that was too frivolous, so we built deadly serious applications like this optics prototyping workbench in which a toothpaste cap on a cardboard box becomes a laser. The beam splitters and lenses are represented by physical objects, and the system projects down the laser beam path. So you've got an interface that has no interface. You operate the world as you operate the real world, which is to say, with your hands. Similarly, a digital wind tunnel with digital wind flowing from right to left -- not that remarkable in a sense; we didn't invent the mathematics. But if you displayed that on a CRT or flat panel display, it would be meaningless to hold up an arbitrary object, a real world object in that. Here, the real world merges with the simulation.

And finally, to pull out all the stops, this is a system called Urp, for urban planners, in which we give architects and urban planners back the models that we confiscated when we insisted that they use CAD systems. And we make the machine meet them half way. It projects down digital shadows, as you see here. And if you introduce tools like this inverse clock, then you can control the sun's position in the sky. That's 8 a.m. shadows. They get a little shorter at 9 a.m. There you are, swinging the sun around. Short shadows at noon and so forth. And we built up a series of tools like this. There are inter-shadowing studies that children can operate, even though they don't know anything about urban planning: To move a building, you simply reach out your hand and you move the building. A material wand makes the building into a sort of Frank Gehry thing that reflects light in all directions. Are you blinding passers by and motorists on the freeways? A zoning tool connects distant structures, a building and a roadway. Are you going to get sued by the zoning commission? And so forth. Now, if these ideas seem familiar or perhaps even a little dated, that's great; they should seem familiar.

This work is 15 years old. This stuff was undertaken at MIT and the Media Lab under the incredible direction of Professor Hiroshi Ishii, director of the Tangible Media Group. But it was that work that was seen by Alex McDowell, one of the world's legendary production designers. But Alex was preparing a little, sort of obscure, indie, arthouse film called "Minority Report" for Steven Spielberg, and invited us to come out from MIT and design the interfaces that would appear in that film. And the great thing about it was that Alex was so dedicated to the idea of verisimilitude, the idea that the putative 2054 that we were painting in the film be believable, that he allowed us to take on that design work as if it were an R&D effort. And the result is sort of gratifyingly perpetual. People still reference those sequences in "Minority Report" when they talk about new UI design. So this led full circle, in a strange way, to build these ideas into what we believe is the necessary future of human machine interface: the Spatial Operating Environment, we call it.

So here we have a bunch of stuff, some images. And, using a hand, we can actually exercise six degrees of freedom, six degrees of navigational control. And it's fun to fly through Mr. Beckett's eye. And you can come back out through the scary orangutan. And that's all well and good. Let's do something a little more difficult. Here, we have a whole bunch of disparate images. We can fly around them. So navigation is a fundamental issue. You have to be able to navigate in 3D. Much of what we want computers to help us with in the first place is inherently spatial. And the part that isn't spatial can often be spatialized to allow our wetware to make greater sense of it. Now we can distribute this stuff in many different ways. So we can throw it out like that. Let's reset it. We can organize it this way.

And, of course, it's not just about navigation, but about manipulation as well. So if we don't like stuff, or we're intensely curious about Ernst Haeckel's scientific falsifications, we can pull them out like that. And then if it's time for analysis, we can pull back a little bit and ask for a different distribution. Let's just come down a bit and fly around. So that's a different way to look at stuff. If you're of a more analytical nature then you might want, actually, to look at this as a color histogram. So now we've got the stuff color-sorted, angle maps onto color. And now, if we want to select stuff, 3D, space, the idea that we're tracking hands in real space becomes really important because we can reach in, not in 2D, not in fake 2D, but in actual 3D. Here are some selection planes. And we'll perform this Boolean operation because we really love yellow and tapirs on green grass.

So, from there to the world of real work. Here's a logistics system, a small piece of one that we're currently building. There're a lot of elements. And one thing that's very important is to combine traditional tabular data with three-dimensional and geospatial information. So here's a familiar place. And we'll bring this back here for a second. Maybe select a little bit of that. And bring out this graph. And we should, now, be able to fly in here and have a closer look. These are logistics elements that are scattered across the United States.

One thing that three-dimensional interactions and the general idea of imbuing computation with space affords you is a final destruction of that unfortunate one-to-one pairing between human beings and computers. That's the old way, that's the old mantra: one machine, one human, one mouse, one screen. Well, that doesn't really cut it anymore. In the real world, we have people who collaborate; we have people who have to work together, and we have many different displays. And we might want to look at these various images. We might want to ask for some help.

The author of this new pointing device is sitting over there, so I can pull this from there to there. These are unrelated machines, right? So the computation is space soluble and network soluble. So I'm going to leave that over there because I have a question for Paul. Paul is the designer of this wand, and maybe its easiest for him to come over here and tell me in person what's going on. So let me get some of these out of the way. Let's pull this apart: I'll go ahead and explode it. Kevin, can you help? Let me see if I can help us find the circuit board. Mind you, it's a sort of gratuitous field-stripping exercise, but we do it in the lab all the time. All right. So collaborative work, whether it's immediately co-located or distant and distinct, is always important. And again, that stuff needs to be undertaken in the context of space.

And finally, I'd like to leave you with a glimpse that takes us back to the world of imagery. This is a system called TAMPER, which is a slightly whimsical look at what the future of editing and media manipulation systems might be. We at Oblong believe that media should be accessible in much more fine-grained form. So we have a large number of movies stuck inside here. And let's just pick out a few elements. We can zip through them as a possibility. We can grab elements off the front, where upon they reanimate, come to life, and drag them down onto the table here. We'll go over to Jacques Tati here and grab our blue friend and put him down on the table as well. We may need more than one. And we probably need, well, we probably need a cowboy to be quite honest. (Laughter) Yeah, let's take that one. (Laughter) You see, cowboys and French farce people don't go well together, and the system knows that.

Let me leave with one final thought, and that is that one of the greatest English language writers of the last three decades suggested that great art is always a gift. And he wasn't talking about whether the novel costs 24.95 [dollars], or whether you have to spring 70 million bucks to buy the stolen Vermeer; he was talking about the circumstances of its creation and of its existence. And I think that it's time that we asked for the same from technology. Technology is capable of expressing and being imbued with a certain generosity, and we need to demand that, in fact. For some of this kind of technology, ground center is a combination of design, which is crucially important. We can't have advances in technology any longer unless design is integrated from the very start. And, as well, as of efficacy, agency. We're, as human beings, the creatures that create, and we should make sure that our machines aid us in that task and are built in that same image.

So I will leave you with that. Thank you.

(Applause)

Chris Anderson: So to ask the obvious question -- actually this is from Bill Gates -- when? (John Underkoffler: When?) CA: When real? When for us, not just in a lab and on a stage? Can it be for every man, or is this just for corporations and movie producers?

JU: No, it has to be for every human being. That's our goal entirely. We won't have succeeded unless we take that next big step. I mean it's been 25 years. Can there really be only one interface? There can't.

CA: But does that mean that, at your desk or in your home, you need projectors, cameras? You know, how can it work?

JU: No, this stuff will be built into the bezel of every display. It'll be built into architecture. The gloves go away in a matter of months or years. So this is the inevitability about it.

CA: So, in your mind, five years time, someone can buy this as part of a standard computer interface?

JU: I think in five years time when you buy a computer, you'll get this.

CA: Well that's cool. (Applause) The world has a habit of surprising us as to how these things are actually used. What do you think, what in your mind is the first killer app for this?

JU: That's a good question, and we ask ourselves that every day. At the moment, our early-adopter customers -- and these systems are deployed out in the real world -- do all the big data intensive, data heavy problems with it. So, whether it's logistics and supply chain management or natural gas and resource extraction, financial services, pharmaceuticals, bioinformatics, those are the topics right now, but that's not a killer app. And I understand what you're asking.

CA: C'mon, c'mon. Martial arts, games. C'mon. (Laughter) John, thank you for making science-fiction real.

JU: It's been a great pleasure. Thank you to you all.

(Applause)