David Merrill: Toy tiles that talk to each other

I want to start out by asking you to think back to when you were a kid, playing with blocks. As you figured out how to reach out and grasp, pick them up and move them around, you were actually learning how to think and solve problems by understanding and manipulating spatial relationships. Spatial reasoning is deeply connected to how we understand a lot of the world around us.

So, as a computer scientist inspired by this utility of our interactions with physical objects -- along with my adviser Pattie, and my collaborator Jeevan Kalanithi -- I started to wonder -- what if when we used a computer, instead of having this one mouse cursor that was a like a digital fingertip moving around a flat desktop, what if we could reach in with both hands and grasp information physically, arranging it the way we wanted? This question was so compelling that we decided to explore the answer, by building Siftables.

In a nutshell, a Siftable is an interactive computer the size of a cookie. They're able to be moved around by hand, they can sense each other, they can sense their motion, and they have a screen and a wireless radio. Most importantly, they're physical, so like the blocks, you can move them just by reaching out and grasping. And Siftables are an example of a new ecosystem of tools for manipulating digital information. And as these tools become more physical, more aware of their motion, aware of each other, and aware of the nuance of how we move them, we can start to explore some new and fun interaction styles. So, I'm going to start with some simple examples.

This Siftable is configured to show video, and if I tilt it in one direction, it'll roll the video this way; if I tilt it the other way it rolls it backwards. And these interactive portraits are aware of each other. So if I put them next to each other, they get interested. If they get surrounded, they notice that too, they might get a little flustered. And they can also sense their motion and tilt.

One of the interesting implications on interaction, we started to realize, was that we could use everyday gestures on data, like pouring a color the way we might pour a liquid. So in this case, we've got three Siftables configured to be paint buckets and I can use them to pour color into that central one, where they get mixed. If we overshoot, we can pour a little bit back.

There are also some neat possibilities for education, like language, math and logic games where we want to give people the ability to try things quickly, and view the results immediately. So here I'm -- (Applause) This is a Fibonacci sequence that I'm making with a simple equation program.

Here we have a word game that's kind of like a mash-up between Scrabble and Boggle. Basically, in every round you get a randomly assigned letter on each Siftable, and as you try to make words it checks against a dictionary. Then, after about 30 seconds, it reshuffles, and you have a new set of letters and new possibilities to try.

(Applause) Thank you. (Applause)

So these are some kids that came on a field trip to the Media Lab, and I managed to get them to try it out, and shoot a video. They really loved it. And, one of the interesting things about this kind of application is that you don't have to give people many instructions. All you have to say is, "Make words," and they know exactly what to do.

So here's another few people trying it out. That's our youngest beta tester, down there on the right. Turns out, all he wanted to do was to stack the Siftables up. So to him, they were just blocks. Now, this is an interactive cartoon application. And we wanted to build a learning tool for language learners. And this is Felix, actually. And he can bring new characters into the scene, just by lifting the Siftables off the table that have that character shown on them. Here, he's bringing the sun out.

Video: The sun is rising.

David Merrill: Now he's brought a tractor into the scene.

Video: The orange tractor. Good job! Yeah!

DM: So by shaking the Siftables and putting them next to each other he can make the characters interact --

Video: Woof!

DM: inventing his own narrative.

Video: Hello!

DM: It's an open-ended story, and he gets to decide how it unfolds.

Video: Fly away, cat.

DM: So, the last example I have time to show you today is a music sequencing and live performance tool that we've built recently, in which Siftables act as sounds like lead, bass and drums. Each of these has four different variations, you get to choose which one you want to use. And you can inject these sounds into a sequence that you can assemble into the pattern that you want. And you inject it by just bumping up the sound Siftable against a sequence Siftable. There are effects that you can control live, like reverb and filter. You attach it to a particular sound and then tilt to adjust it. And then, overall effects like tempo and volume that apply to the entire sequence. So let's have a look.

Video: (Music)

DM: We'll start by putting a lead into two sequence Siftables, arrange them into a series, extend it, add a little more lead. Now I put a bass line in.

Video: (Music) DM: Now I'll put some percussion in. Video: (Music)

DM: And now I'll attach the filter to the drums, so I can control the effect live.

Video: (Music)

DM: I can speed up the whole sequence by tilting the tempo one way or the other.

Video: (Music)

DM: And now I'll attach the filter to the bass for some more expression.

Video: (Music)

DM: I can rearrange the sequence while it plays. So I don't have to plan it out in advance, but I can improvise, making it longer or shorter as I go. And now, finally, I can fade the whole sequence out using the volume Siftable, tilted to the left.

(Applause) Thank you.

So, as you can see, my passion is for making new human-computer interfaces that are a better match to the ways our brains and bodies work. And today, I had time to show you one point in this new design space, and a few of the possibilities that we're working to bring out of the laboratory. So the thought I want to leave you with is that we're on the cusp of this new generation of tools for interacting with digital media that are going to bring information into our world on our terms. Thank you very much. I look forward to talking with all of you.

(Applause)