Aaron Koblin: Artfully visualizing our humanity

So I think data can actually make us more human. We're collecting and creating all kinds of data about how we're living our lives, and it's enabling us to tell some amazing stories. Recently, a wise media theorist Tweeted, "The 19th century culture was defined by the novel, the 20th century culture was defined by the cinema, and the culture of the 21st century will be defined by the interface." And I believe this is going to prove true. Our lives are being driven by data, and the presentation of that data is an opportunity for us to make some amazing interfaces that tell great stories. So I'm going to show you a few of the projects that I've been working on over the last couple years that reflect on our lives and our systems.

This is a project called Flight Patterns. What you're looking at is airplane traffic over North America for a 24-hour period. As you see, everything starts to fade to black, and you see people going to sleep. Followed by that, you see on the West coast planes moving across, the red-eye flights to the East coast. And you'll see everybody waking up on the East coast, followed by European flights coming in the upper right-hand corner. Everybody's moving from the East coast to the West coast. You see San Francisco and Los Angeles start to make their journeys down to Hawaii in the lower left-hand corner.

I think it's one thing to say there's 140,000 planes being monitored by the federal government at any one time, and it's another thing to see that system as it ebbs and flows. This is a time-lapse image of that exact same data, but I've color-coded it by type, so you can see the diversity of aircraft that are in the skies above us. And I started making these, and I put them into Google Maps and allow you to zoom in and see individual airports and the patterns that are occurring there. So here we can see the white represents low altitudes, and the blue are higher altitudes. And you can zoom in. This is taking a look at Atlanta. You can see this is a major shipping airport, and there's all kinds of activity there. You can also toggle between altitude for model and manufacturer. See again, the diversity. And you can scroll around and see some of the different airports and the different patterns that they have. This is scrolling up the East coast. You can see some of the chaos that's happening in New York with the air traffic controllers having to deal with all those major airports next to each other.

So zooming back out real quick, we see, again, the U.S. -- you get Florida down in the right-hand corner. Moving across to the West coast, you see San Francisco and Los Angeles -- big low-traffic zones across Nevada and Arizona. And that's us down there in L.A. and Long Beach on the bottom. I started taking a look as well at different perimeters, because you can choose what you want to pull out from the data. This is looking at ascending versus descending flights. And you can see, over time, the ways the airports change. You see the holding patterns that start to develop in the bottom of the screen. And you can see, eventually the airport actually flips directions.

So this is another project that I worked on with the Sensible Cities Lab at MIT. This is visualizing international communications. So it's how New York communicates with other international cities. And we set this up as a live globe in the Museum of Modern Art in New York for the Design the Elastic Mind exhibition. And it had a live feed with a 24-hour offset, so you could see the changing relationship and some demographic info coming through AT&T's data and revealing itself. This is another project I worked on with Sensible Cities Lab and CurrentCity.org. And it's visualizing SMS messages being sent in the city of Amsterdam. So you're seeing the daily ebb and flow of people sending SMS messages from different parts of the city, until we approach New Year's Eve, where everybody says, "Happy New Year!"

(Laughter)

So this is an interactive tool that you can move around and see different parts of the city. This is looking at another event. This is called Queen's Day. So again, you get this daily ebb and flow of people sending SMS messages from different parts of the city. And then you're going to see people start to gather in the center of the city to celebrate the night before, which happens right here. And then you can see people celebrating the next day. And you can pause it and step back and forth and see different phases.

So now on to something completely different. Some of you may recognize this. This is Baron Wolfgang von Kempelen's mechanical chess playing machine. And it's this amazing robot that plays chess extremely well, except for one thing: it's not a robot at all. There's actually a legless man that sits in that box and controls this chess player. This was the inspiration for a web service by Amazon called the Mechanical Turk -- named after this guy. And it's based on the premise that there are certain things that are easy for people, but really difficult for computers. So they made this web service and said, "Any programmer can write a piece of software and tap into the minds of thousands of people." The nerdy side of me thought, "Wow, this is amazing. I can tap into thousands of people's minds." And the other nerdy side of me thought, "This is horrible. This is completely bizarre. What does this mean for the future of mankind, where we're all plugged into this borg?" I was probably being a little extreme. But what does this mean when we have no context for what it is that we're working on, and we're just doing these little labors?

So I created this drawing tool. I asked people to draw a sheep facing to the left. And I said, "I'll pay you two cents for your contribution." And I started collecting sheep. And I collected a lot, a lot of different sheep. Lots of sheep. I took the first 10,000 sheep that I collected, and I put them on a website called TheSheepMarket.com where you can actually buy collections of 20 sheep. You can't pick individual sheep, but you can buy a single plate block of stamps as a commodity. And juxtaposed against this grid, you see actually, by rolling over each individual one, the humanity behind this hugely mechanical process. I think there's something really interesting to watching people as they go through this creative toil -- something we can all relate to, this creative process of trying to come up with something from nothing. I think it was really interesting to juxtapose this humanity versus this massive distributed grid. Kind of amazing what some people did.

So here's a few statistics from the project. Approximate collection rate of 11 sheep per hour, which would make a working wage of 69 cents per hour. There were 662 rejected sheep that didn't meet "sheep-like" criteria and were thrown out of the flock. (Laughter) The amount of time spent drawing ranged from four seconds to 46 minutes. That gives you an idea of the different types of motivations and dedication. And there were 7,599 people that contributed to the project, or were unique IP addresses -- so about how many people contributed. But only one of them out of the 7,599 said this. (Laughter) Which I was pretty surprised by. I expected people to be wondering, "Why did I draw a sheep?" And I think it's a pretty valid question.

And there's a lot of reasons why I chose sheep. Sheep were the first animal to be raised from mechanically processed byproducts, the first to be selectively bred for production traits, the first animal to be cloned. Obviously, we think of sheep as followers. And there's this reference to "Le Petit Prince" where the narrator asks the prince to draw a sheep. He draws sheep after sheep. The narrator's only appeased when he draws a box. And he says, "It's not about a scientific rendering of a sheep. It's about your own interpretation and doing something different." And I like that.

So this is a clip from Charlie Chaplin's "Modern Times." It's showing Charlie Chaplin dealing with some of the major changes during the Industrial Revolution. So there were no longer shoe makers, but now there are people slapping soles on people's shoes. And the whole idea of one's relationship to their work changed a lot. So I thought this was an interesting clip to divide into 16 pieces and feed into the Mechanical Turk with a drawing tool. This basically allowed -- what you see on the left side is the original frame, and on the right side you see that frame as interpreted by 16 people who have no idea what it is they're doing.

And this was the inspiration for a project that I worked on with my friend Takashi Kawashima. We decided to use the Mechanical Turk for exactly what it was meant for, which is making money. So we took a hundred dollar bill and divided it into 10,000 teeny pieces, and we fed those into the Mechanical Turk. We asked people to draw what it was that they saw. But here there was no sheep-like criteria. People, if they drew a stick figure or a smiley face, it actually made it into the bill. So what you see is actually a representation of how well people did what it was they were asked to do. So we took these hundred dollar bills, and we put them on a website called TenThousandsCents.com, where you can browse through and see all the individual contributions. And you can also trade real hundred-dollar bills for fake hundred-dollar bills and make a donation to the Hundred Dollar Laptop Project, which is now known as One Laptop Per Child. This is again showing all the different contributions. You see some people did beautiful stipple renderings, like this one on top -- spent a long time making realistic versions. And other people would draw stick figures or smiley faces. Here on the right-hand side in the middle you see this one guy writing, "$0.01!!! Really?" That's all I'm getting paid for this?

(Laughter)

So the last Mechanical Turk project I'm going to talk to you about is called Bicycle Built for 2000. This is a collaboration with my friend Daniel Massey. You may recognize these two guys. This is Max Mathews and John Kelly from Bell Labs in the '60s, where they created the song "Daisy Bell," which was the world's first singing computer. You may recognize it from "2001: A Space Odyssey." When HAL's dying at the end of the film he starts singing this song, as a reference to when computers became human. So we resynthesized this song. This is what that sounded like. We broke down all the individual notes in the singing as well as the phonemes in the singing.

Daisy Bell: ♫ Daisy, Daisy ... ♫

Aaron Koblin: And we took all of those individual pieces, and we fed them into another Turk request. This is what it would look like if you went to the site. You type in your code, but you first test your mic. You'd be fed a simple audio clip. (Honk) And then you'd do your best to recreate that with your own voice. After previewing it and confirming it's what you submitted, you could submit it into the Mechanical Turk with no other context. And this is what we first got back from the very first set of submissions.

Recording: ♫ Daisy, Daisy ♫ ♫ give me your answer do ♫ ♫ I'm half crazy ♫ ♫ all for the love of you ♫ ♫ It can't be a stylish marriage ♫ ♫ I can't afford a carriage ♫ ♫ But you'll look sweet upon the seat ♫ ♫ of a bicycle built for two ♫

AK: So James Surowieki has this idea of the wisdom of crowds, that says that a whole bunch of people are smarter than any individual. We wanted to see how this applies to collaborative, distributed music making, where nobody has any idea what it is they're working on. So if you go to the BicycleBuiltforTwoThousand.com you can actually hear what all this sounds like together. I'm sorry for this.

(Noise)

Chorus: ♫ Daisy, Daisy ♫ ♫ Give me your answer do ♫ ♫ I'm half crazy ♫ ♫ all for the love of you ♫ ♫ It can't be a stylish marriage ♫ ♫ I can't afford a carriage ♫ ♫ But you'd look sweet upon the seat ♫ ♫ of a bicycle built for two ♫

AK: So stepping back for a quick second, when I was at UCLA going to grad school, I was also working at a place called the Center for Embedded Network Sensing. And I was writing software to visualize laser scanners. So basically motion through 3D space. And this was seen by a director in L.A. named James Frost who said, "Wait a minute. You mean we can shoot a music video without actually using any video?" So we did exactly that. We made a music video for one of my favorite bands, Radiohead. And I think one of my favorite parts of this project was not just shooting a video with lasers, but we also open sourced it, and we made it released as a Google Code project, where people could download a bunch of the data and some source code to build their own versions of it. And people were making some amazing things. This is actually two of my favorites: the pin-board Thom Yorke and a LEGO Thom Yorke. A whole YouTube channel of people submitting really interesting content. More recently, somebody even 3D-printed Thom Yorke's head, which is a little creepy, but pretty cool.

So with everybody making so much amazing stuff and actually understanding what it was they were working on, I was really interested in trying to make a collaborative project where people were working together to build something. And I met a music video director named Chris Milk. And we started bouncing around ideas to make a collaborative music video project. But we knew we really needed the right person to kind of rally behind and build something for. So we put the idea on the back burner for a few months. And he ended up talking to Rick Rubin, who was finishing up Johnny Cash's final album called "Ain't No Grave." The lyrics to the leading track are "Ain't no grave can hold my body down." So we thought this was the perfect project to build a collaborative memorial and a virtual resurrection for Johnny Cash.

So I teamed up with my good friend Ricardo Cabello, also known as Mr. doob, who's a much better programmer than I am, and he made this amazing Flash drawing tool. As you know, an animation is a series of images. So what we did was cross-cut a bunch of archival footage of Johnny Cash, and at eight frames a second, we allowed individuals to draw a single frame that would get woven into this dynamically changing music video. So I don't have time to play the entire thing for you, but I want to show you two short clips. One is the beginning of the music video. And that's going to be followed by a short clip of people who have already contributed to the project talking about it briefly.

(Music)

(Video) Johnny Cash: ♫ There ain't no grave ♫ ♫ can hold my body down ♫ ♫ There ain't no grave ♫ ♫ can hold body down ♫ ♫ When I hear the trumpet sound ♫ ♫ I'm going to ride right out of the ground ♫ ♫ Ain't no grave ♫ ♫ can hold my body ... ♫

(Applause)

AK: What better way to pay tribute to the man than to make something for one of his songs.

Collaborator: I felt really sad when he died. And I just thought it'd be wonderful, it'd be really nice to contribute something to his memory.

Collaborator Two: It really allows this last recording of his to be a living, breathing memorial.

Collaborator Three: For all of the frames to be drawn by fans, each individual frame, it's got a very powerful feeling to it.

Collaborator Four: I've seen everybody from Japan, Venezuela, to the States, to Knoxville, Tennessee.

Collaborator Five: As much as is different from frame to frame, it really is personal. Collaborator Six: Watching the video in my room, I could see me not understanding at the beginning of it. And I just worked and worked through problems, until my little wee battles that I was fighting within the picture all began to resolve themselves. You can actually see the point when I know what I'm doing, and a lot of light and dark comes into it. And in a weird way, that's what I actually like about Johnny Cash's music as well. It's the sum total of his life, all the things that had happened -- the bad things, the good things. You're hearing a person's life.

AK: So if you go to the website JohnnyCashProject.com, what you'll see is the video playing above. And below it are all the individual frames that people have been submitting to the project. So this isn't finished at all, but it's an ongoing project where people can continue to collaborate. If you roll over any one of those individual thumbnails, you can see the person who drew that individual thumbnail and where they were located. And if you find one that you're interested in, you can actually click on it and open up an information panel where you're able to rate that frame, which helps it bubble up to the top. And you can also see the way that it was drawn. Again, you can get the playback and personal contribution. In addition to that, it's listed, the artist's name, the location, how long they spent drawing it. And you can pick a style. So this one was tagged "Abstract." But there's a bunch of different styles. And you can sort the video a number of different ways. You can say, "I want to see the pointillist version or the sketchy version or the realistic version. And then this is, again, the abstract version, which ends up getting a little bit crazy.

So the last project I want to talk to you about is another collaboration with Chris Milk. And this is called "The Wilderness Downtown." It's an online music video for the Arcade Fire. Chris and I were really amazed by the potential now with modern web browsers, where you have HTML5 audio and video and the power of JavaScript to render amazingly fast. And we wanted to push the idea of the music video that was meant for the Web beyond the four-by-three or sixteen-by-nine window and try to make it play out and choreograph throughout the screen. But most importantly, I think, we really wanted to make an experience that was unlike the Johnny Cash Project, where you had a small group of people spending a lot of time to contribute something for everyone. What if we had a very low commitment, but delivered something individually unique to each person who contributed?

So the project starts off by asking you to enter the address of the home where you grew up. And you type in the address -- it actually creates a music video specifically for you, pulling in Google maps and Streetview images into the experience itself. So this should really be seen at home with you typing in your own address, but I'm going to give you a little preview of what you can expect.

(Video) Win Butler: ♫ Now our lives are changing fast ♫ ♫ Now our lives are changing fast ♫ ♫ Hope that something pure can last ♫ ♫ Hope that something pure can last ♫ ♫ Ooh we used to wait ♫ ♫ Ooh we used to wait ♫ ♫ Ooh we used to wait ♫ ♫ Sometimes it never came ♫ ♫ Sometimes it never came ♫ ♫ Still moving through the pain ♫ ♫ We used to wait for it ♫ ♫ We used to wait for it ♫ ♫ We used to wait for it ♫

AK: So I think, if there's one thing to take away from my talk today, it's that an interface can be a powerful narrative device. And as we collect more and more personally and socially relevant data, we have an opportunity, and maybe even an obligation, to maintain the humanity and tell some amazing stories as we explore and collaborate together.

Thanks a lot.

(Applause)