Clay Shirky on institutions vs. collaboration

How do groups get anything done? Right? How do you organize a group of individuals so that the output of the group is something coherent and of lasting value, instead of just being chaos? And the economic framing of that problem is called coordination costs. And a coordination cost is essentially all of the financial or institutional difficulties in arranging group output. And we've had a classic answer for coordination costs, which is, if you want to coordinate the work of a group of people, you start an institution, right? You raise some resources. You found something. It can be private or public. It can be for profit or not profit. It can be large or small. But you get these resources together. You found an institution, and you use the institution to coordinate the activities of the group.

More recently, because the cost of letting groups communicate with each other has fallen through the floor -- and communication costs are one of the big inputs to coordination -- there has been a second answer, which is to put the cooperation into the infrastructure, to design systems that coordinate the output of the group as a by-product of the operating of the system, without regard to institutional models. So, that's what I want to talk about today. I'm going to illustrate it with some fairly concrete examples, but always pointing to the broader themes.

So, I'm going to start by trying to answer a question that I know each of you will have asked yourself at some point or other, and which the Internet is purpose-built to answer, which is, where can I get a picture of a roller-skating mermaid? So, in New York City, on the first Saturday of every summer, Coney Island, our local, charmingly run-down amusement park, hosts the Mermaid Parade. It's an amateur parade; people come from all over the city; people get all dressed up. Some people get less dressed up. Young and old, dancing in the streets. Colorful characters, and a good time is had by all. And what I want to call your attention to is not the Mermaid Parade itself, charming though it is, but rather to these photos. I didn't take them. How did I get them? And the answer is: I got them from Flickr.

Flickr is a photo-sharing service that allows people to take photos, upload them, share them over the Web and so forth. Recently, Flickr has added an additional function called tagging. Tagging was pioneered by Delicious and Joshua Schachter. Delicious is a social bookmarking service. Tagging is a cooperative infrastructure answer to classification. Right? If I had given this talk last year, I couldn't do what I just did, because I couldn't have found those photos. But instead of saying, we need to hire a professional class of librarians to organize these photos once they're uploaded, Flickr simply turned over to the users the ability to characterize the photos. So, I was able to go in and draw down photos that had been tagged "Mermaid Parade." There were 3,100 photos taken by 118 photographers, all aggregated and then put under this nice, neat name, shown in reverse chronological order. And I was then able to go and retrieve them to give you that little slideshow.

Now, what hard problem is being solved here? And it's -- in the most schematic possible view, it's a coordination problem, right? There are a large number of people on the Internet, a very small fraction of them have photos of the Mermaid Parade. How do we get those people together to contribute that work? The classic answer is to form an institution, right? To draw those people into some prearranged structure that has explicit goals. And I want to call your attention to some of the side effects of going the institutional route.

First of all, when you form an institution, you take on a management problem, right? No good just hiring employees, you also have to hire other employees to manage those employees and to enforce the goals of the institution and so forth. Secondly, you have to bring structure into place. Right? You have to have economic structure. You have to have legal structure. You have to have physical structure. And that creates additional costs. Third, forming an institution is inherently exclusionary. You notice we haven't got everybody who has a photo. You can't hire everyone in a company, right? You can't recruit everyone into a governmental organization. You have to exclude some people. And fourth, as a result of that exclusion, you end up with a professional class. Look at the change here. We've gone from people with photos to photographers. Right? We've created a professional class of photographers whose goal is to go out and photograph the Mermaid Parade, or whatever else they're sent out to photograph.

When you build cooperation into the infrastructure, which is the Flickr answer, you can leave the people where they are and you take the problem to the individuals, rather than moving the individuals to the problem. You arrange the coordination in the group, and by doing that you get the same outcome, without the institutional difficulties. You lose the institutional imperative. You lose the right to shape people's work when it's volunteer effort, but you also shed the institutional cost, which gives you greater flexibility. What Flickr does is it replaces planning with coordination. And this is a general aspect of these cooperative systems.

Right. You'll have experienced this in your life whenever you bought your first mobile phone, and you stopped making plans. You just said, "I'll call you when I get there." "Call me when you get off work." Right? That is a point-to-point replacement of coordination with planning. Right. We're now able to do that kind of thing with groups. To say instead of, we must make an advance plan, we must have a five-year projection of where the Wikipedia is going to be, or whatever, you can just say, let's coordinate the group effort, and let's deal with it as we go, because we're now well-enough coordinated that we don't have to take on the problems of deciding in advance what to do.

So here's another example. This one's somewhat more somber. These are photos on Flickr tagged "Iraq." And everything that was hard about the coordination cost with the Mermaid Parade is even harder here. There are more pictures. There are more photographers. It's taken over a wider geographic area. The photos are spread out over a longer period of time. And worst of all, that figure at the bottom, approximately ten photos per photographer, is a lie. It's mathematically true, but it doesn't really talk about anything important -- because in these systems, the average isn't really what matters.

What matters is this. This is a graph of photographs tagged Iraq as taken by the 529 photographers who contributed the 5,445 photos. And it's ranked in order of number of photos taken per photographer. You can see here, over at the end, our most prolific photographer has taken around 350 photos, and you can see there's a few people who have taken hundreds of photos. Then there's dozens of people who've taken dozens of photos. And by the time we get around here, we get ten or fewer photos, and then there's this long, flat tail. And by the time you get to the middle, you've got hundreds of people who have contributed only one photo each.

This is called a power-law distribution. It appears often in unconstrained social systems where people are allowed to contribute as much or as little as they like -- this is often what you get. Right? The math behind the power-law distribution is that whatever's in the nth position is doing about one-nth of whatever's being measured, relative to the person in the first position. So, we'd expect the tenth most prolific photographer to have contributed about a tenth of the photos, and the hundredth most prolific photographer to have contributed only about a hundred as many photos as the most prolific photographer did. So, the head of the curve can be sharper or flatter. But that basic math accounts both for the steep slope and for the long, flat tail.

And curiously, in these systems, as they grow larger, the systems don't converge; they diverge more. In bigger systems, the head gets bigger and the tail gets longer, so the imbalance increases. You can see the curve is obviously heavily left-weighted. Here's how heavily: if you take the top 10 percent of photographers contributing to this system, they account for three quarters of the photos taken -- just the top 10 percent most prolific photographers. If you go down to five percent, you're still accounting for 60 percent of the photos. If you go down to one percent, exclude 99 percent of the group effort, you're still accounting for almost a quarter of the photos. And because of this left weighting, the average is actually here, way to the left. And that sounds strange to our ears, but what ends up happening is that 80 percent of the contributors have contributed a below-average amount. That sounds strange because we expect average and middle to be about the same, but they're not at all.

This is the math underlying the 80/20 rule. Right? Whenever you hear anybody talking about the 80/20 rule, this is what's going on. Right? 20 percent of the merchandise accounts for 80 percent of the revenue, 20 percent of the users use 80 percent of the resources -- this is the shape people are talking about when that happens. Institutions only have two tools: carrots and sticks. And the 80 percent zone is a no-carrot and no-stick zone. The costs of running the institution mean that you cannot take on the work of those people easily in an institutional frame. The institutional model always pushes leftwards, treating these people as employees. The institutional response is, I can get 75 percent of the value for 10 percent of the hires -- great, that's what I'll do. The cooperative infrastructure model says, why do you want to give up a quarter of the value? If your system is designed so that you have to give up a quarter of the value, re-engineer the system. Don't take on the cost that prevents you from getting to the contributions of these people. Build the system so that anybody can contribute at any amount.

So the coordination response asks not, how are these people as employees, but rather, what is their contribution like? Right? We have over here Psycho Milt, a Flickr user, who has contributed one, and only one, photo titled "Iraq." And here's the photo. Right. Labeled, "Bad Day at Work." Right? So the question is, do you want that photo? Yes or no. The question is not, is Psycho Milt a good employee?

And the tension here is between institution as enabler and institution as obstacle. When you're dealing with the left-hand edge of one of these distributions, when you're dealing with the people who spend a lot of time producing a lot of the material you want, that's an institution-as-enabler world. You can hire those people as employees, you can coordinate their work and you can get some output. But when you're down here, where the Psycho Milts of the world are adding one photo at a time, that's institution as obstacle.

Institutions hate being told they're obstacles. One of the first things that happens when you institutionalize a problem is that the first goal of the institution immediately shifts from whatever the nominal goal was to self-preservation. And the actual goal of the institution goes to two through n. Right? So, when institutions are told they are obstacles, and that there are other ways of coordinating the value, they go through something a little bit like the Kubler-Ross stages -- (Laughter) -- of reaction, being told you have a fatal illness: denial, anger, bargaining, acceptance. Most of the cooperative systems we've seen haven't been around long enough to have gotten to the acceptance phase.

Many, many institutions are still in denial, but we're seeing recently a lot of both anger and bargaining. There's a wonderful, small example going on right now. In France, a bus company is suing people for forming a carpool, right, because the fact that they have coordinated themselves to create cooperative value is depriving them of revenue. You can follow this in the Guardian. It's actually quite entertaining.

The bigger question is, what do you do about the value down here? Right? How do you capture that? And institutions, as I've said, are prevented from capturing that. Steve Ballmer, now CEO of Microsoft, was criticizing Linux a couple of years ago, and he said, "Oh, this business of thousands of programmers contributing to Linux, this is a myth. We've looked at who's contributed to Linux, and most of the patches have been produced by programmers who've only done one thing." Right? You can hear this distribution under that complaint. And you can see why, from Ballmer's point of view, that's a bad idea, right? We hired this programmer, he came in, he drank our Cokes and played Foosball for three years and he had one idea. (Laughter) Right? Bad hire. Right? (Laughter)

The Psycho Milt question is, was it a good idea? What if it was a security patch? What if it was a security patch for a buffer overflow exploit, of which Windows has not some, [but] several? Do you want that patch, right? The fact that a single programmer can, without having to move into a professional relation to an institution, improve Linux once and never be seen from again, should terrify Ballmer. Because this kind of value is unreachable in classic institutional frameworks, but is part of cooperative systems of open-source software, of file sharing,

of the Wikipedia. I've used a lot of examples from Flickr, but there are actually stories about this from all over. Meetup, a service founded so that users could find people in their local area who share their interests and affinities and actually have a real-world meeting offline in a cafe or a pub or what have you. When Scott Heiferman founded Meetup, he thought it would be used for, you know, train spotters and cat fanciers -- classic affinity groups. The inventors don't know what the invention is. Number one group on Meetup right now, most chapters in most cities with most members, most active? Stay-at-home moms. Right? In the suburbanized, dual-income United States, stay-at-home moms are actually missing the social infrastructure that comes from extended family and local, small-scale neighborhoods. So they're reinventing it, using these tools. Meetup is the platform, but the value here is in social infrastructure. If you want to know what technology is going to change the world, don't pay attention to 13-year-old boys -- pay attention to young mothers, because they have got not an ounce of support for technology that doesn't materially make their lives better. This is so much more important than Xbox, but it's a lot less glitzy.

I think this is a revolution. I think that this is a really profound change in the way human affairs are arranged. And I use that word advisedly. It's a revolution in that it's a change in equilibrium. It's a whole new way of doing things, which includes new downsides. In the United States right now, a woman named Judith Miller is in jail for not having given to a Federal Grand Jury her sources -- she's a reporter for the New York Times -- her sources, in a very abstract and hard-to-follow case. And journalists are in the street rallying to improve the shield laws. The shield laws are our laws -- pretty much a patchwork of state laws -- that prevent a journalist from having to betray a source. This is happening, however, against the background of the rise of Web logging. Web logging is a classic example of mass amateurization. It has de-professionalized publishing. Want to publish globally anything you think today? It is a one-button operation that you can do for free. That has sent the professional class of publishing down into the ranks of mass amateurization. And so the shield law, as much as we want it -- we want a professional class of truth-tellers -- it is becoming increasingly incoherent, because the institution is becoming incoherent. There are people in the States right now tying themselves into knots, trying to figure out whether or not bloggers are journalists. And the answer to that question is, it doesn't matter, because that's not the right question. Journalism was an answer to an even more important question, which is, how will society be informed? How will they share ideas and opinions? And if there is an answer to that that happens outside the professional framework of journalism, it makes no sense to take a professional metaphor and apply it to this distributed class. So as much as we want the shield laws, the background -- the institution to which they were attached -- is becoming incoherent.

Here's another example. Pro-ana, the pro-ana groups. These are groups of teenage girls who have taken on Web logs, bulletin boards, other kinds of cooperative infrastructure, and have used it to set up support groups for remaining anorexic by choice. They post pictures of thin models, which they call "thinspiration." They have little slogans, like "Salvation through Starvation." They even have Lance Armstrong-style bracelets, these red bracelets, which signify, in the small group, I am trying to maintain my eating disorder. They trade tips, like, if you feel like eating something, clean a toilet or the litter box. The feeling will pass.

We're used to support groups being beneficial. We have an attitude that support groups are inherently beneficial. But it turns out that the logic of the support group is value neutral. A support group is simply a small group that wants to maintain a way of living in the context of a larger group. Now, when the larger group is a bunch of drunks, and the small group wants to stay sober, then we think, that's a great support group. But when the small group is teenage girls who want to stay anorexic by choice, then we're horrified. What's happened is that the normative goals of the support groups that we're used to, came from the institutions that were framing them, and not from the infrastructure. Once the infrastructure becomes generically available, the logic of the support group has been revealed to be accessible to anyone, including people pursuing these kinds of goals.

So, there are significant downsides to these changes as well as upsides. And of course, in the current environment, one need allude only lightly to the work of non-state actors trying to influence global affairs, and taking advantage of these. This is a social map of the hijackers and their associates who perpetrated the 9/11 attack. It was produced by analyzing their communications patterns using a lot of these tools. And doubtless the intelligence communities of the world are doing the same work today for the attacks of last week.

Now, this is the part of the talk where I tell you what's going to come as a result of all of this, but I'm running out of time, which is good, because I don't know. (Laughter) Right. As with the printing press, if it's really a revolution, it doesn't take us from Point A to Point B. It takes us from Point A to chaos. The printing press precipitated 200 years of chaos, moving from a world where the Catholic Church was the sort of organizing political force to the Treaty of Westphalia, when we finally knew what the new unit was: the nation state.

Now, I'm not predicting 200 years of chaos as a result of this. 50. 50 years in which loosely coordinated groups are going to be given increasingly high leverage, and the more those groups forego traditional institutional imperatives -- like deciding in advance what's going to happen, or the profit motive -- the more leverage they'll get. And institutions are going to come under an increasing degree of pressure, and the more rigidly managed, and the more they rely on information monopolies, the greater the pressure is going to be. And that's going to happen one arena at a time, one institution at a time. The forces are general, but the results are going to be specific.

And so the point here is not, "This is wonderful," or "We're going to see a transition from only institutions to only cooperative framework." It's going to be much more complicated than that. But the point is that it's going to be a massive readjustment. And since we can see it in advance and know it's coming, my argument is essentially: we might as well get good at it. Thank you very much. (Applause)