Tim Harford: Trial, error and the God complex

It's the Second World War. A German prison camp. And this man, Archie Cochrane, is a prisoner of war and a doctor, and he has a problem. The problem is that the men under his care are suffering from an excruciating and debilitating condition that Archie doesn't really understand. The symptoms are this horrible swelling up of fluids under the skin. But he doesn't know whether it's an infection, whether it's to do with malnutrition. He doesn't know how to cure it. And he's operating in a hostile environment. And people do terrible things in wars. The German camp guards, they've got bored. They've taken to just firing into the prison camp at random for fun. On one particular occasion, one of the guards threw a grenade into the prisoners' lavatory while it was full of prisoners. He said he heard suspicious laughter. And Archie Cochrane, as the camp doctor, was one of the first men in to clear up the mess. And one more thing: Archie was suffering from this illness himself.

So the situation seemed pretty desperate. But Archie Cochrane was a resourceful person. He'd already smuggled vitamin C into the camp, and now he managed to get hold of supplies of marmite on the black market. Now some of you will be wondering what marmite is. Marmite is a breakfast spread beloved of the British. It looks like crude oil. It tastes ... zesty. And importantly, it's a rich source of vitamin B12. So Archie splits the men under his care as best he can into two equal groups. He gives half of them vitamin C. He gives half of them vitamin B12. He very carefully and meticulously notes his results in an exercise book. And after just a few days, it becomes clear that whatever is causing this illness, marmite is the cure.

So Cochrane then goes to the Germans who are running the prison camp. Now you've got to imagine at the moment -- forget this photo, imagine this guy with this long ginger beard and this shock of red hair. He hasn't been able to shave -- a sort of Billy Connolly figure. Cochrane, he starts ranting at these Germans in this Scottish accent -- in fluent German, by the way, but in a Scottish accent -- and explains to them how German culture was the culture that gave Schiller and Goethe to the world. And he can't understand how this barbarism can be tolerated, and he vents his frustrations. And then he goes back to his quarters, breaks down and weeps because he's convinced that the situation is hopeless. But a young German doctor picks up Archie Cochrane's exercise book and says to his colleagues, "This evidence is incontrovertible. If we don't supply vitamins to the prisoners, it's a war crime." And the next morning, supplies of vitamin B12 are delivered to the camp, and the prisoners begin to recover.

Now I'm not telling you this story because I think Archie Cochrane is a dude, although Archie Cochrane is a dude. I'm not even telling you the story because I think we should be running more carefully controlled randomized trials in all aspects of public policy, although I think that would also be completely awesome. I'm telling you this story because Archie Cochrane, all his life, fought against a terrible affliction, and he realized it was debilitating to individuals and it was corrosive to societies. And he had a name for it. He called it the God complex. Now I can describe the symptoms of the God complex very, very easily. So the symptoms of the complex are, no matter how complicated the problem, you have an absolutely overwhelming belief that you are infallibly right in your solution.

Now Archie was a doctor, so he hung around with doctors a lot. And doctors suffer from the God complex a lot. Now I'm an economist, I'm not a doctor, but I see the God complex around me all the time in my fellow economists. I see it in our business leaders. I see it in the politicians we vote for -- people who, in the face of an incredibly complicated world, are nevertheless absolutely convinced that they understand the way that the world works. And you know, with the future billions that we've been hearing about, the world is simply far too complex to understand in that way.

Well let me give you an example. Imagine for a moment that, instead of Tim Harford in front of you, there was Hans Rosling presenting his graphs. You know Hans: the Mick Jagger of TED. (Laughter) And he'd be showing you these amazing statistics, these amazing animations. And they are brilliant; it's wonderful work. But a typical Hans Rosling graph: think for a moment, not what it shows, but think instead about what it leaves out. So it'll show you GDP per capita, population, longevity, that's about it. So three pieces of data for each country -- three pieces of data. Three pieces of data is nothing. I mean, have a look at this graph.

This is produced by the physicist Cesar Hidalgo. He's at MIT. Now you won't be able to understand a word of it, but this is what it looks like. Cesar has trolled the database of over 5,000 different products, and he's used techniques of network analysis to interrogate this database and to graph relationships between the different products. And it's wonderful, wonderful work. You show all these interconnections, all these interrelations. And I think it'll be profoundly useful in understanding how it is that economies grow. Brilliant work. Cesar and I tried to write a piece for The New York Times Magazine explaining how this works. And what we learned is Cesar's work is far too good to explain in The New York Times Magazine.

Five thousand products -- that's still nothing. Five thousand products -- imagine counting every product category in Cesar Hidalgo's data. Imagine you had one second per product category. In about the length of this session, you would have counted all 5,000. Now imagine doing the same thing for every different type of product on sale in Walmart. There are 100,000 there. It would take you all day. Now imagine trying to count every different specific product and service on sale in a major economy such as Tokyo, London or New York. It's even more difficult in Edinburgh because you have to count all the whisky and the tartan. If you wanted to count every product and service on offer in New York -- there are 10 billion of them -- it would take you 317 years. This is how complex the economy we've created is. And I'm just counting toasters here. I'm not trying to solve the Middle East problem. The complexity here is unbelievable. And just a piece of context -- the societies in which our brains evolved had about 300 products and services. You could count them in five minutes.

So this is the complexity of the world that surrounds us. This perhaps is why we find the God complex so tempting. We tend to retreat and say, "We can draw a picture, we can post some graphs, we get it, we understand how this works." And we don't. We never do. Now I'm not trying to deliver a nihilistic message here. I'm not trying to say we can't solve complicated problems in a complicated world. We clearly can. But the way we solve them is with humility -- to abandon the God complex and to actually use a problem-solving technique that works. And we have a problem-solving technique that works. Now you show me a successful complex system, and I will show you a system that has evolved through trial and error.

Here's an example. This baby was produced through trial and error. I realize that's an ambiguous statement. Maybe I should clarify it. This baby is a human body: it evolved. What is evolution? Over millions of years, variation and selection, variation and selection -- trial and error, trial and error. And it's not just biological systems that produce miracles through trial and error. You could use it in an industrial context.

So let's say you wanted to make detergent. Let's say you're Unilever and you want to make detergent in a factory near Liverpool. How do you do it? Well you have this great big tank full of liquid detergent. You pump it at a high pressure through a nozzle. You create a spray of detergent. Then the spray dries. It turns into powder. It falls to the floor. You scoop it up. You put it in cardboard boxes. You sell it at a supermarket. You make lots of money. How do you design that nozzle? It turns out to be very important. Now if you ascribe to the God complex, what you do is you find yourself a little God. You find yourself a mathematician; you find yourself a physicist -- somebody who understands the dynamics of this fluid. And he will, or she will, calculate the optimal design of the nozzle. Now Unilever did this and it didn't work -- too complicated. Even this problem, too complicated.

But the geneticist Professor Steve Jones describes how Unilever actually did solve this problem -- trial and error, variation and selection. You take a nozzle and you create 10 random variations on the nozzle. You try out all 10; you keep the one that works best. You create 10 variations on that one. You try out all 10. You keep the one that works best. You try out 10 variations on that one. You see how this works, right? And after 45 generations, you have this incredible nozzle. It looks a bit like a chess piece -- functions absolutely brilliantly. We have no idea why it works, no idea at all. And the moment you step back from the God complex -- let's just try to have a bunch of stuff; let's have a systematic way of determining what's working and what's not -- you can solve your problem.

Now this process of trial and error is actually far more common in successful institutions than we care to recognize. And we've heard a lot about how economies function. The U.S. economy is still the world's greatest economy. How did it become the world's greatest economy? I could give you all kinds of facts and figures about the U.S. economy, but I think the most salient one is this: ten percent of American businesses disappear every year. That is a huge failure rate. It's far higher than the failure rate of, say, Americans. Ten percent of Americans don't disappear every year. Which leads us to conclude American businesses fail faster than Americans, and therefore American businesses are evolving faster than Americans. And eventually, they'll have evolved to such a high peak of perfection that they will make us all their pets -- (Laughter) if, of course, they haven't already done so. I sometimes wonder. But it's this process of trial and error that explains this great divergence, this incredible performance of Western economies. It didn't come because you put some incredibly smart person in charge. It's come through trial and error.

Now I've been sort of banging on about this for the last couple of months, and people sometimes say to me, "Well Tim, it's kind of obvious. Obviously trial and error is very important. Obviously experimentation is very important. Now why are you just wandering around saying this obvious thing?"

So I say, okay, fine. You think it's obvious? I will admit it's obvious when schools start teaching children that there are some problems that don't have a correct answer. Stop giving them lists of questions every single one of which has an answer. And there's an authority figure in the corner behind the teacher's desk who knows all the answers. And if you can't find the answers, you must be lazy or stupid. When schools stop doing that all the time, I will admit that, yes, it's obvious that trial and error is a good thing. When a politician stands up campaigning for elected office and says, "I want to fix our health system. I want to fix our education system. I have no idea how to do it. I have half a dozen ideas. We're going to test them out. They'll probably all fail. Then we'll test some other ideas out. We'll find some that work. We'll build on those. We'll get rid of the ones that don't." -- when a politician campaigns on that platform, and more importantly, when voters like you and me are willing to vote for that kind of politician, then I will admit that it is obvious that trial and error works, and that -- thank you.

(Applause)

Until then, until then I'm going to keep banging on about trial and error and why we should abandon the God complex. Because it's so hard to admit our own fallibility. It's so uncomfortable. And Archie Cochrane understood this as well as anybody. There's this one trial he ran many years after World War II. He wanted to test out the question of, where is it that patients should recover from heart attacks? Should they recover in a specialized cardiac unit in hospital, or should they recover at home? All the cardiac doctors tried to shut him down. They had the God complex in spades. They knew that their hospitals were the right place for patients, and they knew it was very unethical to run any kind of trial or experiment.

Nevertheless, Archie managed to get permission to do this. He ran his trial. And after the trial had been running for a little while, he gathered together all his colleagues around his table, and he said, "Well, gentlemen, we have some preliminary results. They're not statistically significant. But we have something. And it turns out that you're right and I'm wrong. It is dangerous for patients to recover from heart attacks at home. They should be in hospital." And there's this uproar, and all the doctors start pounding the table and saying, "We always said you were unethical, Archie. You're killing people with your clinical trials. You need to shut it down now. Shut it down at once." And there's this huge hubbub. Archie lets it die down. And then he says, "Well that's very interesting, gentlemen, because when I gave you the table of results, I swapped the two columns around. It turns out your hospitals are killing people, and they should be at home. Would you like to close down the trial now, or should we wait until we have robust results?" Tumbleweed rolls through the meeting room.

But Cochrane would do that kind of thing. And the reason he would do that kind of thing is because he understood it feels so much better to stand there and say, "Here in my own little world, I am a god, I understand everything. I do not want to have my opinions challenged. I do not want to have my conclusions tested." It feels so much more comfortable simply to lay down the law. Cochrane understood that uncertainty, that fallibility, that being challenged, they hurt. And you sometimes need to be shocked out of that. Now I'm not going to pretend that this is easy. It isn't easy. It's incredibly painful.

And since I started talking about this subject and researching this subject, I've been really haunted by something a Japanese mathematician said on the subject. So shortly after the war, this young man, Yutaka Taniyama, developed this amazing conjecture called the Taniyama-Shimura Conjecture. It turned out to be absolutely instrumental many decades later in proving Fermat's Last Theorem. In fact, it turns out it's equivalent to proving Fermat's Last Theorem. You prove one, you prove the other. But it was always a conjecture. Taniyama tried and tried and tried and he could never prove that it was true. And shortly before his 30th birthday in 1958, Yutaka Taniyama killed himself. His friend, Goro Shimura -- who worked on the mathematics with him -- many decades later, reflected on Taniyama's life. He said, "He was not a very careful person as a mathematician. He made a lot of mistakes. But he made mistakes in a good direction. I tried to emulate him, but I realized it is very difficult to make good mistakes."

Thank you.

(Applause)