This conversation is closed.

Trial and error is inefficient

Tim Harford argue that trial and error is a good way to find solutions. I totally disagree with this statement. If trial and error allows to find a good solution (and in fact the best possible solution) it is usualy the worst way to find a solution. The problem with trial and errors is not the obtained solution, but the time (multiple trials) and the errors. In nearly every domaines, we cannot afford the time needed to make a sufficients amount of trials and we cannot afford the errors in some cases (errors that conducts to death for exemple). The cost of trials and errors are just too high. That's one of the purpose of intelligence : avoid errors and find solutions with the less amount of trials.
Of course, computer simulations is a good way to apply trial and error, but it implies that the simulator is perfect (or at least reallistic enough) which is rarely the case, even in physics.

Random trials lead to a majority of failures for a few solutions.
Intelligence and thinking lead to a majority of solutions and a few errors.

  • Jul 19 2011: Yes, trial and error is inefficient - but what is the alternative? Trial and success? No, the alternative to trial and error is the fear of error, and policies which place, as their highest criterion, the avoidance of all error. Such thinking is a disease in our education and our culture. It leads to excessive leaning on old, safe techniques, and is an effective way of cutting off precisely those creative directions that may lead to breakthroughs. Of course, it is always possible to look back on any new discovery and find reasons why the route chosen was somehow more correct than others, but in the moment, there's no label distinguishing error from truth, and both may appear wild and unpredictable. In fact, an approach where investigators deliberately pursue error is likely to be more productive than one where they try to play it safe. I see this daily in the world of creative writing (giving students the assignment to write something bad invariably produces far better results than telling them to write something good), and there are many examples from science - for example, Feynman's playful (and, at the time, pointless) calculations of the movements of a dinner plate, which gave him insights into electron spin, and led to a Nobel Prize. This sequence from misdirected effort to success is not some fluke - it's the way creativity naturally works. It's fixating on getting things "right" that is inefficient, because it hobbles the mind.
  • Jul 16 2011: I see your point. It certainly makes sense to me that we can avoid unnecessary experiments by using what we already know - "intelligence" as you called it. However, as far as I can tell, that intelligence was developed through trial and error. Ideas have to be tested to verify that they work because as we know, many ideas do not work. We have to try things out to see which one work and which do not.

    After all, people make errors in judgement; we make mistakes. So what was once thought to be intelligent may no longer be. For example, at a point in time, people presumably thought that the world was flat. This intelligence would have led to other false conclusions. It is through trial and error (sailing around the world at the risk of falling off the edge) that we can discover reality.
  • thumb
    Jul 19 2011: Yes, I understand your argument, Ajidicia, and, in a certain way, agree with it. However, I think that the mainly objetive of Tim Harford is not to make us suddenly drop all the tecniches we've developed these years to solve problems reducing errors, but to make us realize that it is not expected from us that we have all the answers. And even more, that it is important to make mistakes, because sometimes it is the only way we can find out something completely unexpected.

    To realize that we are free to make trials open our minds and really unleashes the creative process, which is the base not only to solve problems but to discover mysteries. The ability to admit that there are empty spaces in our knowledge - and that they will always exist - is not only a powerful weapon against proud, vanity and non-cooperative feelings - the caracteristics of the God complex - but it is also the first step to a better way of life, free from stress, phsycological pressure and the responsability of being perfect.
  • thumb
    Jul 18 2011: .
    Ajidicia, I think Tim Harford also meant we need to invest more in "experimentation".

    Take the example of development aid, international aid to poor countries. For decades we've been guessing, and switching approaches without measuring our impacts.

    Esther Duflo, from MIT, now came up with a brilliant idea: just go out and *test* different potential solutions, in social experiments. (Check out her TED presentation: http://www.ted.com/talks/esther_duflo_social_experiments_to_fight_poverty.html ) Nobody has done this before. But it yields the best answers.

    True, it takes a bit more time, but the solutions are empirically tested. Not just guesswork.

    And the result: no more billions spent on useless interventions. In the long-term this pays off.

    I'm sure that in other sectors (such as technology design), real a "trial and error" procedure is simply too costly and time-consuming. But in others it may be worth gold.
    • thumb
      Jul 18 2011: thumbs up for the link
    • Jul 19 2011: While I enjoyed this talk the problem I have with it is that I'm not sure who it is aimed at. Anybody involved with R&D, such as the Unilever engineers will be well aware of Design of Experiments and Optimization. Indeed, the example given with the spray nozzle was a rudimentary Genetic Algorithm.

      Harford talks about these well used and very popular development techniques but refers to them as trial and error. As an R&D engineer I find this almost insulting; the years of research and programming I have spent in order to learn and correctly utilise these techniques, let alone the hard work applied by the people who developed them is undermined by labelling it as something as trivial as "trial and error".

      Applying this to other real-world situations throws up a whole load of ethical errors. Harford discussed this with the example of cancer sufferers recovering in hospital or at home. In this case the trial turned out to be correct and the cancer patients were better off at home, but what if this hadn't worked? The doctor in question would be struck off these days for endangering patients. Just because this example turned out to be correct doesn't mean the doctor who conducted the experiment was right. As a result I feel that even using an example such as this is morally and ethically wrong.
      • thumb
        Jul 19 2011: i don't think it was a rudimentary genetic algorithm. i think it was an actual genetic algorithm, and was designed as such.
        • Jul 19 2011: I meant rudimentary as in basic or elementary. Taking the best solution from a random set and then randomly modifying it to generate a new set each iteration seems, to me, to be fairly simple.

          The GAs I've written use several parent variants (usually selected from a fractional factorial design space) which breed to create a new set of child variants (along with a random mutation so that the gene pool isn't exhausted too quickly). From these the best ones are selected to reproduce. I usually use three parents to generate six children and then select the best three of the nine for the next generation. Couple this with a few random variants thrown in to simulate gene pool mutations.

          Once this has converged to a solution I apply a Hooke-Jeeves algorithm to search for local minima at a higher resolution than the GA can handle efficiently. I find this combination converges to a good solution far quicker and more robustly than a trial and error technique such as Simulated Annealing.
  • thumb
    Jul 23 2011: Let me put it this way, some problem can only be solve with trial and error.
    It's the only reason why human never "hang".
    If we could create machine that "trial and error" the machine no longer needs human.

    It might not always be efficient but it is ALWAYS effective.
  • Jul 21 2011: i think the problem here is that you assume thinking about something takes little time whereas random trials take a lot of time. this is not necessarily the case: say you were wondering which of 2 programs would run the fastest. you could spend hours reading through the code of each and get a feeling as to which would take longest, or you could just run them both elapsing only a few seconds.

    in some cases of course you'd be right, such as when there are a huge number of variables, but this is hardly cause to conclude that trial and error is not an efficient method.
    • Jul 21 2011: I think you said it well, Ben. How do we know whether an analytical approach would be more effective? The only way to know is to try the analytical approach side by side with an experimental approach. Sometimes, the analytical approach takes far longer and may not even result in a solution due to complexity.
      • Jul 22 2011: yeah things are often more complex than we realise. it's quite easy to come up with a perfect solution, yet when we try it we discover another part of the problem that we weren't aware of - the solution was indeed perfect, but perfect for an incomplete set of data.

        actually i think this is behind all these self-professed geniuses who think they know how to fix the american education system, none of them are even teachers which is why it keeps getting worse. education was fine back when teachers had control and non-teachers (school boards, psychologists, government bureaucrats) weren't telling them how to teach.
  • thumb
    Jul 17 2011: The reality is that we are all already involved in the biggest trial and error experiment of them all: EVOLUTION. Problems begin when we try to resist this reality instead of learning to work with it in a more umh... intelligent way. Our best results come from the complexity, layering and understanding that is built over countless attempts to find answers. And our worst? When we try to play God of course.
  • thumb
    Jul 17 2011: seems that you disagree with the ways of nature. evolution is a great showcase of trial and error, and it produced species of marvelous sophistication.

    also you seem to refuse capitalism, which brought about the largest economic growth in human history.
    • Jul 18 2011: I do not desagree with nature or refuse anything : but trial and error (as in evolution), means RANDOM trials. And with our intelligence and memory, we can avoid some trials for which we can predict failur (based on past experiments or logical consequences). If we do not avoid those trials, it would only be a waste of energy.
      • thumb
        Jul 18 2011: not it does not mean random actually. it means many. evolution works with random trials, and it works good. the capitalist economy works with non-random but highly intuitive trials. the rule is simple: we put in the knowledge we have, and let trial and error deal with the rest. evolution has no knowledge, so it relies purely on randomness in its trials. we humans know more, and thus we incorporate a great deal of knowledge. but what we know is dwarfed by what we don't know, so the bigger part of the job still falls upon trial and error.

        what is very interesting is that this observation about our economy being unimaginably complex is not at all new. it was described by ludwig von mises in 1920, but he added upon previous works. hayek got the nobel prize for re-discovering it. but today, almost a hundred years later, it is still unknown for most.
      • Jul 19 2011: Ah, I think we are talking past one another. I do not believe (and I do not believe that Harford believes) that "trials" are completely random. Take the Unilever example he used again. If one were to *randomly* design a pipe attachment and then test it to see if it creates a suitable spray...well most random designs wouldn't create a spray at all. If they were truly randomly designed, many would be solid blocks that simply prevent the passage of fluid.

        Clearly the starting point was not a truly random design at all, but multiple devices that each conformed to the basic understanding of a "spray nozzle". Most likely the engineer(s) involved had some expectations of what would create a better spray (some right and some wrong, no doubt) and incorporated those features into the initial batch.

        There was clearly a role for some expert input at that top level, with the primary point being that the experts were not of much use in designing an optimal solution, other than in that selection of sensible starting points and sensible variations to test once a winner was selected in each round.
        • Jul 19 2011: Kurt -> Thank you for writing this response, exactly my thoughts while reading.

          Furthermore, evolution is not "purely random" any more than 2Hsub2 + Osub2 becoming 2Hsub2O is purely random when in proximity of each other. The randomness of Brownian motion is not equivalent to the statistical models of complex systems. These open systems are largely indeterminant, due to unobservable causation (e.g. Maxwell's Demon). The independent nature of random variables in statistical models is an admission of not knowing all these inter-relations. The model is not the physical system itself, which is quite causal.

          And this is exactly the point of the talk. The God Complex compels many well educated people to equate human tools with actual physical reality, more than is efficient. History tends to sweep under the rug all the approaches culture was sure was right for hundreds of years, when in fact it was quite wrong.

          Trial and error, as described in the talk as making "good mistakes", is one tool among many that has untapped potential to dispel our experts' preconceptions.
      • thumb
        Jul 20 2011: Ajidicia, you have referred several time to a "waste" of time and/or energy as your reason for reducing/eliminating trial and error in most learning endeavors.
        That would imply that there is a finite or limited supply of time and/or energy, when, in fact, there is an unlimited, abundant supply of all, which we as humans on earth get to use in any way we fancy. Isn't it wonderful to be a creative, sentient human being with so much to learn and experience?
    • Jul 19 2011: Evolution is obviously not random. We do not fully understand nature's selection process.

      Why do we make simple things complicated? Our intelligence is limited so we have to use a some degree of trial and error, but no sane person would use trial and error blindly!
      • thumb
        Jul 19 2011: and you reply to what exactly?

        i was not talking about the selection process. the trial part of evolution is mutation and combination, both are completely blindly random. the selection is non-random, but it has nothing to do with the trial method, it is the mechanism that selects what is considered error and what is success. we kind of understand this process, so i have no idea what do you mean by "not fully" understanding it.

        i didn't say we need to try blindly. nobody said. it was said that we don't have complete or even close to complete knowledge, so we need to fall back on trials. not blind, but smart ones.
        • Jul 19 2011: What I mean to say is we all agree because this is all common sense but we're all getting mixed up in jargon.

          I'm confused though, you say mutations in evolution are blind but our scientific trials are 'smart'? I suggest that you have effectively concluded that mutations in evolution must be smart not blind.
      • thumb
        Jul 19 2011: surely evolution is "blind" in its trials, and sciences is mostly not (but sometimes is).

        but that in no way means that evolution would be smart. it means the exact opposite.
  • Jul 17 2011: All of these statements seem untested to me. The cost of trial and error is too high compared to what? Compared to methods that reach solutions faster? How do you know that applying raw intelligence will reach a solution faster and more efficiently without conducting trials/experimentation, especially in the contexts he is discussing, that of problem solving in complex systems?

    It's certainly true that in a non-complex system, trial and error can be inefficient. If you are trying to find the square root of 27, then solving solely by trial and error is a terribly slow way to do it, usually. Still, if you had no calculator or computer how would you approach that problem? And that isn't complex at all. In the Unilever problem Hartford raises, anyone who understands the problem will immediately see why trial and error beats using mathematicians and/or physicists to calculate a specific solution. Fluid dynamics are chaotic systems and the calculations so sensitive to initial conditions that designing a truly "optimal" nozzle is virtually impossible, even on paper. In situations as complex an economic systems, finding a single optimal solution could very easily be beyond the brain power of even geniuses...and our politicians are almost never geniuses.

    So the problems may not be soluble by geniuses, and yet even if our politicians hire geniuses, we'd not get genius results. Why? Because in government geniuses have to work in groups (if only the group of the individual genius and the elected leader, though usually larger groups), but studies are clear that "group intelligence" is *not* correlated with or related to either the average intelligence of the members of the group or with the intelligence of the group's smartest members. (See the research done by Anita Woolley from Carnegie Mellon University, much discussed late last year.) So, even if a genius could solve our problems (they can't) their intelligence is not a factor in most social situations.
    • Jul 17 2011: much agreed .......
      a "genius" could only come up with a starting point and how close that starting point is is what makes someone in this situation a genius.
  • Jul 22 2011: Most of us forget what we are dealing with in trying to find answers. Simple linear mechanical problems can be analyzed and accurately predicted. Complex problems with networked elements like weather patterns and turbulent flows are best answered with simulation or building models. Living agents are adaptable and completely unpredictable; that's where we should be making small changes and waiting to see how the system adapts.
  • Jul 21 2011: You are absolutely right that trial and error is a horrible way to find solutions. However, I don't think that Mr. Harford is arguing that trial and error is the best way to find solutions. What he's saying is that in certain cases, trial and error is the only possible way to find a solution. The hard sciences are great for problems that have clear black and white answers, but in other areas like economics, the problems are so unfathomably complex, that the only way to solve it is through trial and error. However... the first step comes from admitting that we really don't know. Otherwise, countless hours are wasted in emotional arguments over theories that have no actual data to back them up.
  • thumb
    Jul 19 2011: I thinkHarford's talk speaks to the importance of not standardizing "best practice"s prematurely. An organization that believes it has identified "best practices" sometimes stifles or chooses to punish any sort of experimental deviation from what the current cast of managers believes is best.

    Stifling innovation in this way removes a key opportunity for the organization to learn and improve its services..

    I am not arguing that the organization should endorse "random" variations but rather that the organization should tolerate or even promote experimentation based on solid professional judgment or experience.
  • Jul 19 2011: All this talk about which is better leads to nothing when the powers that be kill any new thing they will not be able to control. Yes, trial & error is inefficient. It is supposed to be. That is why not everyone is a millionaire or a pioneer or other. Power and control by the few requires the efficient and systematic elimination of new unknown or unpredictable variables and the cultural formation of drones, sheeple, robotic consumers of the system. This is why the education system is now controlled and fed by big business in order to manufacture consumers. This is also why very successful products are discontinued because they are found to be useful in more ways than the producer could control. This is why Bluetooth on the iPhone cannot be used to transfer data from one phone to the next like every other brand of phone and among many other deliberate limitations - because here comes a new device on the market with unknown potential in the hands of the masses who will try and err until they use it to do advanced things, potentially dangerous things that not even law enforcement could control. Trial and error is how Christopher Columbus found the Americas. Trial and error is how we have achieved our advances. And it is how anyone can break free from under this oppressive global economic system of power. And we repeatedly place ourselves into this system of greed: we use trial and error to come up with an amazing new business then use the said "god complex" to control and prevent everyone else from doing the same in fear of competition with patents, registrations and perpetual copyrights... INTELLECTUAL PROPERTY, for goodness sakes!!! So I get persecuted for adapting an idea to my small corner of the world in continued trial and error?
  • Jul 17 2011: What's the difference between intelligence and trial and error?

    Intelligence is internalised (to the imagination) trial and error based on remembered data/observations. Much like a computer simulation is a trial (prone to error), or like the genetic algorithm (an AI technique) which Tim gives as an example of trial and error (the nozzle design problem).

    For linear problems, where an A to B solution is computable, trail and error is quite inappropriate (such as guessing the answer to a large sum). But when it comes to non-linear problems (such as most design problems), then an element of trial and error is NECESSARY. Unless you already know something about the solution, some degree of guessing is required.

    You are correct in observing that to fully implement every guess is wasteful, that is, it is much more efficient to employ intelligence to imagine or simulate outcomes, but it's still the same kind of process. Furthermore when imagination fails (as it does) what remains is to get your hands dirty with real world data collection.
  • Jul 17 2011: Even the approach of a "genius" is nothing but a "try".

    Presuming that this were not so, then *is* the error *before* (andlamentably instead of) the trying (and its due evaluation).

    The choice is NOT between
    either "doing it right"
    or merely "trying to do it right"
    but RATHER between
    either "presuming to do it right (and not evaluating, hence not correcting)"
    or "presuming to merely try and carefully evaluating and if necessary correcting".
  • Jul 17 2011: Think about how products are developed. Someone has an idea and an engineer turns it into a prototype. Then the company begins to tweak it, modifying the products and changing the features and design until they think they've got it right. This is trial and error, and there is not really a better way to do it. Most solutions to problems can be improved by people who see things from a different perspective, and so collaboration and trial and error produces the best product. Imagine if we didn't field test products before their release because it takes extra time and money. For most products, such as cars and medicines, this would be dangerous and undesirable. Having the right people makes this process easier and shorter, but trial and error is still necessary and is much better than the alternative of assuming that the first solution is the right one.
  • thumb
    Jul 17 2011: Science is based on trial and error. Actually, trial and error is part of what defines science. Science is based on observations and experimentation.
    On the other hand, treating a disease we don't want to rely on trial and error, because that could easily lead to our demise.
    That doesn't mean there are no trials and errors involved, but that they were already done in advance (and preferable not on humans) to sort out what works and what not.
    So, in order for us to advance, trials (and errors are inevitable) cannot be avoided. We as humans are a result of trial error as is the whole universe.
    • Jul 17 2011: I would not say that science is trial and error driven. Science is based on hypothesis which are then tested. We expect a specific result, and we see if we were right (or wrong). So it's not a random trial, but a planned experimentation. (I would never be able to publish an article without argumented hypothesis)
      • thumb
        Jul 18 2011: True, I agree, it's not totally random because there usually is a reason why we run a particular experiment. Still, there are many experiments done that result in failure and we just keep going refining our theories and experiments until we finally succeed. So, although not completely random, we still can't avoid trial and error completely.
        • Jul 19 2011: Yes, this is the take home point: it is a false dichotomy to assert that things are either completely random or completely planned. Even science is set within an open system. Again, the point is to make "smart mistakes", and then learn from them to refine subsequent efforts. This can fairly, if informally, be called "trial and error". The talk never suggested that each iteration is not informed by former iterations.
  • Jul 18 2011: In genetics and biology GWAS is an example of a shot gun approach. Some like it, others not so much. However, try this, try that, and then something else. Why not? Cost, time, materials may or may not be an issue. As biological screening becomes cheaper and computers and AI increase in ability while dropping in cost Trial and Error will be productive.

    As for experiments in education, medicine, and welfare? Why not try this that and the other to see what works? And maybe a genetic upgrade would help in medicine, education, welfare, and work!

    Biology works this way! Why should we expect anything different?
  • Jul 17 2011: i dont know if somebody already said this i dont have time atm to read the coments but just replying to your topic...

    we use trial and error whether or not we mean to, if there is a HUGE problem and i have an idea that it seems right than i should stick to it right? acording to your statment yes... even if my idea ends up being wrong in the end and killing lots of people= "God Complex"?
    if i admit i may be wrong than i can abandon my idea or change it... and eventualy come to a decent conclusion (at least for the moment)= a few people dead and the HUGE problem solved
    naturaly you wouldnt just start at random; you would start at what we think is the best answer. Then be willing to give/change (admit your wrong)
    my take on the subject
    kevin
  • Jul 17 2011: Every replies are right, but my point is that trial and error should be used in last resort. Typicaly when the problem is too complex (the exemple of fluid dynamics is perfect, there is also the exemple of cars crash tests, clinical trials and many others).

    However, in some complex cases, the cost (in time and consequences) seems too high. An exemple would be in politics, or social issues because the results obtained on small groups cannot be extended to larger groups (the mechanism of large groups are differents). Then we would be forced to test on a whole state to have true results. But can we afford to test a politic of anarchy ? or self-education (and wait a full generation to see the results) ? We do not know if it is benefic or not. We can guess, but we don't know. I would say that the cost to obtain the answer is simply too high.

    Trials and errors are not the panacea to find solutions, but rather the most expensive way to find solutions. However it's sometimes the only way. (And of course, it's the only way to find the perfect solution on condition that every possible trials are tested)
  • Jul 17 2011: Trial and error maybe ineficient in our current strategies for solving problems exactly because our companies and workgroups are not designed for that. Our structures are designed for long planning periods before any execution. When the whole structure (people, team, industrial facilities) is designed for fast trial and error, it increases R&D processes speed and it allows unprecedent innovation capabilities. Many examples are available On current markets, from Toyota's industrial strategy to Google's "always beta" services.