‘Best practices’ and the survivorship bias

by Prof Ares Kalandides

Collecting, analysing and sharing Best Practices, i.e. examples of projects, policies, cases, etc. that have worked out in one place and could be applied to others, is a very common practice in Place Management. It is argued that people and organizations in one place can learn from the experiences of their counterparts in another and that, after considering their adaptability, can apply similar techniques in their own context. This seems like a reasonable assumption: while we mostly learn from our own experiences, and psychologists have demonstrated the validity of this argument, we do take into consideration what other people have experienced elsewhere, albeit marginally.

Best Practice in Business Improvement Districts. Research by the BID Foundtion.

There is no doubt the Best Practices technique can be very useful. Among other things, I would argue, Best Practices can serve as an inspiration, they can motivate people to do things, show them that the implementation of new ideas is possible. Research conducted by the BID Foundation, Best Practice in Business Improvement Districts, is a good example of how Best Practices can be shared among places, be a source of stimulus and knowledge exchange. This report is also an example of a ‘place-sensitive’ approach. The cases researched are all in the UK, and despite local differences (some of them profound), all share both similar institutional and cultural backgrounds. This means, that a lot of the ideas and knowledge may well be transferable from one place to the other. Indeed, one of the main criticisms of Best Practices is precisely this: that usually the technique lacks the ‘place sensitivity’ needed to demonstrate the sometimes fine but decisive differences of the societal contexts of each case. This, it is argued, creates false expectations about their transferability and encourages a form of ‘imitation culture’.

However, I see one more caveat and I will need to digress a little (or a lot) to illustrate this:

By McGeddon – Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=53081927

During World War II, US Air Force researchers conducted studies of the damage done to aircrafts that had returned from missions. By examining that damage (s. image on the left) they recommended that the parts of the aircrafts that showed the most of it be reinforced to minimize bomber losses to enemy attacks. The argument was straightforward: if damage by enemy weapons is systematically detected in particular areas of returning aircrafts, then it is statically those areas that are more likely to get hit and thus special attention should be paid to them (s. image above).  “Not quite” retorted Abraham Ward, a Hungarian mathematician, who contributed immensely to decision theory: “What we consider here” he apparently said, “are aircrafts that have survived their missions, those that got back safely, whilst we know nothing about the ones that got lost”. This means that the holes in the returning aircraft body show areas where the aircraft can take damage and survive. On the contrary, areas where the returning aircrafts showed no damage, were those that, if hit, would cause the plane to be lost. It is those unscathed spots, argued Ward, that needed to be reinforced. The term ‘survivorship bias’ was coined and operational research, i.e. the discipline that deals with the application of advanced analytical methods to help make better decisions, was born.

A Review of Business Improvement Districts in the UK. Research by the BID Foundation.

In the Best Practice technique we also examine the ‘aircraft that return’. We focus on those ones who survived, the cases that worked well, the policies that bore results. In short, Best Practices is generally about success. But, what do we know about what is lost to the eye when we do not think about the projects that failed, the policies that did not deliver, the projects that were discontinued? Surely all projects are evaluated, but the assessment results get archived in some obscure drawer somewhere. Why don’t we get to hear about them more often? I have occasionally, but very rarely, heard people present their failed projects in conferences and in meetings and I find such accounts extremely useful. The hesitation is of course understandable, as, after all, who wants to present themselves as a failure? Again, the BID Foundation has conducted another excellent research project, looking at BIDs in the UK, A STATE-OF-THE-ART REVIEW OF BUSINESS IMPROVEMENT DISTRICTS IN THE UK: Setting the agenda for policy, practice and research. A whole chapter (Chapter 6) is dedicated to ‘Unsuccessful ballots and BID terminations’, paying due attention to the reasons for their discontinuation.

Although, as I have summarized above, I do see its virtues, I take issue with the Best Practice habit – which I must admit I often use myself – on several grounds:

Firstly, only enumerating instances of success creates the illusion that most examples have been successful, while very often quite the opposite is the case. Whilst this can indeed be a source of motivation, it makes one’s own failure (which will inevitably take place occasionally) almost unbearable: ‘I am a failure while everybody else is a success’.

Secondly, concentrating on the factors of success, easily conceals all the things that can go wrong (including the things that did go wrong). We keep saying that we all ‘learn from our mistakes’ – yes, but we learn from other people’s mistakes, too, if we can see and understand them.

Third, if we want to try out new things (and I mean new things, not imitate things that were done successfully in other places), then we need to embrace the possibility of failure. All experiments, all basic research, all innovative endeavours carry with them a level of risk, which may be more or less foreseeable and manageable. We are used to performing risk evaluations of our work; why do we then avoid talking about our failures? A culture that nourishes risk is a culture that nourishes innovation. It is also a culture that accepts that failure itself is part of the whole process of moving forward.

All of the above does not mean that we should abandon the Best Practices technique altogether. Quite the opposite. It means that we need to be conscious of what we’re doing, that we know its limitations and that we find ways to mitigate them. This could include, for example, making clear statements about the differences between places (not generic ones, but analysed and documented), it should include consciously using place-sensitive language and be very cautious about generalisations and transferability and finally, it could include mishaps, deviations, wrong decisions, failure and – why not? – Worst Practices. The latter has good chances of becoming my personal favourite, as I have enough failures to add to it myself.