Taking evidence-based policy seriously

Evidence-based policy

by Prof Ares Kalandides

Evidence-based policy‘ has been a catchword in politics for some time now. It was allegedly coined by the Blair government, which aimed to design policy driven less by ideology and more by scientific evidence. Two decades later the term is still going strong, with calls for ‘evidence-based’ policy being the norm rather than the exception. However, both the terms ‘evidence’ and ‘scientific’ need some clarification when we’re talking about the social sciences, if we want to take evidence-based urban policy seriously.

Social ‘sciences’ (with or without quotation marks) follow a rather different logic than exact sciences. As this is not a new debate and not one I want to open again now, suffice to remember some basic differences:

It is almost impossible to break down the complexity of social life into objective individual factors. Any social analysis is value-laden and as such ideological.

Even if you manage to identify (your) factors behind (your) social phenomena, it will be hard to link one phenomenon to one single factor. What you then need to do, is show a system of factors that interact to produce a phenomenon.

You may be able to show correlation, but you can hardly ever prove causation. More generally, you can almost never ‘prove’. You can formulate hypotheses and if you’re lucky enough, you may even be able to ‘demonstrate’ how something works.

As we do not have a ‘society lab’, you cannot replicate – which means that you can also not falsify. Non-replicable, non-falsifiable findings are not scientific findings stricto sensu.

And finally, social phenomena do not happen ‘on the head of a pin’. They have their own spatiality and cannot be transferred easily from one place to another. What works here may not work elsewhere, even if we’ve done our best to identify why it worked here.

Let me give two examples that may illustrate this better:

In 2016 I worked on a research project in the German State of Nordrhein-Westphalia, in which we were asked to find good examples of reuse of vacant town-centre retail spaces and identify the factors behind their success. Our team worked over a period of several months. Research was initially conducted on the internet to come up with a long list of projects, which was then refined down to a detailed analysis of 25 cases, based on site visits and interviews. By analysing our data, we came up with what we then called ‘parameters’ for the successful reuse of empty retail spaces. The choice of word was not accidental: we believe that the word ‘parameter’ is better at describing a broader environment of factors that may have influenced the project’s success. Such parameters, for example, include strong organisations of the civil society, local pride, leadership, political engagement, global connections etc. We found it impossible to reduce one project to one factor. And as our results were meant to be used as a policy tool for local councils, our recommendation was that people should look for any of our parameters in their own localities, examine how they interact locally and find ways to foster one or the other. You can download the full report for free here (in German).

Evidence-based policyThe second example is a piece of extensive research conducted by the Institute of Place Management that identified the role of markets in the economic, social and political health of towns and cities. Through a comprehensive review of the published evidence the research team managed to demonstrate how markets add to the vitality of specific centres. The report identified the 35 most important reasons why markets matter, such as “with low barriers to entry, markets are excellent business incubators and support business formation”, “markets are places of social interaction”, “markets are places of innovation, experiment and education”, etc. You can read more about the report here, on this blog.

In both cases researchers have not produced a straight line between A and B, but have demonstrated how a local mix of different parameters can produce one effect (successful reuse of vacant town-centre retail spaces) or how one economic activity, the market, can have different effects in different places. In both cases the correlations are not isolated or straightforward, but are embedded into very place-specific systems. What will really happen in one place may be very different to what will happen in another, even if they seem very similar.

Of course, research is never value free. How we framed our questions, the methods we used, the analysis we performed, what we looked for, what we decided to include and what not, are directly linked to our own belief systems. Also, social research will always be imperfect, incomplete and contradictory. It is better to address such biases right away than pretend to be delivering an exact representation of the real world.

Whereas evidence-based policy is indeed what we should be aiming for, overreliance on ‘objective’, ‘rational’, ‘generalisable’ and ‘quantifiable’ data is not necessarily the best way about it, as the latter does not always constitute good evidence. Rather, I think that what we need is a place-and-evidence-based approach in which our own ideological biases, the incompleteness of research and the specificity of place are explicitly addressed and rendered visible.