Evidence based?

A couple of weeks ago the apparent banning of certain words in the US caused considerable concern. The context of this news report is important, as this article makes clear. So it’s probably a lot of fuss about nothing, although as a signal of what is going on in within the US government it may point to a worrying trend.

I don’t agree with banning language in general but conversations about this report, with particular reference to the term “evidence based”, have prompted some reflection. I think it would be no great loss if “evidence based” fell out of use since its meaning is unhelpful. (I’m not planning to delve into philosophy, law or any relevant literature: these are just some fairly random reflections, based on my own experience, and occasionally informed by Google. Does this count as “evidence based”? )

“Evidence based” usually precedes the word “policy” or “practice”.

Evidence based practice seems intuitively like a Good Thing, especially in the field of medicine. You’d probably want your medical practitioner to be using treatments that are tried and tested, which is what the description implies.

Google Books Ngram Viewer shows that there was a very steep rise in the use of the term through the 1990s, levelling out since 2004. Medical practice is where the term “evidence based” first arose, which is slightly worrying, given that it is so recent.  What was medical practice based on before? According to Wikipedia evidence based practice is contrasted to “rules of thumb, tradition and folklore”. (Folklore must surely have its place: aspirin is derived from plants in the willow family which were used in folk remedies long before the 19^th century discovery of salicylic acid. )

I don’t know anything about the politics of medical research but some superficial investigation online suggests that the term was introduced by Cochrane who advocated randomised controlled trials to support medical practice. You can read about him here  and here.

You can also search the Cochrane database.

 This provides a clear summary of systematic reviews of all the available evidence collected from reports of clinical trials etc. Importantly, it notes the extent of the evidence summarised and draws conclusions on the strength of the evidence.  But a word of warning: if you spend much time in the Cochrane database you may start to wonder what we do actually know about medicine with any certainty… For example, if you have been encouraged to have a flu jab, you might want to read this and consider whether it was worth it.

But I’m more interested in the application of the term to policy because I think this is more problematic. Googling “evidence based policy” throws up a huge  number of links (and alerted me to this book which looks very interesting and apparently critiques the Cochrane approach.)

There is an extensive literature on “evidence based policy”. Ngram shows that the term developed at around the same time as “evidence based practice”. Possibly the widespread adoption of “evidence based practice” led to some questioning of the evidence base of health service policy.

As individuals, we have our own criteria for judging the quality of arguments put to us. These may vary widely because such criteria will not be based solely on scientific evidence that we know about but will also be moderated by our own beliefs.  (The balance may not always be in favour of scientific evidence: see, for example, climate change deniers. And there is still a Flat Earth Society.)

So the value of evidence in supporting arguments is likely to vary at an individual level depending on all sorts of contextual factors. What about at the broader level of society?  If evidence based policy is a Good Thing, what evidence will be used and how will it be used in developing policy? How will ideology and evidence be balanced? Does “evidence based” mean that evidence outweighs ideology?

Here are two examples of policy making relating to corporate governance.  As far as I’m aware, neither of these have been claimed specifically to be evidence based, although they clearly are - to a varying extent.

1. The establishment of the Cadbury Committee was driven by the agenda of professional accountancy bodies, seeking to deflect attention from criticism of auditing , following several financial scandals.  The process of preparing the draft report involved the chair talking to interested parties - the intention was to produce a code based on best practice so these were generally practitioners of corporate governance. He also read relevant material and with the help of the secretary produced papers for discussion by the committee. Comments were invited on the draft report and the Code and recommendations were prepared thereafter. The evidence used did not include academic research: notably, academic studies had already questioned the monitoring efficacy of NEDs, which was a key assumption of the Cadbury Code.

2. The original Davies report on board gender diversity cited supporting academic research but ignored empirical studies which raised questions about the possibly negative consequences of implementing requirements about board composition. 

Each of these policy developments involved a process. Consultations took place. How reliable is the consultation process in collecting evidence? That’s difficult to judge because the process is often opaque. What are the evidence boundaries? Will gaps be identified?  A year ago I was involved in analysing responses to a UK government committee consultation. At the behest of the committee secretary, these were summarised according to respondent categories. One obvious category of interested parties had not provided any responses: although I drew this to the attention of the secretary, I don’t know if it was noted in the papers provided for committee members.

And who decides which interested parties will be consulted? And how will they be consulted? And what weight will be given to their views? How will the information they provide be presented to the ultimate decision makers? 

All policy is evidence based. Did the policy arise out of the evidence? Or was the policy  developed and evidence collected to support its advancement? Who decides what the evidence shall be, how it shall be collected and how it shall be interpreted?  I think the term “evidence based” implies that this process is neutral and objective. I don’t think it is.  You can’t take the politics out of policy.