Evidence, Policy and Regulations series - Evidence Synthesis by Building a Case (Workshop 1: Evidence amalgamation: weighing the evidence v building a case)
It is surely better to use evidence to inform predictions about which policies will work if we implement them and which not. Most policies involve complex interventions and almost all are into complex settings. In these cases a vast array of evidence of different kinds from different sources will be relevant. But how does this variety of evidence add up? How are policy makers to figure out what, all told, the evidence points to?
There are a variety of evidence synthesis schemes available that are reasonably good but of restricted applicability. Many can treat evidence even from different disciplinary approaches – so long as the evidence is homogeneous, that is: the separate pieces of evidence speak directly to the same immediate hypothesis. In general the pieces of evidence needed for policy evaluation are not homogeneous. They are not discrete masses all speaking immediately to one overall conclusion about how effective a policy is, but rather multiple parts of a structure of different claims that together may argue for this overall conclusion.
In this case synthesis is more complicated. The project will build on the idea that rather than thinking in terms of ‘weighing up’ evidence, the metaphor of ‘building a case’ is more useful. The argument for the overall conclusion involves a network of relationships among claims about different facets of the process by which the policy achieves (or not) its effects. Fans of courtroom drama are familiar with instances of this metaphor in which a case appears weak until evidence about a few critical facts is provided that make the case fit together and explain how the crime occurred. The aim is to lay the groundwork for a protocol for the synthesis of non-homogeneous evidence in ‘building a case’ to arrive at an overall evaluation, a protocol that would provide a basis for causal inference at different levels and for different parts of the intervention and that can aggregate different kinds of evidence from different kinds of sources and different perspectives (e.g. statistical, ethnographic).
Organised by Professors Nancy Cartwright and Julian Reiss, Philosophy Department and CHESS, there will be a Thursday morning reading group to survey and discuss the available literature in Michaelmas term at Durham’s Centre for Humanities Engaging Science and Society, and three workshops for policy makers, methodologists and anyone interested in working out how to answer these questions. Participation is open to anyone with an interest in the issues.
Workshop 1: 10 November 2015, Senate Suite, University College (Durham Castle) Evidence amalgamation: weighing the evidence v building a case Organised by Dr Jacob Stegenga (Victoria University and IAS Fellow) and Professor Nancy Cartwright (Durham University)
This workshop will review conventional methods for amalgamating evidence and explore techniques for synthesis by building a case, investigating the relative advantages and disadvantages of the two kinds of methods. Please note that this workshop is no longer an all-day event.
Agenda: 10th November 2015, 2pm – 6pm, Senate Suite, University College
- 2.00pm - Short Introduction - Dr Jacob Stegenga (University of Victoria, Canada)
- 2.10pm - 'Probabilistic causal assessment and evidence amalgamation in pharmacology' - Barbara Osimani (Munich)
- 2.50pm - 'Weighing evidence: a mechanism-based approach’ - Jaakko Kuorikoski (Helsinki)
- 3.30pm - 'How the Intergovernmental Panel on Climate Change Aggregates Evidence’ - Casey Helgeson (LSE)
- 4.10pm - Tea/coffee break
- 4.20pm - ‘Who’s afraid of mechanisms?’ - Phyllis Illari (UCL)
- 5.00pm - General thinking about how to build a case - Michael Woolcock (World Bank), Nancy Cartwright (Durham)
- 6.00pm - Finish
Workshop 2: 11 February 2016, Senate Suite, University College (Durham Castle) Amalgamation and the Principle of Total Evidence
Organised by Professor Julian Reiss (Durham University)
The Principle of Total Evidence is the recommendation to use all the available evidence when estimating the probability of a hypothesis (Carnap 1947). While seemingly uncontroversial, upon reflection the principle is more problematic than at first sight. On the one hand, it is, at least implicitly, denied by proponents of evidence-based approaches in the sciences: they recommend instead only to use evidence that has passed a certain quality threshold. On the other, what the principle really entails depends greatly on how to interpret ‘available’. Does it mean ‘known by the agent who makes the probability estimate’ or ‘known by the community of researchers’ or ‘knowable after a suitable search period’? Might it entail a demand to make new observations and experiments? Please note that this workshop is no longer an all-day event.
Workshop 3: 10 May 2016. Senate Suite, University College (Durham Castle) ‘What will work in my school?’ What does the evidence say?
Organised by Professor Nancy Cartwright (Durham University)
This workshop will focus on education as one of the many concrete policy areas where understanding how to amalgamate evidence matters to practice. Participants will propose and probe practical advice for educators who want to use evidence but have to make real decisions in real time with limited resources. Please note that this workshop is no longer an all-day event.
Please register interest by contacting the Durham’s Centre for Humanities Engaging Science & Society (CHESS) administrator at firstname.lastname@example.org
Contact: Professor Nancy Cartwright (email@example.com)
Contact firstname.lastname@example.org for more information about this event.