What is Good Science: Evidence for Evidence-Based Policy
From BluWiki
"Evidence for Evidence-Based Policy" (2008) is part of a talk at a Home Office Seminar on Criminology and Evidence-Based Policy on 10 June 2008 by Nancy Cartwright.
Summary
- Author begins by stating that she is in favour of evidence-based policy - The disappointing results may exist due to not giving good advice how to apply it
Two lost questions
- Here: Focus on ‘evidence for effectiveness’: the claim, that an implemented policy will work as expected
- For the evaluation of the effectiveness of a policy, one needs credible evidence (=truism), generating three questions:
- Credibility: What is credible evidence? What ‘evidence’ can be considered as ‘true’?
- Relevance: What evidence claims are relevant –> how strong may a point pro/con be weighted?
- Integration: How should evidence be integrated?
- Credibility: check whether the claims that are made are true, even wrong assumptions can lead to a false outcome
- Relevance: Even if one could have the whole truth, one needs to focus on those issues that are relevant to a specific topic
- Example 1: “How to evaluate whether educational intervention is supported by rigorous evidence”:
- Strict rules are set up to define how evidence could look like
- Labels, as “well designed”, “well implemented” can support this trend
- Include warnings that all new rules may work out differently in different surroundings
- Basically good approach, however
- “Violation of principle of total evidence”: impossible to take into account all relevant information
- Example 2: The Maryland rules are another example of a ranking system: basically affecting the third question (integration)
- Categorisation acc. to the Maryland rules:
- Works: at least two studies found with methodological accuracy greater than 3 report positive effects
- Doesn’t work: At least two studies with methodological accuracy greather than 3 found that it has no significance
- Promising: At least one study (…) found significance for the project
- However: no integration at all!
- Usually, only question 1 is answered, however, no one really answers why method X, which is tested under circumstances Y may work equally under circumstances Z
- Question 1, however, is not a policy question, but only a scientific one
- Overall problem: Solutions are only scientific-based (answering question 1) and do usually not offer any real advice to decision-makers. Therefore: “need to develop far better guidelines for questions 2 and 3”
- Categorisation acc. to the Maryland rules:
How to use evidence: the need for a causal model
- If you could choose between two magnets, a small one and an industrial one, you would first need to know wheather the materials are magnetic and then you need to know if the one that is used fits your requirements (the small one will never be able to lift a car), and what are the physical conditions
- So: the effectiveness depends on the causal structure of the situation -> which magnets works effective and fits to our situation?
- Solution: “the most direct way of predicting its effects is to construct a causal model of the situation and estimate them” and advice on the two questions
Consequence:
- Reasonable causal models provide a good direction for answers to the three questions
- However, one then needs more information, contrasted to e.g. the ranking system (guides show that magnets can lift, because they lifted somewhat somewhere, but: do not tell us whether applicable in the concrete situation)
- Hence: “The probability ef the effectiveness is only as secure as the weakest link in our chain of reasonning”
Critique / Questions / Reflection / Comments
