ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd

1. Limitations and opportunities of the likelihood ratio approach for evidence evaluation ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd norman@agena.co.uk Fenton, N. E., D. Berger, D. Lagnado, M. Neil and A. Hsu, "When ‘neutral’ evidence still has probative value (with implications from the Barry George Case)", Science and Justice, Volume 54, Issue 4, Pages 274–287, July 2014

2. Overview • Revisiting the Likelihood Ratio – first principles, theoretical benefits and limitations • Revisiting the LR in two well-known cases • Why Bayesian networks are needed • Conclusions and way forward

3. 1. Revisiting the Likelihood Ratio – first principles

4. Was Mrs Peacock the murderer? H: “Mrs Peacock guilty” P(H) = 1/6 E: “The murderer was female P(E| H) = 1 P(E) = 1/2 By Bayes P(H | E) = 1/3

5. When does evidence Esupport a hypothesisH? When our belief in H increases as a result of observing E, i.e. when P(H | E) > P(H) So, suppose E supports H and that H’ is an alternative mutually exclusive hypothesis. Can we conclude that our belief in H’ must have decreased? NO!! (suppose H’ is “Miss Scarlett is murderer”) Except whenH’ = not H P(H’ | E) = 1 – P(H | E) < 1 – P(H) = P(H’)

6. A simple formal definition of probative value of evidence The ratio R: • R > 1 means E supports H • R < 1 means E supports notH • R = 1 means E is neutral for H Why do we never see this definition used?

7. Because of obsessive and irrational fear of the explicit PRIOR P(H)

8. Instead the Likelihood Ratio is used In addition to H we have to consider an alternative mutually exclusive hypothesis H’ L= By Bayes Theorem providing that H’=not H we can conclude • L>1 if and only if R>1 • L<1 if and only if R<1 • L=1 if and only if R=1 So L is a valid measure of the probative value of evidence E for H (when H’ = not H)

9. Benefits of the LR Simple formula for probative value of evidence No need to explicitly consider prior for H Forces forensic experts to consider the likelihood of both the prosecution hypothesis and the defence hypothesis

10. But note:It is meaningless to talk about the Likelihood Ratio being a measure of the probative value of evidence without explicit reference to Bayes Theorem

11. And especially note….. When H’≠ not H the notion that the LR is a measure of probative value of evidence is tenuous and potentially misleading

12. When H’ ≠ not H… • Knowing that LR>1 just tells us that the ratio of posterior probabilities (of H and H’) is greater than the ratio of prior probabilities • So all we can conclude is E supports H more than it supports H’. But E may not support H at all because we can still have P(H|E) < P(H) • Similarly LR=1 only tells us E supports both H and H’ equally.That does NOT mean E is neutral; P(H|E) might be very different to P(H)

13. Was Mrs Peacock the murderer? H: “Mrs Peacock guilty” E: “The murderer was female P(E | H) = 1 P(E | not H) = 2/5 LR= 2.5 But if H’: “Miss Scarlet guilty” P(E | H’) = 1 LR=1

14. Issues and limitations of the LR • ‘probative value’ is not what people think it means when H’ is different from not H • But it is difficult to work with exhaustive pairs of hypotheses • Priors can never be truly ignored • Evidence E is rarely ‘simple’ – normally involves E1 and E2 that require separate likelihoods • Can be difficult even to avoid non-mutually exclusive hypotheses in practice • Even if we get it all right LR of source level hypotheses tells us NOTHING about LR of offense level hypotheses

15. Example Fred and Joe live at the same address. Gun X is registered to that address. Bob is found murdered from a gun shot. Evidence E: “there is a gun in Fred’s house with FDR that matched that from the crime scene.” Fred is charged with the murder of Bob. The offence level hypotheses are: Hp: Fred fired the shot that killed Bob not Hp: Fred did not fire the shot that killed Bob The source level hypotheses are: Hp1: Fred owned gun that killed Bob not Hp1: Fred did not own gun that killed Bob

16. Some reasonable assumptions Essentially irrelevant LR=1 for source level hypotheses….. but Ehas real probative value on Hp

17. Prior state of the BN

18. Calculating the probability of evidence E under the two values for H1p P(E | H1p) = 0.9891 (unchanged from prior) P(E | not H1p) = 0.9891 (unchanged from prior)

19. Evidence is observed Probability of Hp jumps from 1% to over 9% ..the evidence is certainly not neutral

20. 2. Revisiting the LR in two well-known cases

21. R v Sally Clark 1999-2003 Convicted and ultimately cleared of murdering her 2 children

22. Sally Clark Revisited: A new issue in the probability experts’ reasoning Hd : Sally Clark’s two babies died of SIDS Hp : Sally Clark murdered her two babies “(Prior) probability of Hd over 100 times greater than (prior) probability of Hp” “So assuming LR of 5 posterior of Hd still 20 greater Hd : Sally Clark’s two babies died of SIDS Hp : Sally Clark murdered at least one of her two babies. (Prior) probability of Hd only 2.5 times greater than the (prior) probability of Hp

23. R v Barry George, 2001-2007 Jill Dando

24. R v Barry George (revisiting the Appeal Court judgment) Hp: Hypothesis “BG was man who shot JD” E: “Single particle of FDR matching that from the gun that killed JD found in BG coat pocket Defence likelihood P(E|notHp) = 1/100 …But Prosecution likelihood P(E| Hp) = 1/100 So LR = 1 and evidence ‘has no probative value’ But the appeal transcript suggests a problem… Fenton, N. E., D. Berger, D. Lagnado, M. Neil and A. Hsu, "When ‘neutral’ evidence still has probative value (with implications from the Barry George Case)", Science and Justice, Volume 54, Issue 4, Pages 274–287, July 2014

25. Confusion from experts about the hypotheses • Not clear that Hp stated was really the same prosecution hypothesis considered by the experts • H1p: “The particle found in BG’s pocket came from a gun fired by BG”. • H2p: “The particle found in BG’s pocket came from the gun that killed JD”. • Transcript suggests the experts did not adhere to the assumption that defence hypothesis Hd was simply “not Hp”, i.e. “BG was not the man who shot JD”. • H1d: “Integrity of BG coat was corrupted”

26. LR=1? P(E | Hp) = P(E | H1d) but the evidence E is not neutral as concluded by expert and accepted by the court. It favours Hp.

27. 3. Why Bayesian networks are needed

28. More comprehensive BN model needed in BG case

29. Even single piece of forensic match evidence is NOT a 2-node BN Target is type X Target is source Target tested X Source is type X Source is type X Source tested X

30. Bayesian nets: what we need to stress Separate out assumptions from calculations Can incorporate subjective, expert judgement Can address the standard resistance to using subjective probabilities by using ranges. Easily show results from different assumptions …but must be seen as the ‘calculator’

31. The potential of Bayesian Networks “I assert that we now have a technology that is ready for use, not just by the scholars of evidence, but by trial lawyers.” Edwards, W. (1991). "Influence Diagrams, Bayesian Imperialism, and the Collins case: an appeal to reason." Cardozo Law Review 13: 1025-107

32. 4. Conclusions and way forward

33. Summary LR and probative value of evidence may not be what people think it is In isolation the LR may be highly misleading Doing things correctly requires fuller models - BNs But Bayesian arguments cannot be presented from first principles

34. Blatant Plug for Book CRC Press, ISBN: 9781439809105 , ISBN 10: 1439809100