Interesting Examples of the Use & Misuse of Statistics from the Law

1. Interesting Examples of the Use & Misuse of Statistics from the Law H. James Norton, PhD norton100@bellsouth.net Website: www.jimnortonphd.com

2. Oliver Wendell Holmes, Jr.

3. Oliver Wendell Holmes, Jr.The Path of the Law10 Harvard Law Review (1897): 457-469. “For the rational study of the law the black letter man may be the man of the present, but the man of the future is the man of statistics and the master of economics.”

4. People v. Collins (1968)Crim. No. 11176Supreme Court of CaliforniaMarch 11, 1968 • A woman had her purse stolen. • The witnesses did not get a good look at the robber’s face. • Witnesses were able to describe some characteristics of the robber, the get-away car, and the driver. • Prosecution calls an Instructor of Mathematics to testify. • Instructor explains the product rule for multiplying probabilities of independent events.

5. Prosecutor suggests these probabilities: • Black man with a beard 1 in 10 • Man with a moustache 1 in 4 • White woman with pony tail 1 in 10 • White woman with blonde hair 1 in 3 • Yellow automobile 1 in 10 • Interracial couple in car 1 in 1000 • Asks instructor what the probability would be under these estimates. • 1 in 12,000,000. • Prosecutor claims these estimates are conservative. • “Chances of having every similarity … something like 1 in a billion.” • Jury finds defendant guilty.

6. The ruling of the appeal’s court: • “It is a curious circumstance of this adventure in proof that the prosecutor not only made his own assertions of these factors in the hope that they were conservative… but invited the jury to substitute their estimates.” • “There was another glaring defect in the prosecution’s technique, namely an inadequate proof of the statistical independence of the six factors.”

7. The final ruling of the appeals court: “Mathematics, a veritable sorcerer in our computerized world, while assisting the trier of fact in the search for truth, must not cast a spell over him. We reverse the judgment.”

8. The Sally Clark Case

9. Sally Clark was a solicitor in Cheshire, England. • Her son, Harry Clark, born 3 weeks premature, died 8 weeks after birth. • In addition, her first child had died less than 3 weeks after birth. His autopsy concluded he had died of natural causes. He had signs of a respiratory infection. • She was arrested for 2 counts of murder, despite the fact that there was very little evidence against her.

10. Sally had no history of violent or unusual behavior. Harry had some evidence of being shaken but this was consistent with her report to the police that she had shaken the baby when she noticed that he was not breathing. • Prosecutor’s main argument was that it would be very unlikely that 2 babies in same family would die of cot death. In the U.S. we would use the term Sudden Infant Death Syndrome (SIDS).

11. Prosecution calls Sir Roy Meadow Professor of Paediatrics St. James University Hospital President British Paediatric Association 1994-1997

12. HistestimonywasbasedontheConfidentialEnquiryforStillbirthsandDeaths,astudy of deaths of babiesininfancy, in 5 regions of England from 1993 to 1996. • Probability random baby dies of a cot death = 1 in 1303. • Probability random baby dies of a cot death if the mother is > 26 years old, affluent, and a non smoker = 1 in 8543. • Probability two children from such a family both die from a cot death = (1 in 8543) x (1 in 8543) = 1 chance in 73 million.

13. Judge’s summary to jury, “Although we do not convict people in these courts on statistics, … the statistics in this case are compelling.” • Jury convicts on a 10 to 2 vote. • One juror said, “Whatever you say about Sally Clark, you can’t get round the 1 in 73 million figure.” • Sally’s conviction upheld on appeal.

14. 2001, Royal Statistical Society issues a news brief condemning the use of the multiplication rule for independence in this case. • “This approach is statistically invalid. … The well publicized figure of 1 in 73 million has no statistical basis.” • 2002, Ray Hill, Professor of Mathematics at the University of Salford, analyses other published data. He concludes the probability of having a second child die a cot death, given a first child in a family died a cot death, may be as high as 1 in 60.

15. In 2003, after spending 3 years in jail, Sally’s second appeal was upheld, and she was released from jail. This was only after a new pro bono lawyer, while reviewing the evidence, discovered a pathology report revealing that Harry was infected with staphylococcus aureus and that this fact had been hidden from her defense team. • Two other women whom Meadow had testified against at the murder trial of their children were released upon appeal. • In 2007, Sally Clark died, of apparently natural causes, due to acute alcohol intoxication.

16. New Evidence on S. aureus & SIDS • “Infection and sudden unexpected death in infancy (SUDI): a systematic retrospective case review. M.A. Weber. Lancet May 31 2008;371:1848-53. • “Significantly more cultures from infants whose death was unexplained contained S. aureus (262/1628, 16%) than did those from infants whose deaths were of a non infective cause (19/211, 9%, p=0.005). • From editorial by Morris, “but this work … provides support for the idea that S. aureus and E. coli could have a causal role in some cases of unexplained SUDI.”

17. The Case of Lucia de Berk Lucia de Berk, a nurse, was sentenced to life imprisonment in a Dutch court in 2003 on 7 counts of murder and 3 counts of attempted murder in 3 hospitals where she worked. The circumstantial evidence against her was that she was on duty when several unexplained deaths had occurred. A law psychologist stated at the trial that the chance of a nurse working at the 3 hospitals and being present for so many unexplained deaths was 1 in 342 million. The probability was based on the number of suspicious deaths on the shifts she worked versus the number of suspicious deaths on the shifts she did not work. Her conviction was upheld on appeal in 2004.

18. After her incarceration, experts in statistics, toxicology, and medicine reviewed the evidence used to convict her. • A report requested by the prosecution after the verdict but prior to the appeal shows the level of digoxin in the first unexplained death was not a lethal concentration. • The way decisions were made concerning the gathering of the evidence are criticized. For instance in the case of one of the deaths, 5 experts reported that the death was not suspicious but 1 expert disagreed and his opinion was used to classify the death. In another, an expert changed his opinion after becoming aware of the controversy. • Statisticians report that the figure 1 in 342 million was incorrectly calculated and was meaningless. Several do their own analysis and report estimates of 1 in 50 or 1 in 9.

19. A witness against Lucia was a detainee in the criminal psychological observation unit who testified she told him that “I released these 13 people from their suffering.” He recanted his testimony. • A key toxicologist who testified for the prosecution decided , based upon new medical information, that his original conclusions may not be correct. • It has been proven that a medical specialist and his assistant were with the first baby that died and her statement that she was not near the baby at the time it died was true. • It has been noted that in one unit 7 suspicious deaths occurred prior to her working on the unit and only 6 occurred in a similar time-period after she joined. • In 2008 the Dutch Supreme Court freed Lucia until a new trial could take place. • In December 2009 a court ruled that recently gathered evidence confirms that 3 of the deaths were due to natural causes.

20. Further court hearings took place March 17, 2010. • Prosecutor says that the statenow believes that Lucia de Berk did not murder or attempt to murder anyone. • However the Prosecutor felt obliged to remind the court that during the investigation they found evidence that Lucia had committed the following crimes: 1. Exaggerated her high school grades. 2. Stole a tube of disinfectant gel. 3. Did not return 2 library books. • While Lucia was in jail for approximately 5 years she had a stroke.

21. TheCaseofRaymondEason • In 1999, Raymond Eason, a 49 year old man living in Swindon England, was arrested for burglary. • His DNA had a 6 loci match with evidence left at the scene. • Forensic scientists reported to the prosecutor that this DNA pattern would only occur in 1 in 37 million people. • Pretty convincing evidence? But how did his DNA come to match the evidence at the crime scene?

22. His blood sample had been taken when he was arrested, but later released, in a domestic dispute. • He had an alibi that he was home taking care of a sick daughter. The crime was committed 200 miles from his home. • The results of the DNA analysis was entered into a database with 700,000 suspects. • Comparing one sample to many others in a database increases the likelihood of finding a match by chance. • The complexity of this statistical calculation is explained in “The Evaluation of Forensic DNA Evidence.” • A rule of thumb, that some forensic scientists use, is to divide the match probability by the number of subjects in the database.

23. If this is done the match probability is 0.0027 • His lawyer asked for the DNA sample to be reanalyzed by another laboratory. • The laboratory found the same 6 loci match, but found his DNA did not match on 4 newly tested loci. • He was released from jail. • Oh, by the way, Mr. Eason had severe Parkinson’s disease. He could not drive a car, and could barely dress himself. • As the appeals court judge in the Collins case had warned, the prosecutor had let the numbers cast a spell over his common sense!

24. 228 Conn. 610 State of Connecticut v. Roy E. Skipper. No. 14744. Supreme Court of Connecticut. Argued Dept 23, 1993. Decided Feb. 22, 1994 Defendant was convicted in the Superior Court of sexual assault in the second degree and risk to injury of child. Defendant appealed. The Appellate Court transferred the case to The State of Connecticut Supreme Court.

25. A victim (a young girl) gave a statement to the police. The next day, the victim’s mother took her for a medical examination that revealed she was pregnant. The victim had an abortion. One of the key points for the prosecution at the defendant’s trial was evidence comparing the DNA of the defendant to the DNA of the fetus. An expert witness testified at the trial regarding the defendant’s paternity index (PI). He calculated this to be 3496, “indicating that only 1 out of 3497 randomly selected males would have phenotypes compatible with the fetus in question.” “McElfresh further testified that the PI could be converted into a statistics indicating the percentage of the defendant’s probability of paternity.” He did this using Bayes’ Theorem and derived a value of 99.97 percent. However, when Bayes’ Theorem is used in this calculation a prior probability of paternity is required. Bayes’ Theorem in and of itself does not cast this prior as any particular number. McElfresh set the prior odds to 50%. “Generally experts.. choose this number.”

26. The ruling of the court: “Although the probability of paternity is the mathematically equivalent of the PI if a 50% prior probability is used, the two statistics have different connotations. In the present case, the paternity index of 3496 means that if 3497 random males were chosen, 3496 would not have a DNA profile compatible with producing the fetus in question . … It is the application of Bayes’ Theorem that says the likelihood that this defendant is the father of the fetus in question is 99.97%.” “If we assume the presumption of innocence stand would require the prior probability to be zero, the probability of paternity in a criminal case would always be 0 …”

27. ”We conclude that the trial court should not have admitted the expert testimony stating a probability of paternity statistic.” Supreme Court held that the admission of probability of paternity calculated based on Bayes’ Theorem was reversible error. … Reversed and remanded for new trial. 637 Atlantic Reporter, 2d Series, pages 1101-1110

28. The purported first courtroom case using statistical evidence reported by Meier & Zabell (1980) • The contested will of Silvia Ann Howland by her niece Hetty Robinson in 1865. • She claimed an earlier will in 1862 left the entire estate to her and the will stated no later will should be honored.

29. The attorney for the estate claimed the signature on the earlier will was a forgery. • Her attorney called Oliver Wendell Holmes, Sr., Professor of Anatomy and Physiology at Harvard, to testify about the authenticity of the signature. • He examined the signature under a microscope and found no evidence of a forgery.

30. The attorney for the executor called Benjamin Peirce, Professor of Mathematics at Harvard, to testify. • Peirce used statistical techniques to decide if the signature was “too similar” to another known signature of Silvia Howland. • His concluded that the “probability of finding 30 matches in a pair of signatures was once in 2,666 millions of millions of millions.”

31. Meier and Zabell criticize Peirce’s methods of supporting his model by means of a graphical test of goodness-of-fit. • They discuss the “use and abuse of the product rule for multiplying probabilities of independent events.” • Court ruled that there was lack of evidence to support Hatty’s claim that “the 1862 will was in fulfillment of a contract.”

32. U.S. v. Burton The case involved a Sergeant who was accused of obtaining an illegal copy of an exam that was used as a basis for promotions. However, no copy of the exam was found on the defendant, at his home, or workplace. The prosecution presented a statistical argument that they claimed was strong evidence that the defendant must have had a copy of the test prior to the exam. This statistical argument had been used in 4 previous court-martials. I (hjn) was asked to help refute this assertion.

33. Evidence against Sergeant Burton • The testimony of Sergeant X. • The fact that Sergeant Burton was a friend of another Air Force Sergeant at the Pentagon who had access to the exam. No charges were ever brought up against this person. • Sergeant Burton scored high on the test, approximately at the 95th percentile. The information the test is based upon is from a 120-page manual that everyone taking the test has access to and they are encouraged to study from. However, Sergeant Burton’s greatest improvement came the year before when he improved from the 40th percentile to the 85th percentile. There was no evidence that he had cheated in the prior exam. • The statistical procedure developed by a Colonel in the Mathematics Department at the Air Force Academy.

34. The statistical procedure consisted of: • For each wrong answer the percent of soldiers choosing this answer was found. For instance if Sergeant Burton answered question #1 incorrectly, for example he choose answer B while answer A was correct, and 20% of the soldiers answered B, then 0.20 was recorded. • This was done for each of his 11 incorrect answers. • These probabilities were multiplied together. • Their conclusion was since the product of these numbers was very small, he choose an unusual set of incorrect answers. This implied he had a copy of the exam prior to the test.

35. The following are some of the points that I brought up with the defense lawyers: • The method employed seemed to have nothing to do with proving cheating or even with choosing unusual answers. Was there any published reference that justified this procedure? • There was a paradox in their procedure, as the more questions one got incorrect, the smaller the probability. Did this make sense that the more questions one got wrong the more evidence of cheating? • The simple product rule for probability requires independence. What evidence was there that the questions were independent? • How many soldiers had a lower product than Sergeant Burton?

36. A Major from the Air Force Academy was brought in to defend the Colonel’s statistical method. • During the pretrial motion to suppress the “statistical evidence”the Major presented an “ad hoc” method to show independence of the test questions. • Military law states that, “the test for determining whether to admit expert scientific testimony does not apply to admissibility of expert testimony based on technical or specialized knowledge.” • The judge ruled that statistics is not a science but specialized knowledge. • “Military justice is to justice what military music is to music.” • The Major was rigorously cross-examined at the trial. • Defense calls only 3 witnesses to testify against Sgt. X. • Jury deliberates 40 minutes and returns a not-guilty verdict.

37. Comparing bullets using Compositional Analysis of Bullet Lead (CABL) Bullet from Crime Scene • “Since the 1960s, FBI testimony in thousands of criminal cases has relied on evidence from Compositional Analysis of Bullet Lead (CABL), a forensic technique that compares the elemental composition of bullets found at a crime scene to the elemental composition of bullets found in a suspect's possession. The report assesses the scientific validity of CABL, finding that the FBI should use a different statistical analysis for the technique, and that, given variations in bullets manufacturing processes, expert witnesses should make clear the very limited conclusions that CABL results can support. “ • Authors: Committee on Scientific Assessment of Bullet Lead Elemental Composition Comparison, National Research Council = ? Bullet in Subject’s Possession