A Blind Analysis

1. A Blind Analysis You are not allowed to peek! Prof. Donald Koetke Senior Research Professor of Physics Valparaiso University

2. Outline • What is a “blind analysis”? • What is a “bias”? • Does “bias” = “systematic error”? • What is the special (subtle) bias? • What is the history of the blind analysis”? • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ? • Some thoughts and reflections -- VU Colloquium

3. Outline • What is a “blind analysis”? • What is a “bias”? • Does “bias” = “systematic error”? • What is the special (subtle) bias? • What is the history of the blind analysis”? • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ? • Some thoughts and reflections -- VU Colloquium

4. Outline • What is a “blind analysis”? • What is a “bias”? • Does “bias” = “systematic error”? • What is the special (subtle) bias? • What is the history of the blind analysis”? • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ? • Some thoughts and reflections -- VU Colloquium

5. Outline • What is a “blind analysis”? • What is a “bias”? • Does “bias” = “systematic error”? • What is the special (subtle) bias? • What is the history of the blind analysis”? • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ? • Some thoughts and reflections -- VU Colloquium

6. Outline • What is a “blind analysis”? • What is a “bias”? • Does “bias” = “systematic error”? • What is the special (subtle) bias? • What is the history of the blind analysis”? • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ? • Some thoughts and reflections -- VU Colloquium

7. Outline • What is a “blind analysis”? • What is a “bias”? • Does “bias” = “systematic error”? • What is the special (subtle) bias? • What is the history of the blind analysis”? • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ? • Some thoughts and reflections -- VU Colloquium

8. A Blind Analysis What is a “bias”? Does “bias” = “systematic error”? What is the special (subtle) bias? “A prejudice in favor of or against” -- may bedue to computer codes, equipment performanceor setting, decisions/selections/cuts imposed, etc. Yes - if it causes a systematic shift in a result(This is not a “mistake” or “blunder”.) Experimenter biasin making decisions/choices to achieve a desired answer. Examples … VU Colloquium

9. A Blind Analysis You do an experiment and… You know what the answer “should be” -i.e., you know what the “accepted” answer is -- but your answer is different -outside of errors.What do you do? Asymmetry!! You make a measurement and within errorsit agrees with the “accepted” answer --What do you do? VU Colloquium

10. A Blind Analysis The goal of a “blind analysis” is to preventthe experimenter from (unconsciously - or consciously) making decisions in the analysis that would affect the result based on: • The predictions of a model or theory (e.g., the Standard Model for particle physics) • Previous measurements known to the experimenter • The experimenter’s intuition or other predisposition These are all examples of personal bias VU Colloquium

11. Computer codes have been developed and tested • Decisions about the number of events (trials) needed have been made (cuts have been selected) • Apriori agreements are reached about what to do when the real answers from the experiment are revealed; no further analysis! A Blind Analysis What is a “blind analysis”? A “blind analysis” is an analysis of measured data in which the final answer is kept hidden from the experimenters until all of the decisions about the analysis have been made: VU Colloquium

12. Outline • What is a “blind analysis”? • What is a “bias”? • Does “bias” = “systematic error”? • What is the special (subtle) bias? • What is the history of the blind analysis”? • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ? • Some thoughts and reflections -- VU Colloquium

13. Medical research -- blind tests! Patients don’t know whether they are getting -- the medicine/treatment, or, a placebo Patients are assigned to (a) or (b) ramdomly History Blind analysis begins in ~1930s with - Therefore - the patients are “blind” to their treatment They may imagine symptoms or cure, etc., but only the researcher/physician knows if these can be due to the medicine/treatment. VU Colloquium

14. History Blind analysis begins in ~1930s with - Medical research -- blind tests! • Patients talk to medical researcher e.g., how are you feeling? is pain less or more? are the other problems? serious or not so serious? before or after you take the medicine? …etc. • Researcher hears what patient says • Researcher examines the patient • Researcher records all this as “data” Do you see a problem here? c.f.,1937 JAMA 26 June 2178/2. VU Colloquium

15. History Blind analysis begins in ~1930s with - Medical research -- blind tests! The patients are “blind” to their treatment - but, the researcher is NOT! • Therefore, the medical researcher can (and will) unconsciously and unintentionally interpret and record the information (data) with this bias (knowledge). The solution is… VU Colloquium

16. 1948 Am. Heart Jrnl., XXXVI, 529. 1950 Am. Jrnl. Med., IX, 142/1. History Blind analysis begins in ~1930s with - Medical research -- The double blind test! The patients are “blind” to their treatment - and, the researcher is “blind” to who is getting treated! • The list of patients and their treatment is prepared and maintained by someone who is NOT participating in the research. The list is sealed in the “black box”. • The list can be retrieved from the “black box” only after all the analysis is completed. VU Colloquium

17. History Blind analysis begins in ~1930s with - Medical research -- The double blind test! Consider a simple analysis -- “Was the medicine effective?” • The data on all patients will indicate whether the medication was effective for each patient. • When this data analysis is complete (and free from mistakes) and any disagreement among the researchers have been settled and any concerns about bias have been removed, i.e., then (and only then) - • The “black box” can be opened and the list can be retrieved. • The data on all patients is now grouped into two groups: group (a) and group (b) VU Colloquium

18. History Blind analysis begins in ~1930s with - Medical research -- The double blind test! The simple analysis -- “Was the medicine effective?” • For what fraction of group (a) [fa] and group (b) [fb] was the medicine effective? • Do a statistical analysis to determine whether the difference in these two fractions is statistically significant - or whether it is consistent with a random occurrence.The result is now known! • The researchers agree to accept the results obtained and no further analysis of the data is permitted - unless there has been a blunder (mistake) - in which case fix the blunder and report both results and the nature of the “fix”. VU Colloquium

19. History Blind analysis has become the standard methodology in clinical trials. Blind analysis has been used in the physical sciences only in recent years. Physicists(and astronomers) are, of course ---- Careful-- Quantitative-- Attentive to bias (systemtic errors) Why should they need a blind analysis? VU Colloquium

20. History Ernest Rutherford (1934) -- “It seems to me that in some way it is regrettable that we had a theory of the positive electron before the beginning of the experiments. Blackett* did everything possible not to be influenced by the theory, but the way of anticipating results must inevitably be influenced to some extent by the theory. I would have liked it better if the theory had arrived after the experimental facts had been established.”Ernest Rutherford, Proc. Solvay Conference, (Gauthier-Villars, Paris 1934), p 177.*Nobel prize in physics for discovering the positron (1948) VU Colloquium

21. In every case the data agreed with the theoretical ratios within less than the standard errors. Taking the whole together, 2 was 41.6 on 84 degrees of freedom, giving a probability that hewould have measured this well to be only7x10-5 !! History Gregor Mendel (1865) -- The classic case is Gregor Mendel’s work on inheritance. VU Colloquium

22. History Modern examples -- From nuclear and subnuclear physics --- Reason: It’s the field in which the blind analysis techniques have been widely used - and - It is the field with which I am most familiar. VU Colloquium

23. History VU Colloquium

24. History VU Colloquium

25. B-meson lifetime ratio The mean ratio has a 2 of 4.5 for 13 degrees of freedom; P=0.985 History 8 Major experiments Over 2000 physicists Bottom line: The agreement appears to be too good!! VU Colloquium

26. Outline • What is a “blind analysis”? • What is a “bias”? • Does “bias” = “systematic error”? • What is the special (subtle) bias? • What is the history of the blind analysis”? • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ? • Some thoughts and reflections -- VU Colloquium

27. An example of a “search” for an uncommon occurrence A reaction that violates the Standard Modelconservation of lepton number If you do not find the reaction, you have not shown the Standard Model to be incorrect If you do find the reaction, you had better be very sure that you have got it right!!! VU Colloquium

28. Signal: Ee = E = 52.8 MeV Boxes represent 2 boundaries Examples MEGA Search for   e Question: When and How are the values of  determined? VU Colloquium

29. Signal: Ee = E = 52.8 MeV Boxes represent 2 boundaries Examples MEGA Search for   e Blind Analysis: The boundaries aredetermined before the analysis of (most of) the data! VU Colloquium

30. VU Colloquium

31. Examples p = 29.8 MeV/c Measure: pe() TWIST High precision   ee A search for physics beyond (not included in) the Standard Model. VU Colloquium

32. Analyzed expimentaldata Nexp(x) = a’ + b’x Simulation ofthe experimentusing a,b Number N(x) = a + bx a,b from theory X = E/Emax Examples Finda’ &b’ bycomparison ofreal datawithsimulateddata The detector system distorts the distribution Simulation = “Monte Carlo” VU Colloquium

33. Examples Muon decay spectrum Current SM  = -0.007 ± 0.013 0  = 0.7518 ± 0.0026 3/4  = 0.7486 ± 0.0026 ± 0.0028 3/4 P = 1.0027 ± 0.0079 ± 0.0030 1 VU Colloquium

34. Energy Cos() Examples Muon decay spectrum from Standard Model VU Colloquium

35. TWISTmeasured spectrum Standard Model spectrum Monte Carlo computer code ,,, Energy Energy Cos() Cos() Examples VU Colloquium

36. Compare spectra  ’,’,’,’ TWISTmeasured spectrum TWISTsimulated spectrum ’,’,’,’ ,,, Energy Energy Cos() Cos() Examples Problem! This spectrum is the S.M. spectrum -- nothing is hidden; the experimenter is not “blinded”. VU Colloquium

37. TWISTmeasured spectrum Standard Model spectrum Monte Carlo computer code o,o,o,o Energy Energy Cos() Cos() Examples Unknown spectrum o,o,o,o are generated randomly, are encrypted, stored secretly, and used to generate the simulated data. Nobody knows what the offsets from ,,, are. VU Colloquium

38. TWISTmeasured spectrum TWISTsimulated spectrum ? Energy Energy Cos() Cos() Examples Compare spectra to get  , , ,  ’,’,’,’  = ’ - o = ’ - o = ’ - o = ’ - o VU Colloquium

39. o,o,o,o Examples • Before you can open the black box: • Get all computer codes working and tested • Identify all sources of systematic error and evaluate the size of each one • Take all of the data you will need including data to help estimate the systematic errors • Be sure that the Monte Carlo programs accurately simulate your experiment or you will have false values • Analyze all of the data you intend to use to get the result VU Colloquium

40. o,o,o,o Measure Calculate the results Examples  = ’ - o = ’ - o = ’ - o = ’ - o Compare with Standard Model predictions Write the paper VU Colloquium

41. Examples Let’s look at one of your experiments from PHYS-245 VU Colloquium

42.  2 1 osc D Examples A measurement of the speed of light Present accepted value: c = 299,792,458 m/sc  3 x 108 m/s VU Colloquium

43.  2 1 osc D Examples Blinded A measurement of the speed of light Present accepted value: c = 299,792,458 m/sc  3 x 108 m/s VU Colloquium

44. Outline • What is a “blind analysis”? • What is a “bias”? • Does “bias” = “systematic error”? • What is the special (subtle) bias? • What is the history of the blind analysis”? • What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ? • Some thoughts and reflections -- VU Colloquium

45. Reflections Guide to a blind analysis:If my answer were to come out to be six standard deviations from the expected result, what would I do? Make the list, and then -do all of that before you look at the answer! A blind analysis is intended to guard against experimenter bias. It will not guard against fraud; that requires integrity and honesty. A blind analysis is likely not the fastest way to an answer. VU Colloquium

46. Reflections It is not always possible to achieve perfect “blindness” -- e.g., drug testing. Don’t need to plan everything in the analysisbefore beginning; just keep the answer hidden. A blind analysis may not work for every experiment -but it is worth investigating before you begin. In a blind analysis you want to hide the answer from anyone else who might want to offer advice that may be based on the answer. VU Colloquium

47. Reflections Thankyou A blind analysis removes the “comfort factor” - knowing what answer you are getting so you can make changes, do more analysis, repeat measurements, etc., if the answer is not what you expect. But, that may not be the best science! VU Colloquium

48. VU Colloquium