Evaluating Information

1. Evaluating Information MMC 4420

2. Introduction • Class goal is not to make you a professional researcher, but to make you better at evaluating information, especially research reports • You will have to keep up-to-date in your field. You will have to make decisions based on research. How will you know you are making the right decision?

3. Introduction • Why worry about errors when we are dealing with experts? • BECAUSE THEY MAKE THEM! • Look for error…… • Externally: compare with what is already known • Internally: (our work) checking sampling, measurement instrument, analyzing results and conclusions

4. Quote • “An educated person is one who has learned that information almost always turns out to be at best incomplete and very often false, misleading, fictitious, mendacious – just dead wrong.”

5. Science as a way of knowing • Accurate observation and measurement followed by accurate analysis and conclusions • Reliance on evidence • A need to control error • Standard of publicness

6. 2 fundamental concerns • Are the reported results correct? • If they are, to whom do these results apply? (generalizability)

7. Approaching information • Be prepared to challenge the methods used • Constructs, indicators • Assumptions made • Create counterarguments • Look at additional arguments • Check generalizability

8. Supporting research conclusions • With simple reason and claims of plausibility (weak) • With replications (best) • Usually in lit review of article • Using good research methods • Readers must assess the arguments, the methods, the analysis, the conclusions

9. Questions • What additional sources should you consider? • What errors are most likely to occur with each source? • Do the methods used meet the norms of science? (Reliance on evidence, need to control error, standard of publicness)

10. The Error Model: Our Guide • Identifying and removing/reducing sources of potential error • Observation • Communication • Interpretation • ALL could increase error Increase Decrease Error Error Observation Communication Interpretation

11. Observation • Both football fans and researchers can exhibit partisanship • Cannot see EVERYTHING, even if you wanted to • Still interpret observations • Interaction during observations • Expectations of researchers

12. So now what? • ASK: • What is left out? • Are the researcher’s interpretations of the findings acceptable? • What effect did the instruments have on the observations? • Who did the observation? • Where and when were the observations made?

13. Quote • “The world is not a fixed solid array of objects, for it cannot be fully separated from our perception of it. It shifts under our gaze, it interacts with us, and the knowledge that it yields has to be interpreted by us.”

14. Quote • “An empiricist . . . thinks he believes only what he sees, but he is much better at believing than at seeing.”

15. Communication also adds error • An article never represents all that was done, learned during study. So what was left out? • Unconscious and conscious decisions • Not only what was left out of report, but language used

16. Words have an effect on error • Denotative vs. connotative meaning (EX: collision) • Why was article written? Usually clear from first few paragraphs. • Untenured professors HAVE to publish…. Or perish, as the saying goes. • Journal referees supposed to catch all mis-communications • Some journals better than others (acc. rate)

17. Questions to ask? • Is there anything missing that might be important? • Are assumptions/definitions acceptable? • Is the language neutral?

18. Quotes • “There have always been more historians who were more concerned that truth should be on their side than they should be on the side if truth.” • “It is no use looking to scientific papers, for they do not merely conceal, but actively misrepresent the reasoning which goes into the work they describe.”

19. You can be part of the error • You probably don’t read the entire article – at least not carefully, thoughtfully, critically • Your “frame,” your interpretation of the info can be a source of error • Don’t toss out an entire study just because you may disagree with one part • Communication is a two-way street: be open to what the researchers are saying

20. Ask yourself • Are you being close-minded re: the journal or the author? • Are you dismissing the entire article because you disagree with one part? • Do you have pre-conceived notions about the information in the article? • If so, can you suspend them with an open mind?

21. Quote • “Into every act of knowing there enters a passionate contribution of the person knowing what is being known and this coefficient is no mere imperfection, but a vital component of his knowledge.”

22. The nature of error • 2 fundamental types: bias and noise • Bias is systematic error; noise is random error Factors that increase error: Observation Communication Interpretation Factors that increase error: Observation Communication Interpretation Factors that increase error: Observation Communication Interpretation Factors that decrease error: Types of error: bias, noise Types of error: bias, noise Types of error: bias, noise

23. Bias • You must identify possible sources of bias -Researcher conducting the study -The subjects being studied -The research plan

24. The Researcher • Researcher may have expectations, which may be communicated to subjects by accident • Age and sex of interviewer/researcher • Inadequate training • Intentional deception

25. Subjects • Some will try to “psych out” the investigation as an intellectual exercise • Hawthorne Effect • If subjects become “different” by being studied, what happens to generalizability?

26. Research Plan • In an experiment, the use of a pre-test can sensitize the subjects to the study • A longitudinal study can also sensitize subjects • Selection of subjects: who will be studied and where and when the study will be done

27. Bias and error • Bias is – one type of error -- in every study, but it does not always signal a “bad” study • Just determine if bias has occurred, then determine whether it is a major concern • It can usually be reduced, and sometimes eliminated by proper research techniques

28. Questions to ask • What are the potential sources of bias? Are any undetected by the researcher? • What procedures did the researcher do to eliminate or reduce bias? • Which biases might be important effects on the observations, measurements?

29. Quote • “Finnegan’s Finagling Factor: That quantity which, when multiplied times, divided by, added to or subtracted from the answer you got . . . gives you the answer you should have gotten.” • What does the above quote tie in to that we’ve studied?

30. Noise: The “other” error • Noise is unsystematic or random error • Noise errors tend to even out, so bias is worse • Bias is more likely to be unnoticed, so it can’t be controlled • Researcher’s job is to eliminate or correct for bias and to keep noise as low as possible • In any single study, noise can be as bad as bias

31. Noise and statistics • To avoid having to run multiple studies to get noise evened out, you use statistical inference • In essence, an estimate is made of what would happen if the study were done multiple times • This estimate is made with inferential statistics

32. Sources of noise • People: You can’t control certain things about your subjects, such as illness or whether they got a good night’s sleep, and so on • Procedures: You must treat all subjects the same, keep the equipment and setting the same, etc.

33. Ask questions • What are the potential sources of noise? • What did the researcher do to minimize the noise? • Was statistical inference used and was it used properly?

34. What is being studied? • Checking the researcher’s work, i.e., Can you believe it? Factors that increase error: Observation Communication Interpretation Factors that decrease error: Subject Measurement Description Relationships Inference Types of error: bias, noise

35. Measurement • Bias and noise errors can show up here • Instrument error • Measurer error • Reliability obviously an issue • Even validity (is GRE a predictor, measure of knowledge, either?) • E.g., multiple choice involves chance correct

36. Grading • You are most familiar with the assigning of grades to tests and classes that are intended to measure mastery of the material. What are some of the places where bias and noise errors can creep in to this process?

37. Evaluating Control • Can you think of an explanation for what happened other than the one given? • Did the researcher eliminate potential rival explanations? • Was more than one group studied so comparisons can be made? • Were the groups identical in all essential ways? • Are all parts of the proposed cause needed to produce the effect (or could only part of the cause work)?

38. Generalizing • Must be a random sample • Can’t be too small, but need not be huge • Sample size is not related to population size • Response rate • Even with a good sample, other threats exist: sensitizing, for instance

39. The End • There is no certain road to Truth (T=o+e) • A careful examination of research will provide evidence re factual accuracy and generalization of findings, but it is not proof. • We need to increase our tolerance of uncertainty. • Search for Truth among the patterns of similarity in numerous studies.