Introduction to Research Methods

1. Introduction to Research Methods Research Methods for Public Administrators Dr. Gail Johnson Dr. G. Johnson, www.researchdemystified.org

2. Numbers and Facts Any policy debate is awash in numbers that are presented as absolute fact. But are they? Or are we being tricked into “premature certainty” because numbers appear concrete? Dr. G. Johnson, www.researchdemystified.org

3. In the News • According to a 2009 study done for the American Health Insurance Plans by Price Waterhouse and Coopers: • The average family health insurance coverage costs approximately $12,300 today • Key finding: The overall impact of the proposals for health care reform will be to increase the cost of private insurance coverage for individuals, families, and businesses above what these costs would be in the absence of reform. Dr. G. Johnson, www.researchdemystified.org

4. Their Numbers • Health Insurance premiums could be expected to increase to approximately: • $15,500 in 2013 under current law and to $17,200 if these provisions are implemented. • $18,400 in 2016 under current law and to $21,300 if these provisions are implemented. • $21,900 in 2019 under current law and to $25,900 if these provisions are implemented. Dr. G. Johnson, www.researchdemystified.org

5. Policy Debate • What does this data mean in terms of Congressional proposals to require everyone purchase health insurance? • What would you conclude based on this research result? • What do you want Congress to do based on these numbers? • What do you want Congress to do based on your beliefs? Dr. G. Johnson, www.researchdemystified.org

6. Paradox • Research: the search for rationality But: “We must be aware that our knowledge and experiences are finite, and always imperfect.” Hitoshi Kume Dr. G. Johnson, www.researchdemystified.org

7. Assessing Credibility Requires • Basic knowledge of research methods • Maintaining a research perspective • Knowing how data was collected is necessary to help assess whether the results are believable • Critical thinking skills • A hint of detachment and skepticism • Being aware of our beliefs yet setting them aside to see things clearly Dr. G. Johnson, www.researchdemystified.org

8. The Tough Questions • Is it true? • How do you know? • What is the logic of the argument? • What is the evidence? Remember: Much of what is presented as fact melts away under scrutiny. Beware of premature certainty. Dr. G. Johnson, www.researchdemystified.org

9. Critical Thinking • Ask Yourself: • “How do I know that the things I believe are true?” • “What would it take to convince me otherwise?” • The greatest challenge comes when looking at research that reflects what we already believe • Critical thinking challenges our most cherished beliefs • To see clearly, we must be aware of our beliefs while also remaining detached from them Dr. G. Johnson, www.researchdemystified.org

10. The Role of Research in the Public Sector • Guide policy • Target programs • Find out what works and what doesn’t • Convince funders • Provide feedback from customers • Help plan interventions • Advocate for change Dr. G. Johnson, www.researchdemystified.org

11. Research in the Public Sector • The Intersection of Data, Values and Beliefs and Politics • Does welfare help or hurt people? • Should oil companies be prevented from earning too much profit? • Is investment in primary prevention more effective in reducing crime than spending on law enforcement? Dr. G. Johnson, www.researchdemystified.org

12. Different Research Approaches • Qualitative: • Stories • Observations • No numbers • Quantitative: • What you can count. Dr. G. Johnson, www.researchdemystified.org

13. Qualitative Data: Stories • Anecdotes • Example: Michael Moore’s “Sicko” Example: Barbara Erhen “Nickled and Dimed” • Paints a picture, makes it feel real • Limited in scope: • Is this a small or large problem? Dr. G. Johnson, www.researchdemystified.org

14. Quantitative Data: Numbers • Percents, Counts, Correlations • Example: Study: U.S. Ranking for Preventable Deaths: • U.S. lowest of industrialized countries • But need to check how they defined and measured preventable deaths Dr. G. Johnson, www.researchdemystified.org

15. Quantitative Data: Numbers • Be aware that some things are not really knowable—so how did they come up with the numbers? • Number of illegal immigrants • Amount of money spent on illegal drugs • 10-year projection of annual U.S. budget deficits (even 1-year is not precise) • The true cost of mandatory health insurance • Percent of carbon from raising animals Dr. G. Johnson, www.researchdemystified.org

16. As An Advocate Which would be more effective? Voice your concerns about a problem? Or Show data documenting the problem? Dr. G. Johnson, www.researchdemystified.org

17. As a Citizen What do you base decisions on? • Data? • Emotional appeals? • Personal opinion? • Political demands? Do some work better than others in different situations? Dr. G. Johnson, www.researchdemystified.org

18. As a Decision Maker What do you base decisions on? • Data? • Emotional appeals? • Personal opinion? • Political demands? Do some work better than others in different situations? Dr. G. Johnson, www.researchdemystified.org

19. The Tough Question • Generally, we prefer data, facts, hard evidence • But the question is: • How good is the research? • Meaning: does the research yield accurate, reliable, valid, and unbiased information? Dr. G. Johnson, www.researchdemystified.org

20. What is Research? • A systematic search for answers to questions. • Search: to uncover, examine, find by exploration, to investigate, to inquire. • Research: "the systematic inquiry into a subject in order to discover or revise facts, theories Dr. G. Johnson, www.researchdemystified.org

21. What is Research? Research: Root of work means “to know.” Re---Search: conveys the idea of searching again, from a different perspective, using different approaches, looking at different data. Dr. G. Johnson, www.researchdemystified.org

22. What is Research? • Empirical: derived from experience or experiment; observation and experience. • Science: "a branch of knowledge or study dealing with a body of facts or truths systematically arranged and showing the operation of general laws; systematic knowledge" Dr. G. Johnson, www.researchdemystified.org

23. This Thing Called Science • Scientific Method • Curiosity • Systematic observation • Systematic experimentation • Theories and Hypotheses: • Used to Explain Relationships Dr. G. Johnson, www.researchdemystified.org

24. Different Goals Scientists: • More interested in creating knowledge • Not concerned about utility or application of the results • Practitioners: • More interested in application and problem solving • If the research isn’t useful, then so what? Dr. G. Johnson, www.researchdemystified.org

25. Research Process: Simplified 3-step • Plan • Do • Report Dr. G. Johnson, www.researchdemystified.org

26. Research Planning Process • Determine the issues • Decide on the research question(s) • Select measures • Identify the ideal design given the type of research question • Develop data collection methods and instruments Dr. G. Johnson, www.researchdemystified.org

27. Research Planning Process • Decide on sampling frame and strategy • Develop analysis plan • Articulate your entire research plan • Review research plan and test all data collection instruments • Prepare work plan with resource and time requirements Dr. G. Johnson, www.researchdemystified.org

28. Planning Matters • When people think about research, they tend to immediately focus on statistics • But no amount of statistical wizardry can save mistakes made in the planning process • Planning is harder and takes longer than most people expect Dr. G. Johnson, www.researchdemystified.org

29. Doing Research • Gathering the data • Preparing data for analysis • Analyzing and interpreting the data Dr. G. Johnson, www.researchdemystified.org

30. Reporting • Executive Summary • Reports • Charts and Tables • Oral Briefings Dr. G. Johnson, www.researchdemystified.org

31. Research Planning Process • Presented as linear but in practice, it is not • The word is iterative: meaning researchers make some initial assumptions about how to proceed and then go back and make changes to the plan as new information shows that their initial plans will not work • There is a lot of back and forth before it all clicks into place (and researchers live with the limitations of the situation) Dr. G. Johnson, www.researchdemystified.org

32. Ethical Research • Do No Harm • Protect participants from being harmed by the research • Guard the confidentiality of participants • Do not coerce people to participants: they must freely consent • No one should be denied benefits to which they are entitled because they refuse to participate in research project • Do not quickly conclude a program does not work just because there does not appear to be an impact Dr. G. Johnson, www.researchdemystified.org

33. Objective: Strive to See Clearly • Objectivity is a worthy goal even if not 100% attainable • Think in terms of being free of ideological or political blinders or desired outcomes • Be aware of biases • Researchers should state biases and build in quality controls to minimize bias • Quality Control: Have others with different perspectives review the research design, analysis and results. Dr. G. Johnson, www.researchdemystified.org

34. Principles of Good Research • Honest • Do not lie • Do not distort or “spin” the data • Fully disclose methods, definitions, assumptions, biases • Fully disclose limitations of the research and implications for making any conclusions Dr. G. Johnson, www.researchdemystified.org

35. Principles of Good Research • Accurate: Strive to be error free • Build quality control procedures into data collection, analysis, interpretation, and written product • Verify data entry and analysis. • Have someone check your numbers. Dr. G. Johnson, www.researchdemystified.org

36. Principles of Good Research • Technically Correct • Use appropriate designs, data collection methods, analysis, statistics and charts • The right data collection instruments. • The right statistics. Dr. G. Johnson, www.researchdemystified.org

37. Principles of Good Research • Reliable and Valid Measures • Use reliable and valid measures • Measure what matters using systematic approaches with fixed measurement rules • Ask: do they measures actually measure what they claim to measure? • Are they measuring reported crime but claiming to be measuring all crime? Dr. G. Johnson, www.researchdemystified.org

38. Research Flaws • Perfection is not the standard to assess the credibility of research results. • All research is flawed, so the trick is to distinguish minor flaws from major ones. • Skill: to assess the relative strengths and limitations of research to determine the credibility of the results. Dr. G. Johnson, www.researchdemystified.org

39. Research Flaws • It means working in the gray area. • If the research is the best given the circumstances, flawed though it may be, it should be considered. • However, decisions “to do something” based on that data should be made with caution, and implemented incrementally with built-in feedback. Dr. G. Johnson, www.researchdemystified.org

40. Seriously Flawed Research • While you might not want to make decisions based on seriously flawed research, it still may actually be correct, provide useful insights, or some guidance about how to do the next study. • It might provoke important conversations. • Remember: perfection is not a useful standard. Dr. G. Johnson, www.researchdemystified.org

41. Takeaway Lesson Sophisticated Users of Research Question Numbers! “Many a statistic is false on its face. It gets by only because the magic of numbers brings out a suspension of common sense." ---How to Lie With Statistics, Huff, p. 138. Dr. G. Johnson, www.researchdemystified.org

42. Ask Tough Questions • Does the researcher have axe to grind? • Particular political ideology? Desire to get more funding? • Who was included and excluded in the study? • Is the data really knowable? • Number of actual crimes is not the same as the number of reported crimes. Dr. G. Johnson, www.researchdemystified.org

43. Ask Tough Questions • Is reported behavior based on self-reports? • How many people used illegal drugs in the past week? • Does the research make a giant leap to conclusion? • Did a study with all male participants but generalizes to everyone? Dr. G. Johnson, www.researchdemystified.org

44. Creative Commons • This powerpoint is meant to be used and shared with attribution • Please provide feedback • If you make changes, please share freely and send me a copy of changes: • Johnsong62@gmail.com • Visit www.creativecommons.org for more information Dr. G. Johnson, www.researchdemystified.org