Other Research Approaches

1. Other Research Approaches Research Method for Public Administrators Dr. Gail Johnson Dr. Johnson. www.researchdemystified.org

2. Research Complexity • Not all research approaches fit into the Xs and Os framework • Some approaches are really analytic techniques (e.g. cost-benefit analysis) • Some approaches are data collection tools (e.g. survey research) Dr. Johnson. www.researchdemystified.org

3. Key Points about Other Approaches • Each approach has unique characteristics and are useful in specific situations • Each has strengths and limitations • Each has fundamental requirements • Researchers often uses a mix of approaches Dr. Johnson. www.researchdemystified.org

4. Commonly Used Research Approaches • Secondary Data Analysis • Evaluation Synthesis • Content Analysis • Survey Research • Case Studies • Cost-Benefit Analysis Dr. Johnson. www.researchdemystified.org

5. Secondary Analysis of Data • In the planning phase, researchers should determine whether there is data they can use • It is more efficient to use data collected by others if it meets the research requirements and is of high quality. • For example: That National Longitudinal Survey of Youth collected data about drug use. Dr. Johnson. www.researchdemystified.org

6. Secondary Analysis of Data • Large research studies gather data that is specifically related to the research questions and often gather other data that might be useful. • The Framingham study focused on factors related to heart disease, tracking 5,000 people from 1983-2003. • Other researchers used that data to look at why people are happy. • Researchers Conclusion: happiness is contagious! Dr. Johnson. www.researchdemystified.org

7. Things to Consider • Does the database contain the data needed to answer your research question? • If the question is about drug use among high school seniors, then a study looking at drug use among adults will probably be inappropriate. Dr. Johnson. www.researchdemystified.org

8. Things to Consider • How did they collect the data and does it meet research standards? • How did they measure their key variables? • Did they use documented procedures for selecting people, developing data collection instruments, and pre-testing? Dr. Johnson. www.researchdemystified.org

9. Things to Consider • What quality control procedures were used to ensure the accuracy of the data in the database? • What is the error rate? This means the difference between the data collected and the data in the database. Dr. Johnson. www.researchdemystified.org

10. Things to Consider • How hard will it be to gain access to the data? • Will the researchers provide all the documentation, including the data dictionary that specifies every variable and how every value for ever variable is coded? • For example, are Men coded 1 and Women coded 2 or is it Women coded 1 and Men coded 2? Dr. Johnson. www.researchdemystified.org

11. Evaluation Synthesis • Useful when there are many studies that have been done on a specific topic • Nutrition programs for the poor to reduce infant mortality • Early learning programs for disadvantaged children • Impact of race on death penalty sentencing Dr. Johnson. www.researchdemystified.org

12. Evaluation Synthesis • An expanded version of using available data • Cheaper than original data collection • Creates a larger data base, increasing the potential for drawing conclusions • While individual studies often have limited ability to draw conclusions, if the findings of many small studies form a pattern, conclusions have greater credibility Dr. Johnson. www.researchdemystified.org

13. How to Do an Evaluation Synthesis • Collect all studies that appear to be relevant • Establish criteria to rate the quality of each of the studies • Only those meeting the criteria are included • Summarize the key findings and impact measures • Look for similarities, especially among the highest quality studies Dr. Johnson. www.researchdemystified.org

14. Content Analysis • Used to measure the amount and/or message of communication • It can be used to analyze content of speeches, newspaper stories, emails, greeting cards, television, movies, blogs, memos, diaries or journals • Subjects covered in MPA textbooks, amount of violence in Saturday morning cartoons, or images of political women portrayed in news journals Dr. Johnson. www.researchdemystified.org

15. Content Analysis • In the narrowest definition, it converts qualitative data into quantitative data: • It can count the number of stories about the Iraq war on major national network news shows to determine whether it changed over time • It can count the number of times presidential candidates spoke about specific policy issues Dr. Johnson. www.researchdemystified.org

16. Content Analysis • But it is often used to summarize qualitative data • What were the major themes that emerged from the focus groups with teenaged mothers on welfare? • Is there a discernable political bias in the editorials of the major newspapers (NY times, Washington post, Boston globe, LA times)? Dr. Johnson. www.researchdemystified.org

17. Content Analysis: Measurement Challenges • Conceptual and operational definitions • How would conceptually define “political bias”? • Do you want to look at support for Republican or Democrat candidates or liberal or conservative policy positions? Dr. Johnson. www.researchdemystified.org

18. Content Analysis: Measurement Challenges • How would you operationalize “liberal” and “conservative”—and are they they only political ideologies you want to include? • What about libertarians, for example? • What about fiscal conservatives who are liberal on social policy? Dr. Johnson. www.researchdemystified.org

19. Content Analysis:Measurement Challenges • Conceptual and operational definitions • How would you measure “media images” of women governors? • How would you operationalize “strong” or”weak”, or “masculine” or “feminine”? Dr. Johnson. www.researchdemystified.org

20. Content Analysis:Reliability Challenges • Coding Reliability: Will everyone code the same images in exactly the same way? • Requires clear and very detailed decision rules • Requires all team members are trained Dr. Johnson. www.researchdemystified.org

21. Assuring Reliability • Jargon term: Inter-rater reliability • Each team member will code a small sample of the same material • Their coding will be compared and if there are differences, the team will figure out why there were differences and fix the procedures. • Everyone should be coding the same material in the same way. Dr. Johnson. www.researchdemystified.org

22. Survey Research • A data collection approach (discussed later under data collection) is used when the researchers want to determine the views of many people • Typically, the exact same questions are asked with limited response options • Think of a multiple choice exams (will cover more in data collection) • The structure limits bias of interpretation and is easier to analyze Dr. Johnson. www.researchdemystified.org

23. Survey Research • Open-ended questions—like fill-in-the blank exams—are possible but: • Can be burdensome to complete if too many open-ended questions are asked • Hard to analyze • Bias is a threat. Dr. Johnson. www.researchdemystified.org

24. Examples of Survey Research • Opinion polls to a cross-sectional sample of people in your town • Employee attitude survey of everyone at your place of work • Customer survey on the back of a postcard to all who visit the Post Office during the month of February Dr. Johnson. www.researchdemystified.org

25. Survey Research • In-person • Can be costly to do many in-person interviews • But useful when people are not literate • For example: it might be the best way to find out about health care access in rural villages in poor countries Dr. Johnson. www.researchdemystified.org

26. Survey Research • Mail • Requires mail address and participants who can read • Cost of printing and postage • Requires follow-up to get people to complete them Dr. Johnson. www.researchdemystified.org

27. Survey Research • Phone. • Useful in locations where most people have phones. • Works well when the questions are simple (like who are you going to vote for in the next election) and can be done in less than 15 minutes. • Random digit dialing is used to get a random sample; Even those with unlisted phones are called. • Challenges of cell phones: pollsters are now including cell phones. • But you have to call many, many numbers before you find someone who is willing to take the survey. Dr. Johnson. www.researchdemystified.org

28. Survey Research • Email or Web-based • Limited to those with access to this technology • Requires email addresses • Web-based limited to those with access • Can work well in work setting or in associations where everyone has access • Web-based applications often create the database, saving time and money Dr. Johnson. www.researchdemystified.org

29. Focus Groups • Another data collection approach that is sometimes presented as a research design approach. (discussed in more detail under data collection) • Small groups (6-12 people), trained facilitator moderates the discussion, few broad open-ended questions, food, pay to participate. Dr. Johnson. www.researchdemystified.org

30. Case Studies • Useful in answering descriptive and normative questions • Used in describing “best practices” • Can focus on one or several people, groups, communities, organizations, processes, cities, etc. • When it is not possible to gather data from every person, community, program or organization—Case Studies provide a way to say, “for example.” Dr. Johnson. www.researchdemystified.org

31. Case Studies • Case studies are useful when the researchers want to gain an in-depth understanding of a situation, of how a complex process works, or lessons learned about implementing an innovative program • It helps to make a problem or situation concrete: can tell stories of the impact of a program on a few people’s lives • It can capture complexity and variation in a situation Dr. Johnson. www.researchdemystified.org

32. Choosing Case Studies • How many? • No fixed rules • It will depend on the situation and available resources • More is better to capture variation but case studies are always about taking one or a handful of examples from a much larger universe Dr. Johnson. www.researchdemystified.org

33. Choosing Case Studies • Basis for selection • Convenience: easy for researchers to access • Particular aspects: best, typical, worst, or mixed cases • Diversity: mix of locations, people, types of organizations or perspectives • Random: removing researcher bias from decision Dr. Johnson. www.researchdemystified.org

34. Choosing Case Studies • Researchers have to explain their rationale for whatever basis they use to select cases • Selection should have “Face validity”: the choices should make sense to the average, reasonable person Dr. Johnson. www.researchdemystified.org

35. How to do a Case Study • Case studies can employ a wide variety of methods to gather data and often use several in combination • Triangulation: using different methods to gather information from various perspectives increases the validity of the results • Can use available data, surveys, interviews, observations, analysis of files and records, content analysis of memos, cost-benefit analyses Dr. Johnson. www.researchdemystified.org

36. Case Studies: Are They Credible? • Case studies are criticized because they typically have a small scope. • This is true. • But that is also true of classic experiments. • You have to beyond “smallness”—and look at what they did and how they did it. Researchers should state the limitations of their work and conclusions should stay within the boundaries of their limitations. Dr. Johnson. www.researchdemystified.org

37. Case Studies: Do They Make Sense? • Case studies are criticized because they are seen as inherently qualitative. • Case studies may include quantitative data and may in fact be entirely quantitative. • While case studies may be qualitative, there is nothing wrong with doing qualitative research as long as conclusions do not go beyond the limitations of the design. Dr. Johnson. www.researchdemystified.org

38. Case Studies in Public Administration • Case studies make sense in public administration where it is difficult to do national studies because of insufficient resources (time, money, staff) • Visiting six IRS offices may be far more doable than visiting 100 Dr. Johnson. www.researchdemystified.org

39. Case Studies in Public Administration • Like any research, quality is built into the design and procedures used: • Relevant research questions • Clear and appropriate measures • Data collection strategies than minimize error and bias • Honest about limitations • Conclusions stay within those limitations Dr. Johnson. www.researchdemystified.org

40. Cost-Benefit Analysis • Answers Normative Questions: • Do the benefits justify the costs? • Do the costs exceed the benefits? • Assumes that the costs and the benefits can be meaningfully measured in $$Dollars$$ Dr. Johnson. www.researchdemystified.org

41. Cost-benefit Analysis • Is it cost beneficial to get a college degree? • What are the costs? • Tuition, books, room and board • Opportunity cost: • The money forgone from working fulltime in order to go to college • The earnings if that money had been invested Dr. Johnson. www.researchdemystified.org

42. Cost-benefit Analysis • Is it cost beneficial to get a college degree? • What are the benefits? • Difference in income between job with college degree and job without a college degree • Calculated over 20 years • Need to take inflation into account (remember to use constant dollars) Dr. Johnson. www.researchdemystified.org

43. Cost-benefit Analysis • Is it cost beneficial to get a college degree? • What are benefits that cannot be measured in dollars? • Career opportunities: some jobs are only available for those with college degrees, even if they do not pay that much • Increased self-esteem • Transformative experience: exposure to new ideas • Sustains a vibrant democracy Dr. Johnson. www.researchdemystified.org

44. Cost-benefit: Public Sector • Policy analysis: what are 5 different approaches to these issues • Reduce recidivism for drug users • Prevent unwed teenaged parenthood • Provide health care to all citizens • Prevent financial meltdowns like we saw in the banks and financial industries in Fall 2008 • What are assumptions are you making between cause and effect in each scenario? Dr. Johnson. www.researchdemystified.org

45. Cost-benefit: Public Sector • Policy analysis: • Pick one policy issue and consider how you might think about calculating the costs and benefits for each one. • Note: one policy choice is always to do nothing • Assuming there was that was the clear winner in the cost-benefits contest, are dollars the only consideration in the political environment? Dr. Johnson. www.researchdemystified.org

46. Cost-benefit: Public Sector • Values, beliefs, political ideology will all affect the debate • But each side are likely to present their best guess of costs and benefits. • And their estimates are likely to differ Dr. Johnson. www.researchdemystified.org

47. Other Costs • Social Costs and Externalities • Costs paid by the larger society • Take a look at the externalities of the material’s economy: http://www.storyofstuff.com/ Of course—you might want to question where some of the numbers come from but it does point out that we tend not to take externalities (meaning someone else pays a price) into account. Dr. Johnson. www.researchdemystified.org

48. Other Costs • Externalities: costs paid by someone other than those receiving the benefit • Those producing or using goods and services do not pay the full costs • E.g. Corporations pollute but the taxpayers pay for the cleanup • City A benefits from having manufacturing company but City B down river pay for the price: either higher health costs or costs for cleanup of toxics Dr. Johnson. www.researchdemystified.org

49. Other Benefits • Short term benefits are easier to measure than longer-term benefits • Direct benefits are easier to measure than indirect benefits • Direct: savings from health promotion program because less spent on health treatments • Indirect: savings from loss of productivity Dr. Johnson. www.researchdemystified.org

50. Other Benefits • Intangible benefits: avoidance of pain and grief of infant deaths • It is harder to attach a dollar value: if it is your child, the life of your child is priceless Dr. Johnson. www.researchdemystified.org