570 likes | 588 Views
Other Research Approaches. Research Method for Public Administrators Dr. Gail Johnson. Research Complexity. Not all research approaches fit into the Xs and Os framework Some approaches are really analytic techniques (e.g. cost-benefit analysis)
E N D
Other Research Approaches Research Method for Public Administrators Dr. Gail Johnson Dr. Johnson. www.researchdemystified.org
Research Complexity • Not all research approaches fit into the Xs and Os framework • Some approaches are really analytic techniques (e.g. cost-benefit analysis) • Some approaches are data collection tools (e.g. survey research) Dr. Johnson. www.researchdemystified.org
Key Points about Other Approaches • Each approach has unique characteristics and are useful in specific situations • Each has strengths and limitations • Each has fundamental requirements • Researchers often uses a mix of approaches Dr. Johnson. www.researchdemystified.org
Commonly Used Research Approaches • Secondary Data Analysis • Evaluation Synthesis • Content Analysis • Survey Research • Case Studies • Cost-Benefit Analysis Dr. Johnson. www.researchdemystified.org
Secondary Analysis of Data • In the planning phase, researchers should determine whether there is data they can use • It is more efficient to use data collected by others if it meets the research requirements and is of high quality. • For example: That National Longitudinal Survey of Youth collected data about drug use. Dr. Johnson. www.researchdemystified.org
Secondary Analysis of Data • Large research studies gather data that is specifically related to the research questions and often gather other data that might be useful. • The Framingham study focused on factors related to heart disease, tracking 5,000 people from 1983-2003. • Other researchers used that data to look at why people are happy. • Researchers Conclusion: happiness is contagious! Dr. Johnson. www.researchdemystified.org
Things to Consider • Does the database contain the data needed to answer your research question? • If the question is about drug use among high school seniors, then a study looking at drug use among adults will probably be inappropriate. Dr. Johnson. www.researchdemystified.org
Things to Consider • How did they collect the data and does it meet research standards? • How did they measure their key variables? • Did they use documented procedures for selecting people, developing data collection instruments, and pre-testing? Dr. Johnson. www.researchdemystified.org
Things to Consider • What quality control procedures were used to ensure the accuracy of the data in the database? • What is the error rate? This means the difference between the data collected and the data in the database. Dr. Johnson. www.researchdemystified.org
Things to Consider • How hard will it be to gain access to the data? • Will the researchers provide all the documentation, including the data dictionary that specifies every variable and how every value for ever variable is coded? • For example, are Men coded 1 and Women coded 2 or is it Women coded 1 and Men coded 2? Dr. Johnson. www.researchdemystified.org
Evaluation Synthesis • Useful when there are many studies that have been done on a specific topic • Nutrition programs for the poor to reduce infant mortality • Early learning programs for disadvantaged children • Impact of race on death penalty sentencing Dr. Johnson. www.researchdemystified.org
Evaluation Synthesis • An expanded version of using available data • Cheaper than original data collection • Creates a larger data base, increasing the potential for drawing conclusions • While individual studies often have limited ability to draw conclusions, if the findings of many small studies form a pattern, conclusions have greater credibility Dr. Johnson. www.researchdemystified.org
How to Do an Evaluation Synthesis • Collect all studies that appear to be relevant • Establish criteria to rate the quality of each of the studies • Only those meeting the criteria are included • Summarize the key findings and impact measures • Look for similarities, especially among the highest quality studies Dr. Johnson. www.researchdemystified.org
Content Analysis • Used to measure the amount and/or message of communication • It can be used to analyze content of speeches, newspaper stories, emails, greeting cards, television, movies, blogs, memos, diaries or journals • Subjects covered in MPA textbooks, amount of violence in Saturday morning cartoons, or images of political women portrayed in news journals Dr. Johnson. www.researchdemystified.org
Content Analysis • In the narrowest definition, it converts qualitative data into quantitative data: • It can count the number of stories about the Iraq war on major national network news shows to determine whether it changed over time • It can count the number of times presidential candidates spoke about specific policy issues Dr. Johnson. www.researchdemystified.org
Content Analysis • But it is often used to summarize qualitative data • What were the major themes that emerged from the focus groups with teenaged mothers on welfare? • Is there a discernable political bias in the editorials of the major newspapers (NY times, Washington post, Boston globe, LA times)? Dr. Johnson. www.researchdemystified.org
Content Analysis: Measurement Challenges • Conceptual and operational definitions • How would conceptually define “political bias”? • Do you want to look at support for Republican or Democrat candidates or liberal or conservative policy positions? Dr. Johnson. www.researchdemystified.org
Content Analysis: Measurement Challenges • How would you operationalize “liberal” and “conservative”—and are they they only political ideologies you want to include? • What about libertarians, for example? • What about fiscal conservatives who are liberal on social policy? Dr. Johnson. www.researchdemystified.org
Content Analysis:Measurement Challenges • Conceptual and operational definitions • How would you measure “media images” of women governors? • How would you operationalize “strong” or”weak”, or “masculine” or “feminine”? Dr. Johnson. www.researchdemystified.org
Content Analysis:Reliability Challenges • Coding Reliability: Will everyone code the same images in exactly the same way? • Requires clear and very detailed decision rules • Requires all team members are trained Dr. Johnson. www.researchdemystified.org
Assuring Reliability • Jargon term: Inter-rater reliability • Each team member will code a small sample of the same material • Their coding will be compared and if there are differences, the team will figure out why there were differences and fix the procedures. • Everyone should be coding the same material in the same way. Dr. Johnson. www.researchdemystified.org
Survey Research • A data collection approach (discussed later under data collection) is used when the researchers want to determine the views of many people • Typically, the exact same questions are asked with limited response options • Think of a multiple choice exams (will cover more in data collection) • The structure limits bias of interpretation and is easier to analyze Dr. Johnson. www.researchdemystified.org
Survey Research • Open-ended questions—like fill-in-the blank exams—are possible but: • Can be burdensome to complete if too many open-ended questions are asked • Hard to analyze • Bias is a threat. Dr. Johnson. www.researchdemystified.org
Examples of Survey Research • Opinion polls to a cross-sectional sample of people in your town • Employee attitude survey of everyone at your place of work • Customer survey on the back of a postcard to all who visit the Post Office during the month of February Dr. Johnson. www.researchdemystified.org
Survey Research • In-person • Can be costly to do many in-person interviews • But useful when people are not literate • For example: it might be the best way to find out about health care access in rural villages in poor countries Dr. Johnson. www.researchdemystified.org
Survey Research • Mail • Requires mail address and participants who can read • Cost of printing and postage • Requires follow-up to get people to complete them Dr. Johnson. www.researchdemystified.org
Survey Research • Phone. • Useful in locations where most people have phones. • Works well when the questions are simple (like who are you going to vote for in the next election) and can be done in less than 15 minutes. • Random digit dialing is used to get a random sample; Even those with unlisted phones are called. • Challenges of cell phones: pollsters are now including cell phones. • But you have to call many, many numbers before you find someone who is willing to take the survey. Dr. Johnson. www.researchdemystified.org
Survey Research • Email or Web-based • Limited to those with access to this technology • Requires email addresses • Web-based limited to those with access • Can work well in work setting or in associations where everyone has access • Web-based applications often create the database, saving time and money Dr. Johnson. www.researchdemystified.org
Focus Groups • Another data collection approach that is sometimes presented as a research design approach. (discussed in more detail under data collection) • Small groups (6-12 people), trained facilitator moderates the discussion, few broad open-ended questions, food, pay to participate. Dr. Johnson. www.researchdemystified.org
Case Studies • Useful in answering descriptive and normative questions • Used in describing “best practices” • Can focus on one or several people, groups, communities, organizations, processes, cities, etc. • When it is not possible to gather data from every person, community, program or organization—Case Studies provide a way to say, “for example.” Dr. Johnson. www.researchdemystified.org
Case Studies • Case studies are useful when the researchers want to gain an in-depth understanding of a situation, of how a complex process works, or lessons learned about implementing an innovative program • It helps to make a problem or situation concrete: can tell stories of the impact of a program on a few people’s lives • It can capture complexity and variation in a situation Dr. Johnson. www.researchdemystified.org
Choosing Case Studies • How many? • No fixed rules • It will depend on the situation and available resources • More is better to capture variation but case studies are always about taking one or a handful of examples from a much larger universe Dr. Johnson. www.researchdemystified.org
Choosing Case Studies • Basis for selection • Convenience: easy for researchers to access • Particular aspects: best, typical, worst, or mixed cases • Diversity: mix of locations, people, types of organizations or perspectives • Random: removing researcher bias from decision Dr. Johnson. www.researchdemystified.org
Choosing Case Studies • Researchers have to explain their rationale for whatever basis they use to select cases • Selection should have “Face validity”: the choices should make sense to the average, reasonable person Dr. Johnson. www.researchdemystified.org
How to do a Case Study • Case studies can employ a wide variety of methods to gather data and often use several in combination • Triangulation: using different methods to gather information from various perspectives increases the validity of the results • Can use available data, surveys, interviews, observations, analysis of files and records, content analysis of memos, cost-benefit analyses Dr. Johnson. www.researchdemystified.org
Case Studies: Are They Credible? • Case studies are criticized because they typically have a small scope. • This is true. • But that is also true of classic experiments. • You have to beyond “smallness”—and look at what they did and how they did it. Researchers should state the limitations of their work and conclusions should stay within the boundaries of their limitations. Dr. Johnson. www.researchdemystified.org
Case Studies: Do They Make Sense? • Case studies are criticized because they are seen as inherently qualitative. • Case studies may include quantitative data and may in fact be entirely quantitative. • While case studies may be qualitative, there is nothing wrong with doing qualitative research as long as conclusions do not go beyond the limitations of the design. Dr. Johnson. www.researchdemystified.org
Case Studies in Public Administration • Case studies make sense in public administration where it is difficult to do national studies because of insufficient resources (time, money, staff) • Visiting six IRS offices may be far more doable than visiting 100 Dr. Johnson. www.researchdemystified.org
Case Studies in Public Administration • Like any research, quality is built into the design and procedures used: • Relevant research questions • Clear and appropriate measures • Data collection strategies than minimize error and bias • Honest about limitations • Conclusions stay within those limitations Dr. Johnson. www.researchdemystified.org
Cost-Benefit Analysis • Answers Normative Questions: • Do the benefits justify the costs? • Do the costs exceed the benefits? • Assumes that the costs and the benefits can be meaningfully measured in $$Dollars$$ Dr. Johnson. www.researchdemystified.org
Cost-benefit Analysis • Is it cost beneficial to get a college degree? • What are the costs? • Tuition, books, room and board • Opportunity cost: • The money forgone from working fulltime in order to go to college • The earnings if that money had been invested Dr. Johnson. www.researchdemystified.org
Cost-benefit Analysis • Is it cost beneficial to get a college degree? • What are the benefits? • Difference in income between job with college degree and job without a college degree • Calculated over 20 years • Need to take inflation into account (remember to use constant dollars) Dr. Johnson. www.researchdemystified.org
Cost-benefit Analysis • Is it cost beneficial to get a college degree? • What are benefits that cannot be measured in dollars? • Career opportunities: some jobs are only available for those with college degrees, even if they do not pay that much • Increased self-esteem • Transformative experience: exposure to new ideas • Sustains a vibrant democracy Dr. Johnson. www.researchdemystified.org
Cost-benefit: Public Sector • Policy analysis: what are 5 different approaches to these issues • Reduce recidivism for drug users • Prevent unwed teenaged parenthood • Provide health care to all citizens • Prevent financial meltdowns like we saw in the banks and financial industries in Fall 2008 • What are assumptions are you making between cause and effect in each scenario? Dr. Johnson. www.researchdemystified.org
Cost-benefit: Public Sector • Policy analysis: • Pick one policy issue and consider how you might think about calculating the costs and benefits for each one. • Note: one policy choice is always to do nothing • Assuming there was that was the clear winner in the cost-benefits contest, are dollars the only consideration in the political environment? Dr. Johnson. www.researchdemystified.org
Cost-benefit: Public Sector • Values, beliefs, political ideology will all affect the debate • But each side are likely to present their best guess of costs and benefits. • And their estimates are likely to differ Dr. Johnson. www.researchdemystified.org
Other Costs • Social Costs and Externalities • Costs paid by the larger society • Take a look at the externalities of the material’s economy: http://www.storyofstuff.com/ Of course—you might want to question where some of the numbers come from but it does point out that we tend not to take externalities (meaning someone else pays a price) into account. Dr. Johnson. www.researchdemystified.org
Other Costs • Externalities: costs paid by someone other than those receiving the benefit • Those producing or using goods and services do not pay the full costs • E.g. Corporations pollute but the taxpayers pay for the cleanup • City A benefits from having manufacturing company but City B down river pay for the price: either higher health costs or costs for cleanup of toxics Dr. Johnson. www.researchdemystified.org
Other Benefits • Short term benefits are easier to measure than longer-term benefits • Direct benefits are easier to measure than indirect benefits • Direct: savings from health promotion program because less spent on health treatments • Indirect: savings from loss of productivity Dr. Johnson. www.researchdemystified.org
Other Benefits • Intangible benefits: avoidance of pain and grief of infant deaths • It is harder to attach a dollar value: if it is your child, the life of your child is priceless Dr. Johnson. www.researchdemystified.org