Research Methods for Public Administrators: Defining and Designing Your Study

1. What Is the Research Question? Research Methods for Public Administrators Dr. Gail Johnson Dr. G. Johnson, www.researchdemystified.org

2. Steps in the Research Process Planning 1. Determining Your Questions 2. Identifying Your Measures and Measurement Strategy 3. Selecting a Research Design 4. Developing Your Data Collection Strategy 5. Identifying Your Analysis Strategy 6. Reviewing and Testing Your Plan Dr. G. Johnson, www.researchdemystified.org

3. Questions • There are a million questions in the universe • It is a process to figure out what you want to know • It takes more time than you think and it can be frustrating • The first question is rarely the final question Dr. G. Johnson, www.researchdemystified.org

4. The funnel process • Begins with general issues, concerns • Do you want to evaluate a program or assess a policy? • Begin with questions: • Is it cost-effective to privatize a public function or service? • Are the police officers in your city are paid comparably to those in neighboring cities? • Are veterans injured in the Iraq war receiving necessary services and support? • How likely it is that people would vote for a woman president in 2012? Dr. G. Johnson, www.researchdemystified.org

5. More Possible Questions • What is the morale of your employees? • Why there is a high turnover rate? • What is the prevalence of illegal drug use in your town? • How citizens are likely to vote on a referendum? • Why people are not shopping in the downtown area? • What can be done to prevent teenaged out parenthood? • Does global warming exist? If so, what are the causes and consequences? • What do citizens think about the president’s foreign policy? Dr. G. Johnson, www.researchdemystified.org

6. Determine Your Question Ask: • Who wants to know? • Why do they want to know? • How will the results be used? • What’s most important? Dr. G. Johnson, www.researchdemystified.org

7. Determining the Question: Sources of Help • Program documents • Program officials • Program Beneficiaries • Experts • Other studies Dr. G. Johnson, www.researchdemystified.org

8. Do Not Reinvent the Wheel • Find out who has already done research on this topic, program or policy to help you: • Decide on the questions • Get ideas about possible measures and research approaches • Think about likely results • Identify experts who have been down this road Dr. G. Johnson, www.researchdemystified.org

9. Literature Review: Learn from Others • Google may be a starting point • But electronic databases of professional/academic journals and government resources in libraries are better • Once you find a few articles, look at their citations/bibliography to find other similar articles • Contact authors if you have questions Dr. G. Johnson, www.researchdemystified.org

10. Engaging Stakeholders • Who is requesting the study? • Who is likely to be interested in the study’s results? • Interest groups? • Funders? • Clients? • Professional associations? • Employees? • Elected officials? Dr. G. Johnson, www.researchdemystified.org

11. Engaging Stakeholders Two big issues • Diversity • Is everyone who needs to be at the table included? • Are all diverse interests, political perspectives, concerns and people included? • How many • Does everyone need to be there or can there be representatives of the different perspectives and positions? Dr. G. Johnson, www.researchdemystified.org

12. Engaging Stakeholders Maintain an Even Table: • Everyone should feel safe to speak their views • Manage dominant groups • Set clear agreements and expectations about behavior, goals, and process • Make sure there are no hidden agendas Dr. G. Johnson, www.researchdemystified.org

13. Mutual Respect and Good Communication Skills • Researchers • Need to be mindful of prior history • Need to speak in English • Recognize they do not have all the information needed • Managers/Stakeholders • Recognize the technical expertise that researchers bring • Invest time in understanding these concerns while also keeping the project moving forward Dr. G. Johnson, www.researchdemystified.org

14. Types of Questions 1. Descriptive Questions: “ What Is” 2. Normative questions: “What should be” 3.Impact or Cause/Effect Questions: “So What?” or “Why?” Dr. G. Johnson, www.researchdemystified.org

15. Descriptive Questions • Presents a picture of a condition • A snapshot of what is. • Think of it as journalist’s questions: who, what, where, when, how, and how much? Dr. G. Johnson, www.researchdemystified.org

16. Descriptive Questions • How many bike-friendly miles does the city have? • How many people use bikes? • How many miles do people bike ride very day? • What is the trend over time? Dr. G. Johnson, www.researchdemystified.org

17. Normative Questions Are we doing what we are supposed to be doing? Are we hitting out target? Looks at: what is and compares it to what should be Dr. G. Johnson, www.researchdemystified.org

18. Normative Questions Possible Targets • Our stated goals • Program should perform similarly over time (past performance as a bench mark) • Program should do as well as similar programs (comparison) • What the experts say is “good performance” or a good result (criterion). Dr. G. Johnson, www.researchdemystified.org

19. Normative Questions • The key is the criterion! • Did the oil companies make too much profit during 2002—2007? • What is the criterion for determining what is “too much” profit? Dr. G. Johnson, www.researchdemystified.org

20. Normative Questions • Normative does not mean some abstract or inherent judgment about good or bad • but it can be used to determine best or worst performance according to some criteria • For example: what is the best public high school in the U.S.? Dr. G. Johnson, www.researchdemystified.org

21. Determining the Best U.S. Public High School • The U.S. News & World Report named Thomas Jefferson High School for Science and Technology in Annandale, Virginia as the best high school in America in the 2007. • My son went to Jefferson and I admit that I am willing to believe that it is the best (my bias)—but really, I need to ask, “Based on what?” before accepting anyone’s ranking of “the best.” Dr. G. Johnson, www.researchdemystified.org

22. Three Criteria • Overall academic performance on standardized testing in each school compared to the state average • How well disadvantaged students performed compared to the state average, and college readiness. • College readiness as measured by the percent of 12th-grade students who had taken advanced placement courses and the percent of students who passed the advanced placement courses. Robert Morse, how we got from 18,790 public schools to the top 100,” U.S. News and World Report, November 29, 2007 Dr. G. Johnson, www.researchdemystified.org

23. Normative Questions • Takeaway lesson: • You always want to pay attention to the researchers’ criteria if they are asking a normative question: • What are the criteria? • How did they decide to choose that criteria (their rationale)? • Does it make sense to you? • Are the criteria fair, objective and generally accepted as reasonable? Dr. G. Johnson, www.researchdemystified.org

24. Tough Questions about Criteria • Is there some hidden agenda in selecting that particular criteria? • Are the criteria biased in such a way to give the research some pre-determined or desired result? Dr. G. Johnson, www.researchdemystified.org

25. Relationship, Impact or Cause-Effect Questions • What’s the effect of one thing on another? • What’s the relationship between things? • How do we determine the causal effect of: an independentvariable on a dependent variable. Dr. G. Johnson, www.researchdemystified.org

26. Examples • Are poor children more likely to die before age 6 than non-poor children? • What factors are associated with innovation public agencies? • What explains the differences between kids who do well in school and those who don’t? • Does living near a toxic waste site increase the incidence of cancer? Dr. G. Johnson, www.researchdemystified.org

27. Necessary Elements to Determine Causal Relationships 1. Logical theory 2. Time order 3. Co-variation 4. Elimination of rival explanations Remember: association does not equal causation Dr. G. Johnson, www.researchdemystified.org

28. Deciding What Came First • Did the person behind the desk knock the seated person over? • Did the person behind the desk stand and reach out to prevent the seated person from falling over? Dr. G. Johnson, www.researchdemystified.org

29. Cause--Effect Questions 1. Have a logical theory (good theory) which relates your variables. It must make sense. • Quality of the causal relationship depends upon how good your theory is. • The number of storks increases and the birth rates also increase but that does not mean storks cause births. Dr. G. Johnson, www.researchdemystified.org

30. Cause--Effect Questions 2. You must establish time -order. • The independent variable must occur before the dependent variable. Independent Dependent Cause Effect Dr. G. Johnson, www.researchdemystified.org

31. Cause --Effect Questions 3. Variables must co-vary. • As independent variable changes, so does the dependent variable. • This change is, ideally, in the direction you have predicted: • Direction of Relationships • Direct: variables change in the same direction • Inverse: variables change in opposite direction • Non-linear: changes are curvilinear: go up, down, and up again Dr. G. Johnson, www.researchdemystified.org

32. Cause-Effect Questions 4. You must eliminate all other possible explanations or rival hypotheses. • You have to show that no other variables exist which really explains the observed relationship. Dr. G. Johnson, www.researchdemystified.org

33. Impact Questions These are the hardest to answer! Design –our next topic--is important in helping researchers rule out rival explanations. Dr. G. Johnson, www.researchdemystified.org

34. Final Words About Research Questions • Determining the question is the starting point for the research planning process • As you go through the research planning process, the questions may change as new information comes to light • What matters most is to select the right questions for the situation • Remember: the first question is rarely the final question: work the process! Dr. G. Johnson, www.researchdemystified.org

35. Creative Commons • This powerpoint is meant to be used and shared with attribution • Please provide feedback • If you make changes, please share freely and send me a copy of changes: • Johnsong62@gmail.com • Visit www.creativecommons.org for more information Dr. G. Johnson, www.researchdemystified.org