Research at the Intersection of Politics and Administration

Research at the Intersection of Politics and Administration Research Methods for Public Administrators Dr. Gail Johnson Dr. G. Johnson, www.ResearchDemystified.org

Research Process Revisited • Key points • Planning is the key • No statistical magic can fix planning mistakes • What are the assumptions? • What those particular assumptions? • Do the assumptions change the results? Dr. G. Johnson, www.ResearchDemystified.org

Research Process Revisited • Key points • What are the boundaries of the study? • This defines the limitations: if they only survey people in NYC, they cannot make generalizations about people in Detroit • It is easy to conclude more than the research can actually support Dr. G. Johnson, www.ResearchDemystified.org

Why Research Matters • Challenges what we think we know • Explores what we do not know • Provides information that can engage people in discussion and debate to solve public problems • Ideally, provides information as a counterweight to emotion and rhetoric in a super-hot political environment Dr. G. Johnson, www.ResearchDemystified.org

Research Can Help Managers • When seeking information rather than relying on assumptions or beliefs • Customer, citizen or employee feedback • Program/financial auditing, efficiency audits • Monitoring pilot program • Measuring results • Program evaluation, effectiveness assessment • Budget, cost-benefit analysis, unit price • Best practices • Testing performance Dr. G. Johnson, www.ResearchDemystified.org

Managing Research Projects • Scope of work (SOW) • Details of the research project • Time • Budget • Requisite skills • Gantt chart • Visual way to track key tasks • Helps identify consequences of tasks that take more time than allocated Dr. G. Johnson, www.ResearchDemystified.org

Managing Research Projects • Build in quality • Engage key stakeholders early to sharpen the focus of the research and to develop buy-in for the reslts • Use experts to review: measures, design, assumptions, data collection, data analysis, review of draft report • Require an audit trail to verify accuracy of data, analysis and conclusions • Make sure the research methodology is appropriate for the situation and will support the conclusions Dr. G. Johnson, www.ResearchDemystified.org

Using Research Results • Requires the ability to: • Understand the results • Assess the credibility of those results based on the research methods used • Sophisticated users of research results need • An understanding of the research process • Critical thinking abilities • A willingness to consider information that is different from from our beliefs • The ability to change our minds in light of credible information Dr. G. Johnson, www.ResearchDemystified.org

First Take On Research Results • Our first filter: do the results make sense on a gut level? • Based on our experience • Based on our “common sense” • Based on our beliefs about how the world works • But we all have biases and blinders • Remember: we do not actually know as much as we believe we know • And that is true for experts as well Dr. G. Johnson, www.ResearchDemystified.org

Second Take On Research Results • Our second filter: a researcher’s perspective that seeks to see clearly • For any conclusion, ask: “Where’s the evidence?” • Opinions—no matter how persuasive—are not evidence • Judgment—no matter how expert—is not evidence • Begin the process of systematically assessing credibility Dr. G. Johnson, www.ResearchDemystified.org

Assessment Requires Critical Thinking Abilities • Necessity to distinguish fact from fiction, even if it has the appearance of science • To maintain a researcher's perspective—which is a detached critique of the methods used, assumptions made, and the conclusions drawn • Resist the temptation to be overly critical of results that challenge what we believe • Challenge ourselves to critically look at results that support what we believe Dr. G. Johnson, www.ResearchDemystified.org

Assessing Research Credibility • Do the researchers (or their sponsors) have a particular policy position they want to support? • Are the seeking to understand or to win an argument? • Is there bias? • Remember though—they may have an interest in a particular outcome but that does not necessarily mean their research is biased. • You still need evidence of bias. Dr. G. Johnson, www.ResearchDemystified.org

Assessing Credibility • Is the data knowable? • If not, how did they come up with it? • Are the talking about actual behaviors or self-reported behaviors? • Remember: actual crime is not the same as reported crime • What makes the data convincing? Dr. G. Johnson, www.ResearchDemystified.org

Assessing Credibility • What are their assumptions—especially when looking at research which tries to predict the future: • Budget forecasting always assumes something about the economy • “Rosy scenarios”-where the economy is growing-- will lead to different results than “economic collapse” scenarios Dr. G. Johnson, www.ResearchDemystified.org

Assessing Credibility: The Methods Checklist • Types of research questions • Be on the lookout for cause-effect questions, even if they do not sound like one initially • Measures: are they valid and reliable? • Design: experimental, quasi, or non-experimental? • If cause-effect question, what design did they use? • Data collection systematic? • Controls for error and bias? Dr. G. Johnson, www.ResearchDemystified.org

Assessing Credibility: The Methods Checklist • Sample: random or non-random? • Is size and response rate appropriate? • Appropriate analysis? • Did they do tests for statistical significance if they used a random sample? • Did they show measures of association if answering a relationship question. • Conclusions within the framework of the study? • Does the conclusions stay within the limitations of the research methodology? Dr. G. Johnson, www.ResearchDemystified.org

Process for Assessing Credibility • Break the research apart: • Read introduction and conclusion to get the point of the research • Read the methodology • Look at the data analysis • Make your own assessment: • Does the research answer the questions? • Does the analysis support the conclusions? Dr. G. Johnson, www.ResearchDemystified.org

Final Words About Assessing Credibility • Be mindful of your own biases when you review research results • Find a balance between trusting your own instincts and being blinded by your biases • Do not accept research results just because they agree with what you believe Dr. G. Johnson, www.ResearchDemystified.org

Final Words About Assessing Credibility • Do not dismiss a study because it is flawed. • All research is flawed. • The issue is whether it was the best that could have been done in the situation • Do not dismiss a study out of hand because you do not like the results. Dr. G. Johnson, www.ResearchDemystified.org

The Limitations of Science • Not everything is knowable • Not everything can be reduced to numbers • Bias can be embedded in how questions are framed • Social science is not set up to prove anything • It is the search for knowledge • Typical conclusion: more research is needed Dr. G. Johnson, www.ResearchDemystified.org

The Limitations of Science • Paradigm paralysis • Cling to earth-is-flat theories • Market Equilibrium Theory: assumes markets respond to new information • But research shows that markets change even when there is no financial news • Ecology of beliefs, expectations, perceptions, misperceptions, rumors, gossip, fears, speculation can drive market swings Dr. G. Johnson, www.ResearchDemystified.org

Mark Buchanan • “It’s these internal dynamics that make it possible for billions to evaporate from portfolios in a few short months just because people suddenly begin remembering that housing values do not always go up.” • “Equilibrium theory assumes incremental change but data suggests that markets can turn from ice to water in an instant. A new theory is needed.” NY Times: This Economy Does Not Computer, Op-Ed, October 1, 2008 Dr. G. Johnson, www.ResearchDemystified.org

Intersection of Science and Administration • Social science often is too weak to demonstrate cause-effect relationships • Programs can appear not to work because their impact can not be proven • Stokes cynicism about government efficacy • Pressure to take action but the information is far from certain Dr. G. Johnson, www.ResearchDemystified.org

Working at the Intersection • If we accept that the information is imperfect and that we cannot wait for perfect information, then incremental steps make sense • Pilot test programs best on research data • Build in feedback processes so more information is gathered as we go along • Continuous improvement • Incremental adjustments Dr. G. Johnson, www.ResearchDemystified.org

Working At The Intersection • Welcome new information even if it shows that we made an incorrect decision • Sometimes data from the past is a good predictor of the future • But sometimes it is like trying to drive forward while looking through our rear view mirror • Compassion: There is no way to know the decision was incorrect at the time it was made • It was necessary to try something so it generated more –and better—information Dr. G. Johnson, www.ResearchDemystified.org

Challenges Of Working At The Intersection • Controversial issues—vocal competing groups • Beliefs intertwine with policy • Ideology clashes with science • Public administrators are often at the center of this intersection and must find a middle way between the competition of ideas, beliefs and conflicting policy preferences of the various stakeholders Dr. G. Johnson, www.ResearchDemystified.org

The Joys Of Working At The Intersection • The research perspective—the discipline of detachment and honesty—will enable us to engage in deep conversations that might bring us closer to solutions that can solve very difficult problems • It is typically difficult, messy, and frustrating—but is also exciting to be doing work that matters • Finding that middle way is a way to support the democratic process. • It is one way to make a positive difference in the world Dr. G. Johnson, www.ResearchDemystified.org

Takeaway Lessons At The Intersection • Remember: the toughest problems wind up on the government’s doorstep • Simple solutions to complex problems do not exist • Single interventions are not likely to be sufficient for complex problems Dr. G. Johnson, www.ResearchDemystified.org

Final Words: Sophisticated Users of Research Results • Value the search for new ways of looking at the world, ourselves and others • Are open to the emerging situation • Are able to consider at least three alternatives or possible explanations or strategies to any given problem • Manage uncertainty by moving forward in incremental steps while continuing to gather information Dr. G. Johnson, www.ResearchDemystified.org

Final Words About Social Justice • Be mindful of the benefits as well as the limits of social science • An unintended consequence of demanding that that social service programs are effective but not tax loopholes, mortgage deductions, payments to corporations is an issue of social justice • Given the difficulty in “proving” things work using social science tools, we run the risk of wrongly assuming that programs don’t work just because we can’t measure an effect. Dr. G. Johnson, www.ResearchDemystified.org

Final Words About Assessing Research Results • Research methods can be far more complex than the basics presented here • I would not hesitate to call in experts if asked to make a decision based on a complicated study or a study using unfamiliar methods or statistics • I advise you to do the same Dr. G. Johnson, www.ResearchDemystified.org

Takeaway Lesson • We often know far less than we think we know. • It might not seem that way as different sides offer up statistics that appear to be rock solid Remember: "Many a statistic is false on its face. It gets by only because the magic of numbers brings out a suspension of common sense." How to lie with statistics, Darrel Huff, p. 138. Dr. G. Johnson, www.ResearchDemystified.org

Research Methods As Life Lessons • Approach each situation with calm confidence • Be open to new information that pushes you out of your comfort zone • Each step beyond your comfort zone expands it • Ask questions and don’t be intimidated by statistics, social science jargon or experts Dr. G. Johnson, www.ResearchDemystified.org

Takeaway Lessons • Social science is rarely conclusive and public administrators have to n to work in the middle, where things often depend on the context • More research is needed is not a full-employment program for researchers! • It merely reflects that fact that research is a process of discovery and a work in progress Dr. G. Johnson, www.ResearchDemystified.org

Creative Commons • This powerpoint is meant to be used and shared with attribution • Please provide feedback • If you make changes, please share freely and send me a copy of changes: • Johnsong62@gmail.com • Visit www.creativecommons.org for more information Dr. G. Johnson, www.ResearchDemystified.org

Research at the Intersection of Politics and Administration