INTRODUCTION TO RESEARCH DESIGN

1. INTRODUCTION TO RESEARCH DESIGN Jonas Pontusson 04.11.2011

2. what is this about? • common challenges that we all face in developing and executing a research project: • research questions • theory building (note: focus on causal arguments) • testing/evaluating theories through empirical analysis • practical approach: discussion of specific examples (optional readings), mostly in the area of “comparative” but more broadly relevant • key theme: negotiating trade-offs • no prepared discussion of the following: • “methodology” in a technical sense • philosophy of science (epistemology) • strictly “practical” matters (grant applications, field research, etc.) • bottom line: what I think you ought to think about before embarking on your dissertation research (cumulated wisdom)

3. about me • US-trained “comparativist” • started out as qualitative work, turning quantitative • quantitative macro-comparative work: incorporating insights from “qualitative tradition” • recent efforts to engage “multi-level analyses:” combining individual-level survey data and macro-level variables

4. readings for general discussion • King, Keohane and Verba (KKV), Designing Social Inquiry (DSI) (1994) • bridging qualitative/quantitative divide: different data and methods, but common logic of inference • telling qualitative researcher how to improve by adopting standards of qualitative social science • APSR symposium: Laitin, Caporaso, Collier, Rogowski, Tarrow • Brady, Collier and Seawright, eds., Rethinking Social Inquiry (2004)

5. KKV basic vision of how “science” work: • elaborate a (casual) theory • generate empirical implications (the more the merrier) • test the hypotheses advice to qualitative researchers: • specify clear what you are trying to explain (DV)! • make sure that your measures are valid! • maximize “leverage,” minimize “bias”! • maximize variation on X (N), never select on the DV! • estimate uncertainty!

6. today’s agenda 1. research questions: where do we get them from? why are some more interesting than others? 2. link between theory and research 3. analytical leverage (research design) 4. causal mechanisms 5. combining different research designs (“mixed methods”)

7. warning • 48 slides to follow • I conceived this presentation before I saw your projects: much of it is not directly relevant…

8. Topic #1:types of research questions • open-ended questions: why Y? what are effects of X? not so interesting… • testing (reasonably) well-specified theories: is the theory “true”? potentially interesting… • puzzles usually interesting, at least a stepping stone…

9. theory-testing • interest depends on (a) status of theory and (b) empirical results • interesting to confirm a controversial (or counter-intuitive) theory • interesting to invalidate conventional wisdom • not so interesting to confirm conventional wisdom • testing multiple (competing) theories is almost always interesting! • critical importance of engaging with alternative explanations: who are you arguing against? why are they wrong? • articulation of your own theory • empirics: case selection, control variables

10. puzzles • theory-based puzzles: theory X (“conventional wisdom”) would lead us to expect Y (outcome), but this is not what we observe… why? note: single observation can serve this purpose if the theory is sufficiently well-specified (cf. Rogowski) • “real-world” puzzles: why do observe Y in case A, but not case B? note: such puzzles are only interesting if the cases are similar, i.e., share (values on) many Xs (MSSD)

11. framing/marketing research questions • broader (theoretical) implications of narrow research questions • turning topics into research questions of broad appeal... • key question: what is the intellectual community that you are seeking to join? (political science?) • in any case: reading broadly is crucial to effective framing!

12. example of puzzle-driven research project Evan Lieberman, Race and Regionalism in the Politics of Taxation (2003) Research question: Narrow version: Why has South Africa relied more heavily on income taxation than Brazil (to finance similar levels of public spending)? Broad version: What determines reliance on income taxation (relative to other revenue sources)? Point of departure: level of economic development is not an adequate explanation…

15. Lieberman’s historical-institutionalist account • definition of “national political community” at the moment of founding as key => path dependency • Brazil: federal race-inclusionary state undermines upper-class willingness to pay taxes • S.A.: centralized race-exclusionary state promotes upper-class willingness to pay taxes

16. does the argument travel? key principle (KKV): an inductive argument cannot be tested with cases that were used to generate the argument final chapter of Lieberman’s book: cross-section regression analysis of 71 countries (cases) DV=income+property taxation in % of GDP (1990-94)

17. regression results

18. Lieberman’s process 1. scatterplot to identify exceptions to conventional explanation 2. generate new explanation by probing the experience of exceptional cases 3. test the new theory against data for many countries

19. Topic #2:types of explanation and their implications for research design types of explanations: • frequentist vs. system-level explanations • partialist vs. holistic explanationist explanations • determinist vs. probabilistic explanations 2 types of research design (for now) • MSSD vs MDSD (Przeworski and Teune, The Logic of Comparative Social Inquiry (1970); cf. John Stuart Mill, A System of Logic (1843)

20. MSSD vs. MDSD MSSD (Mill “method of difference”/matched-case comparisons) • cases selected on DV (different values/outcomes) • inductive approach: looking for variables (Xs) that can explain the outcome • the more similar the cases, the more confident we are that (a particular) X is the cause of Y MDSD (Mill “method of agreement”/akin to regression) • cases selected on IV (different values) • deductive approach: prior expectation that X influences Y • the more different the cases, the more confident we are that X is the cause of Y

21. Przeworski and Teune:MDSD better than MSSD multi-level questions: why are French workers less likely to vote Left than Swedish workers? • frequentist explanation: there is some individual-level variable (e.g., religiosity) that takes on different values in the two countries (more religious workers in France) • system-level explanation: the relationship between class and voting (also religion and voting) is different in the two countries MSSD immediately jumps to system-level; working in the MDSD mode, we can and should add systemic factors as necessary

22. Przeworski and Teune:substitute variables for proper names! macro level: • continuous variables (e.g., proportionality) • typological variables (e.g., PR vs SMD)

23. partialist vs. holistic explanations • “holistic” approach (MSSD): complete explanation of Y e.g.: Immergut on health-care systems • “partialist” explanation (MDSD): goal is to establish that a causal relationship exists, not to explain (all of) Y e.g., Rogowski, Commerce and Coalitions (1989)

24. deterministic vs. probabilistic explanations • arguments for maximizing N and against selecting on the DV are premised on probabilistic theories and/or theories about marginal effects • deterministic theories (events as outcomes?): (1) X is a necessary cause (condition) of Y => only cases where Y occurs are relevant (2) X is a sufficient cause (condition) of Y => only cases where X occurs are relevant

25. Geddes debate Geddes, “How the Cases You Choose Affect the Answers that You Get, Political Analysis (1991) critique of literature that argues (based on the experience of EA and LA NICs) that political repression promotes econ

26. (1) “indeterminate research design:” no inferences can be drawn

27. (2) truncated sample, East Asia

28. (3) full sample: no relationship

29. retorts 1). the theory is about necessary conditions: only upper-left space is relevant (Jordan?) 2). scope restriction: the theory is only meant to apply to EA (and LA) => Przeworki and Teune: develop a theory that transforms “EA” into a variable (or several variables)! 3). the theory is not about growth per se, but a particular kind of growth: capital-intensive industrialization (Jordan isn’t relevant) 4). there is a relationship, but it is not linear! some but not too much repression promotes industrialization

30. bottom line • what is the scope of the theory you are testing? • what kind of causal claims does it entail?

31. Topic #3:analytical leverage • experimental and quasi-experimental designs • individual variation (political behavior) • variation across “macro-social units” (nations, regions/cantons) • variation across policy arenas • temporal variation (sequencing) prominent hybrids (in comparative): • pooled cross-section time-series analysis • two-level pooling of micro and macro variables

32. an example of qualitative case comparison Immergut, “The Rules of the Game” in Steinmo, Thelen and Longstreth, eds., Structuring Politics (1992) what is to be explained? health systems ca. 1975: • Sweden: de facto national health service (i.e., socialized medicine), doctors as government employees, uniform fees • Switzerland: private system, doctors free to set fees, role of government limited to subsidies to low-income people • France as “half-way house:” public insurance and private provision, regulation of fees => outcomes of political struggles 1945-65

33. Sweden vs. Switzerland • federal vs. centralized • dominant party vs. permanent coalition • overrepresentation of rural areas (CH) • referenda as means to veto federal legislation (CH) but the cases are different on many other dimensions… how similar is similar enough?

34. key leverage: temporal variation in the French case Fourth Republic: • weak executive • party fragmentation • lack of party discipline • parliament as source of veto power for special interests Fifth Republic: • strong executive: directly elected president, decree powers • completion of healthcare reform in early 1960s

35. pooled CSTS analysis method of choice for quantitatively-inclined comparative political economists working on OECD countries • only 18-20 countries (due to data limitations, also scope of theories) • ambition to test conditional arguments • theorized interaction effects (contrast QCA)

36. debate:Comparative Social Research, v. 24 (2007) Shalev’s critique: • we fundamentally care about cross-national variation: adding cross-temporal variation does buy us more leverage • the determinants of cross-sectional and cross-temporal variation are different • the interaction effects involve multiple variables: something like QCA is better suited for this • don’t get rid of the proper names: they contain lots of useful information!

37. my principles • we ought to theorize about specific interaction effects, not simply infer interaction effects from the data • we ought to aspire to an explanatory framework that can explain change over time as well as cross-sectional variation

38. addressing causal complexity within an CSTS framework • are causal dynamics different within different clusters of countries? • varieties of capitalism (LMEs vs. CMEs) • welfare-state regimes (Esping-Andersen) • PR vs. majoritarian systems • do causal dynamics change over time? Kwon and Pontusson, Globalization, Labour Power and Partisan Politics Revisited,” Socio-Economic Review 8(2010):251-281.

41. endogeneity problems ways to address such problems: • instrumental variables • experiments • “quasi-experiments:” discontinuous regression designs

42. Topic #4: causal mechanisms correlation ≠ causation => compelling story (theory) => empirical evidence

43. Kwon-Pontusson theory (adaptation of Garrett): • partisan conflict increases with globalization when unions are strong • partisan conflict declines with globalization when unions are weak • 3-way interaction of government partisanship, globalization and union strength: impossible to estimate (multicollinearity) • second-best solutions…

46. more systematic approaches • individual-level analysis of preferences • process-tracing (qualitative analysis): what did actors think/say/do? • sequencing (history) as key to causal inference

47. example of individual-level analysis Iversen and Soskice, “An Asset Theory of Social Policy Preferences,” APSR 95(2001):874-893. theory about skill specificity and demand for social insurance: specific skills provide for well-paying jobs, but finding an equivalent job is difficult => demand for insurance individual-level analysis: skill specificity associated with support for social spending the analysis doesn’t prove that insurance motives are at work, but do we really need that? from a macro-comparative point of view, this analysis speaks to causal mechanisms…

49. Topic #5: combining different research designs (or modes of argumentation) • example 1: Lieberman (see above) • example 2: Ansell, “University Challenges,” World Politics 60(2008):189-230. 1. formal model as device to illustrate theory, generate specific hypotheses 2. quantitative analysis (pooled CSTS) 3. “stylized” case studies to illustrate (demonstrate?) causal mechanisms

50. Ansell higher education policy: enrollment levels and structure of funding • formal model: preferences of income groups => partisan preferences (tipping point) • quantitative analysis: pooled CSTS, OECD countries, 1980-97 • case studies Britain, Germany and Sweden: causal mechanisms note: new research question, but building on existing theoretical framework