Approaches to Public Policymaking, Policy Analysis & Evaluation Research

1. Approaches to Public Policymaking, Policy Analysis & Evaluation Research Kathy Luckett University of Cape Town

2. Research Paradigms • Not rigid paradigmatic incommensurability • A map for navigating choppy waters around policy analysis and evaluation methodological debates • Post-positivist – experimental, pragmatic • Interpretative – constructionist, post-structuralist • Critical – PAR, empowerment evaluation,CSH • Critical Realism – theory-based evaluation

3. Post-positivist: Quasi-experimental • Popper, Campbell & Stanley (1963, 1966), Lasswell, Rossi, Lipsey & Freeman • Based on methods of the natural sciences, statistical measurement techniques • Social science can contribute to improved governance or management • Establish cause & effect relations bet policy/ programme objectives, inputs & interventions --------- outputs, outcomes & impact • Human performance can be objectively measured tv. Efficiency & effectiveness criteria • Evaluator: objective, neutral, rational • Problem: to secure internal validity of evaluation results • By 19702 disillusion set in, shift to quasi-experimental methods (pre- & post tests, times series, comparison group designs)

4. Critique: Quasi-experimental models • Uses a model developed for closed systems for open social systems • Adopts a flat ontology – reality = regularities bet observable, atomistic objects & events (ignores the non-observable) • Causality = regularities bet variables within stat. sig. samples • Claims about causation usually unclear and unconvincing • Can only provide descriptions (for a few variables on large populations), seldom explanation

5. Post-positivist: Pragmatic (dominant model) • Developed from ‘new public management’ , the ‘evaluative state’ – wants practical, workable results, useful for decision-making • Takes policy/ programme goals as focus of evaluation • Methods: a) open-ended case study (improvement) e.g. Pattonb) closed-system sets up criteria & performance indicators to measure performance & accountability of individuals & institutions e.g. programme accreditation • Sets up criteria and performance indicators to measure performance & accountability by institutions and individuals – a closed system

6. Critique: Pragmatic Models • Assumes stable external environment • Difficult to set measurable objectives, criteria & indicators for actual performance • Difficult to control variables in open soc systems - possibility of rival explanations, difficult to prove cause & effect • Ignores context & stakeholder meanings, ‘black box’ evaluation – seldom diagnostic • Can be prescriptive, leading to conformity

7. Interpretive: Constructionist • 1970s – 80s ‘linguistic turn’, 1980s policy sociology: meaning socially constructed, human action culturally and discursively mediated – rejection of naturalism • Vickers (1995) policymaking as communicative activity for institutional regulation, a process of norm-setting • Neo-institutional theory emphasises cognitive and normative factors in policy adoption and implementation • Guba & Lincoln (1989), 4th generation evaluation: focus on subjective stakeholder meanings, values & interests, evaluator as facilitator, truth as agreement, evaluation useful to insiders

8. Critique: of Constructionist Models • Over-socialised, emphases subjectivity at expense of structure, truth located in subjectivities of respondents • Ignores systemic asymmetries of power • Inability to rise above context • Relativist ontology

9. Interpretive: Post-structuralist • Foucault’s ‘geneaology’, Ball (1993), Gale (2000) • Discourse is socially constitutive, in dialectical relation to practice – sets up systems of power/ knowledge, norms & values • Policy as political artefact – as text & discourse – with unequal material & discursive effects that should be exposed • Policy has a normalising & regulatory role, sets up subject positions that constrain ways of speaking & thinking • Technologization of language for institutional ends • How do certain discourses become dominant? What discourses are at work when those who govern, govern? How do they become institutionalised & supported legislatively, professionally & financially?

10. Critique: Post-structuralist Models • Weak on method, selectivity of data, dominance of researcher as interpreter, tendency to jump from data to (preconceived) narrative • Quest to successfully link the micro and macro levels of analysis difficult to achieve • All of social life gets reduced to discourse,(materiality of the social world gets lost) • Knowledge reduced to conditions of its production and interests of its producers (epistemological relativity)

11. Critical: Emancipatory • Neo-Marxist insights, Frankfurt School (Habermas empancipatory interest) • Critical policy analysis (the ‘argumentative turn’) policy discourses construct social problems & policy solutions, policymaking a form of argument to persuade & manufacture consent • Challenge: how do discourses become institutionalised & reflected in institutional practices? • Ulrich (1994) Critical systems heuristics: policy to be normatively acceptable to those affected by it, value clarification – diff groups of stakeholders

12. Critical: Emancipatory • Developmental evaluation (Patton) • PAR • Empowerment evaluation (Fetterman 1996) • Transformative evaluation (Mertens 2005) • Development of evaluees, giving voice to the silenced, inclusion of marginalised groups affected by the results

13. Critique: Emancipatory Models • Utopian: the ‘better argument’ is produced through power not rational dialogue - all communication already penetrated by power • Why should the involved (the powerful) bother to take into account the views and concerns of the affected (the powerless)? • Cannot work under conditions of coercion requires a fully functioning public sphere • Needs to hold material conditions and structures as contexts for vlaues & interests • Post-structuralists: consensus is neither possible nor desirable

14. Critical Realist: Theory-based • Bhaskar (1978, 1998), Sayer (1992, 2000) • Reality is stratified – empirical (experiences), actual (events) & real (non-observable structures & causal powers) • Holds tog ontological realism + epistemological relativism • Both agency & structure have causal powers – attend to both (analytically separate) • Openess of the social world, plurality and contingency of causality • Key to successful intervention = change of social practice

15. Critical Realist: Theory-based • Pawson & Tilley (1997) Realist evaluation: what works, how, for whom and under what conditions? (builds in context & subjectivity) • Evaluator to make programme theory explicit & to check it out with stakeholders: C + M = Otests assumptions about causal relations & change • Tests goal realisation, but places in context of wider social explanation • Evaluation can be cumulative – middle range theories • Critique: Demanding to operationalise, time-consuming

16. Conclusion • Be aware of tradition & model you’re working in - & of other possibilities • Complex nature of policy analysis & evaluation justifies methodological pluralism • But don’t use methods opportunistically, select according to values, purpose of evaluation, stage of the policy/programme cycle & practical constraints • Think purpose (teleology), ontology, epistemology 1st – then methodology!