1 / 16

Approaches to Public Policymaking, Policy Analysis & Evaluation Research

Approaches to Public Policymaking, Policy Analysis & Evaluation Research. Kathy Luckett University of Cape Town. Research Paradigms. Not rigid paradigmatic incommensurability A map for navigating choppy waters around policy analysis and evaluation methodological debates

hercules
Download Presentation

Approaches to Public Policymaking, Policy Analysis & Evaluation Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Approaches to Public Policymaking, Policy Analysis & Evaluation Research Kathy Luckett University of Cape Town

  2. Research Paradigms • Not rigid paradigmatic incommensurability • A map for navigating choppy waters around policy analysis and evaluation methodological debates • Post-positivist – experimental, pragmatic • Interpretative – constructionist, post-structuralist • Critical – PAR, empowerment evaluation,CSH • Critical Realism – theory-based evaluation

  3. Post-positivist: Quasi-experimental • Popper, Campbell & Stanley (1963, 1966), Lasswell, Rossi, Lipsey & Freeman • Based on methods of the natural sciences, statistical measurement techniques • Social science can contribute to improved governance or management • Establish cause & effect relations bet policy/ programme objectives, inputs & interventions --------- outputs, outcomes & impact • Human performance can be objectively measured tv. Efficiency & effectiveness criteria • Evaluator: objective, neutral, rational • Problem: to secure internal validity of evaluation results • By 19702 disillusion set in, shift to quasi-experimental methods (pre- & post tests, times series, comparison group designs)

  4. Critique: Quasi-experimental models • Uses a model developed for closed systems for open social systems • Adopts a flat ontology – reality = regularities bet observable, atomistic objects & events (ignores the non-observable) • Causality = regularities bet variables within stat. sig. samples • Claims about causation usually unclear and unconvincing • Can only provide descriptions (for a few variables on large populations), seldom explanation

  5. Post-positivist: Pragmatic (dominant model) • Developed from ‘new public management’ , the ‘evaluative state’ – wants practical, workable results, useful for decision-making • Takes policy/ programme goals as focus of evaluation • Methods: a) open-ended case study (improvement) e.g. Pattonb) closed-system sets up criteria & performance indicators to measure performance & accountability of individuals & institutions e.g. programme accreditation • Sets up criteria and performance indicators to measure performance & accountability by institutions and individuals – a closed system

  6. Critique: Pragmatic Models • Assumes stable external environment • Difficult to set measurable objectives, criteria & indicators for actual performance • Difficult to control variables in open soc systems - possibility of rival explanations, difficult to prove cause & effect • Ignores context & stakeholder meanings, ‘black box’ evaluation – seldom diagnostic • Can be prescriptive, leading to conformity

  7. Interpretive: Constructionist • 1970s – 80s ‘linguistic turn’, 1980s policy sociology: meaning socially constructed, human action culturally and discursively mediated – rejection of naturalism • Vickers (1995) policymaking as communicative activity for institutional regulation, a process of norm-setting • Neo-institutional theory emphasises cognitive and normative factors in policy adoption and implementation • Guba & Lincoln (1989), 4th generation evaluation: focus on subjective stakeholder meanings, values & interests, evaluator as facilitator, truth as agreement, evaluation useful to insiders

  8. Critique: of Constructionist Models • Over-socialised, emphases subjectivity at expense of structure, truth located in subjectivities of respondents • Ignores systemic asymmetries of power • Inability to rise above context • Relativist ontology

  9. Interpretive: Post-structuralist • Foucault’s ‘geneaology’, Ball (1993), Gale (2000) • Discourse is socially constitutive, in dialectical relation to practice – sets up systems of power/ knowledge, norms & values • Policy as political artefact – as text & discourse – with unequal material & discursive effects that should be exposed • Policy has a normalising & regulatory role, sets up subject positions that constrain ways of speaking & thinking • Technologization of language for institutional ends • How do certain discourses become dominant? What discourses are at work when those who govern, govern? How do they become institutionalised & supported legislatively, professionally & financially?

  10. Critique: Post-structuralist Models • Weak on method, selectivity of data, dominance of researcher as interpreter, tendency to jump from data to (preconceived) narrative • Quest to successfully link the micro and macro levels of analysis difficult to achieve • All of social life gets reduced to discourse,(materiality of the social world gets lost) • Knowledge reduced to conditions of its production and interests of its producers (epistemological relativity)

  11. Critical: Emancipatory • Neo-Marxist insights, Frankfurt School (Habermas empancipatory interest) • Critical policy analysis (the ‘argumentative turn’) policy discourses construct social problems & policy solutions, policymaking a form of argument to persuade & manufacture consent • Challenge: how do discourses become institutionalised & reflected in institutional practices? • Ulrich (1994) Critical systems heuristics: policy to be normatively acceptable to those affected by it, value clarification – diff groups of stakeholders

  12. Critical: Emancipatory • Developmental evaluation (Patton) • PAR • Empowerment evaluation (Fetterman 1996) • Transformative evaluation (Mertens 2005) • Development of evaluees, giving voice to the silenced, inclusion of marginalised groups affected by the results

  13. Critique: Emancipatory Models • Utopian: the ‘better argument’ is produced through power not rational dialogue - all communication already penetrated by power • Why should the involved (the powerful) bother to take into account the views and concerns of the affected (the powerless)? • Cannot work under conditions of coercion requires a fully functioning public sphere • Needs to hold material conditions and structures as contexts for vlaues & interests • Post-structuralists: consensus is neither possible nor desirable

  14. Critical Realist: Theory-based • Bhaskar (1978, 1998), Sayer (1992, 2000) • Reality is stratified – empirical (experiences), actual (events) & real (non-observable structures & causal powers) • Holds tog ontological realism + epistemological relativism • Both agency & structure have causal powers – attend to both (analytically separate) • Openess of the social world, plurality and contingency of causality • Key to successful intervention = change of social practice

  15. Critical Realist: Theory-based • Pawson & Tilley (1997) Realist evaluation: what works, how, for whom and under what conditions? (builds in context & subjectivity) • Evaluator to make programme theory explicit & to check it out with stakeholders: C + M = Otests assumptions about causal relations & change • Tests goal realisation, but places in context of wider social explanation • Evaluation can be cumulative – middle range theories • Critique: Demanding to operationalise, time-consuming

  16. Conclusion • Be aware of tradition & model you’re working in - & of other possibilities • Complex nature of policy analysis & evaluation justifies methodological pluralism • But don’t use methods opportunistically, select according to values, purpose of evaluation, stage of the policy/programme cycle & practical constraints • Think purpose (teleology), ontology, epistemology 1st – then methodology!

More Related