How to apply PSIA? 10 Steps and some lessons learned

How to apply PSIA? 10 Steps and some lessons learned -------------------------------------------- Renate Kirsch Nairobi, December 2006

A 10 Step approach to PSIA 1. Selecting the Reform 2. Identifying stakeholders 3. Understanding transmission channels 4. Assessing institutions 5. Gathering data and information 6. Analyzing impacts 7. Enhancing design and compensatory schemes 8. Assessing risks 9. Establishing monitoring and evaluation systems 10. Fostering policy debate and feedback into policy choice

1. Selecting the reform and mapping out research questions • Criteria for selection of reform • Expected size and direction of impacts • Prominence of issue in the government’s policy agenda • Timing and urgency of policy or reform • Level of national debate surrounding the reform • Formulating the key questions • Identify key problems/constraints that policy will address • Make development objectives explicit • Formulate causal hypotheses linking objectives to actions to likely short-term and long-term impacts • Define the alternative (other option, status quo)

Operational lesson 1: identify reforms • Need for PSIA should emerge from PRS • Identifying reforms for PSIA should be part of national PRS process (no duplication) • In practice, work in progress. Selection should strengthen broader process, not undermine/duplicate it • Selectivity/prioritization essential • Costly and time consuming • PSIA most meaningful and effective when applied to specific reforms

2. Identifying stakeholders • Stakeholders affected by policy reform positively and negatively • Differentiated by ethnic, religious, age, spatial, livelihood, or other criteria • Stakeholders affecting the reform • Institutional stakeholders • Powerful interest groups within the public sector, private sector, and civil society • Focus on • key characteristics • interests in relation to the policy • importance to the reform, influence on the process

Analytical lesson 2: different groups • Traditionally, distributional impacts measured on income/consumption groups • Useful to understand overall effectiveness and comparing aggregate impacts of alternatives • But groups are artificial constructs and do not allow analysis of behavioral responses • Need also focus on spatial, social, occupational groups that allow to understand behaviors  Operational dimensions

3. Understanding transmission channels Impacts transmitted through multiple channels: • Employment (Guyana sugar) • Prices – production, consumption, wages (utility prices) • Access to goods and services (credit, basic services) • Assets – physical, natural, financial, human, social (land,education, health) • Transfers and Taxes (tariffs, subsidies, import tax, VAT) • Authority (power relations, legal regulations, institutional capacity, political economy)

Analytical lesson 4: Multiple channels • Most reforms have multiple transmission channels • Impacts might change direction/size when considering them

4. Assessing Institutions • Institutions mediate the effect of policy changes on the welfare of people • Examine relevant social and market institutions • Institutions may themselves be the objective of policy reforms • Analyze changes in incentives and rules • Policy changes depend on organizations for their implementation • Incentives, performance and capacity are key • Transaction costs affect reform outcomes • Markets, legal systems, public organizations

5. Gathering data and information • Map out desirable data and information • Take stock of existing data and analysis • Adapt PSIA to data limitations: • Adapt analytical approach • Collect further data (multiple types) • Postpone the reform • Build data basis and capacity for future poverty and social impact analysis

6. Analyzing impacts Expected direction & magnitude of impact • Describe nature and size of principal impacts • Income and non-income impacts • Long-term and short-term impacts • Direct and indirect impacts • State underlying assumptions regarding • Intended benefits • Organizational capacity and institutional performance • Stakeholder behavior, including behaviors of affected persons, investors and regulators

Analytical lesson 1: negative and positive impacts • Central concern is the poor • But, PSIA is not only about mitigation measures • Hence, needs to focus on all impacts, both positive and negative on all groups • Analysis of support and opposition to reform • Allows for influence on design of the reform, not only on mitigation • Political economy critical

Analytical lesson 3: short/long term • Reforms have short and long term impacts • Often linked to direct versus indirect impacts • A same group could have positive net impacts in short term and negative ones in longer term • Assessing short term impacts is relatively easy • Longer term impacts require more complex and challenging analysis • Assess importance of short and long term, direct and indirect impacts Analyze all relevant ones to define net impacts

7. Enhancing design & compensatory schemes In light of analysis of impacts: • Consider an alternate design: • Alternative design • Different pace and sequence • Triggers to invoke additional risk management measures, reform modifications or an exit strategy • Consider direct compensation measures • Consider delay or suspension of the reform

8. Assessing risks • Types of risk: • Institutional risks (reform complexity exceeds institutional capacity, vested interests in agency) • Political economy risks (interest groups undermine reform or capture benefits) • Exogenous risks (conflict, financial crisis, terms of trade shocks, natural disaster) • Country risks (elections or political instability, ethnic conflict, post-conflict environment) • Assess likelihood of occurrence and importance to the policy

9. Monitoring & Evaluating Impacts • M&E allows to: • Validate policy analysis • Inform policy adjustment during implementation • Promote ownership of reforms (participatory monitoring) • Promote accountability • M&E should: • Indicators defined before the reform is implemented: • tied to transmission channels and assumptions • correlated with reform • that can be measured in time to suggest improvements • Build on existing systems to develop national monitoring system and capacity

Operational lesson 5: Monitoring • PSIA often ex-ante and based on assumptions • Assumptions, actual impacts must be monitored during implementation to allow corrections if needed • PSIA indicators should be integrated in country systems to ensure continued improvement • If PSIA elements too specific for country systems, ensure they will be monitored after end of core PSIA: • NGOs • Development partners on the ground • Research institutes and universities…

10. Fostering policy debate and feedback into policy choice • PSIA draws on public discussions: • When identifying reform for analysis • When analyzing stakeholders, • When validating technical impact analysis, • When leveraging social accountability. • PSIA should inform policy discussions and consideration of alternatives • PSIA needs an institutional home to incorporate results into the policy process

Operational lesson 2: Analysis design • Analysis typically includes participatory elements (e.g. stakeholder analysis) • Does not mean analysis is designed in participatory manner • One doesn’t analyze by consensus. Analysis a scientific process, based on professional norms and standards • But methodology must be transparent, in public domain for informed decision-making process

Operational lesson 3: Analytical process • No monopoly on who does the research: Government, university, research institute, NGO, private sector, development agency… • … as long as methodology transparent, rigorous • Rigor doesn’t mean ignoring stakeholders - their views are essential inputs • But analysis independent, not an expression of the views of a particular (vocal) group • Agencies responsible for reform must be part of analytical process, to be able to utilize the results

Operational lesson 4: Policy dialogue • PSIA contributes most when closely aligned with ongoing policy dialogue: • Research design based on options actually considered • Results relate to all stakeholders • Needs to be anchored in government policy cycle (national PMS, policy research group) • PSIA part of broader policy dialogue • Dissemination of PSIA results is key • Results produced early enough to influence dialogue • Policy processes w/o clear beginning or end • Discrete action part of series of inter-related actions • PSIA one element to inform broader process • PSIA needs to be absorbed by main actors in governments, civil society and within donor agencies • Ex ante, during, ex post

Analytical lesson 5: Choice of methods and team • PSIA can use various techniques and tools • Depends on question, data, resources, time • Complementarity, triangulation: • analytical techniques, • quantitative/qualitative data • Building teams: Skills for different aspects • Economists: prices/quantity, equilibrium • Social development specialists: stakeholders, institutions, risks • Sector specialists: policy issue and reform design  Multi-disciplinary work provides best rigor, but expensive and difficult

Mixed method approach • Combining Social and Economic Analysis • Bringing a social, economic and sectoral lens to the research questions • Combining quantitative and qualitative methods • Assess research questions with different methods and tools

Analytical focus vs type of data and analysis Qualitative analysis Quantitative analysis Social Economic

Qualitative Methods Identifies relationships and patterns can help to probe and affirm those relationships and explain contextual differences in the quality of those relationships inductively throws up interesting, often surprising and sometimes counterintuitive relationships and patterns, are applied to a specific locality, case or social setting are described as contextual.. Quantitative Methods produce data that can be aggregated and analyzed to describe and predict relationships able to ask “how much?” and establish how confident we can be in these “working hypotheses”. can be applied across the entire population or a section of the population, e.g. a region. They are referred to as non-contextual. Complementarity of methods

Combining tools from different disciplines • Use qualitative methods to understand context, relationships, patterns – informs the design of a survey questionnaire • Use quantitative methods to assess extent to which phenomena occur (generalization, representation) • Use qualitative methods to unpack issues which are hard to explain from survey results

Three ways to combine methods Joint conceptual framework • In parallel • In sequential • Iterative Basis for identifying results and developing recommendations

How to apply PSIA? 10 Steps and some lessons learned