1 / 40

Influence of Evaluation in Socio-Economic Development Plan Implementation in Vietnam

This analysis explores the influence of evaluation in the context of Socio-Economic Development Plan (SEDP) implementation in Vietnam, focusing on the case of Hai Lang district, Quang Tri province. The study identifies factors affecting evaluation influence and provides research and policy implications.

lbonilla
Download Presentation

Influence of Evaluation in Socio-Economic Development Plan Implementation in Vietnam

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An analysis of influence of evaluation in the context of Socio-Economic Development Plan (SEDP) implementation in Vietnam - The case of Hai lang district, Quang tri province Ha Minh Tri HCMC, 2 April 2015

  2. Outline (1/2) Introduction 1.1 The problemstatement 1.2 The research context 1.3 The purpose of the study 1.4 Overview of research steps Literature review 2.1 Literature review on evaluation use & influence 2.2 Conceptual framework 2.3 Research questions Case study setting (Hai Lang district)

  3. Outline (2/2) Research methodology 4.1 Research design 4.2 Research instruments & data collection 4.3 Hypotheses 4.4 Limitations Analysis and findings 5.1 Data analysis 5.2 Findings 5.3 Construct validity and reliability Research and policy implications 6.1 Contributions to theory 6.2 Implications for future research & policy

  4. 1. Introduction

  5. 1.1 The problem statement Importance of accountability and learning. Accountability = challenge for public sector management. Monitoring and evaluation (M&E) findings underused to serve SEDP targets and objectives. Empirical studies on evaluation influence: none in the public sector Vietnam.

  6. 1.2 Research context – Monitoring & Evaluation in Vietnam • Inspection system still limited to cope with state management requirements. • M&E system for Poverty Reduction Strategy Paper (PRSP) and SEDP at central level switching from traditional to evidence-based. This shift is at a slower speed at district and commune levels.

  7. 1.3 Purpose of the study - To identify and explain factors which may affect evaluation influence in the public sector of Hai Lang.

  8. 1.4 Overview of research steps • Developing a conceptual framework based on Theory of Evaluation Influence • Exploring research setting: M&E in Vietnam • Conducting literature review: to contextualise the research • Start with problem statement • Research design & methodology: Mixed Methods Explanatory Design: (1) QUAN methodology uses EFA, CFA and Structural Path Model and (2) QUAL study is used to explain QUAN results. • Developing specificresearch questions + hypotheses • Developing research instruments & collecting data • Empirical research & data analysis, presenting findings, analysis & discussion (with QUAL analysis enhancing QUAN findings) • Conclusions & implications

  9. 2. Literature review

  10. 2.1 Literature review – Main findings on evaluation use (1/6) Evaluation that was underused in the US in late 1960s stimulated research on evaluation use. Literature on evaluation use emerged since 1970s. Five most common types of evaluation use: instrumental, conceptual, persuasive, process and imposed use.

  11. 2.1 Factors affecting evaluation use (2/6)

  12. 2.1 Main findings on evaluation influence (3/6) • Evaluation influence appeared since 2000s as scope of evaluation use was limited. • Evaluation influence evolved with 2 remarkable models/theories: (1) Integrated model of influence (Kirkhart, 2000), and (2) Theory of Evaluation Influence (Mark and Henry, 2004)

  13. 2.1 Mark and Henry’s Theory of Evaluation Influence (2004)

  14. 2.1 Where is this dissertation positioned? (6/6) • This work • Evaluation influence in Vietnam public sector

  15. 2.2 Conceptual framework (1/2) Evaluation Inputs Evaluation Activities Evaluation Influence • Evaluation context • Lessons learned/ best practices in evaluation • Evaluation partners • Evaluation capacity • Involvement of programme holders in evaluation processes • Generation of evaluation reports • Dissemination of evaluation reports/findings • Evaluation influence at individual level • Decision/Policy setting • Evaluation plans & methods • Contingencies in environment • Facilitating factors: e.g. willingness to cooperate, etc. • Inhibiting factors: e.g. wrong time, time pressure, etc. • Source: Developed from Mark & Henry (2004).

  16. 2.2 Conceptual framework (2/2) • Theory-based evaluation approach was used to elaborate the conceptual framework to establish a theory of change of evaluation influence framework. • This theory of change was used as a guide to develop a research instrument.

  17. Two central questions guided the study: 2.3 Research questions What are the factors which may affect evaluation influence at district and commune levels of Hai Lang district as perceived by the staff involved? How do the identified factors impact on evaluation influence at district and commune levels of Hai Lang district?

  18. 3. Case study setting

  19. 3. Case study setting (1/2)

  20. 3. Case study setting (2/2) • Consisting of 21 communes and township. • A typical rural, in-land district, not a border, remote or mountainous district. • Not a district with a high percentage of ethnic minorities. • Supported by a Finnish-funded Rural Development Programme for 13 years (1997-2009) in 3 phases. Phase I (1997-2000) for 14/21 communes and township. Phases II & III for all communes and townships. • Programme’s main objectives include (1) improving local livelihoods, and (2) strengthening local capacity (participatory planning, M&E, public administration reform, etc.)

  21. 4. Research methodology

  22. Quantitative • Data collection • and analysis • Qualitative • Data collection and analysis • Follow up with • Interpretation 4.1 Research design • Source: Creswell (2012). The study applied a mixed methods explanatory design Methodology: QUAN survey + QUAL study

  23. 4.2 Research instruments & data collection (1/2) For quantitative survey Used a 39-item questionnaire measured by a 7-point Likert scales for a sample of 275 participants Measured 8 dimensions: (1) Lessons learned/best practices in evaluation (2) Programme partners (3) Evaluation capacity (4) Evaluation plans and methods (5) Involvement of programme holders in evaluation processes (6) Generation of evaluation reports (7) Dissemination of evaluation reports/information to stakeholders (8) Evaluation influence

  24. 4.2 Research instruments & data collection (2/2) For qualitative study Used semi-structured interviews and document review. Interviewed 55 purposefully selected participants with 12 different characteristics (see next slide). Documents for review include inspection reports, project/programme proposals, progress and evaluation reports.

  25. 4.3 Hypotheses H1: Involvement of evaluation partners (Factor 1) is positively associated with evaluation influence. H2: Evaluation capacity (Factor 2) is positively associated with evaluation influence. H3: Evaluation plans and methods (Factor 3) is positively associated with evaluation influence. H4: Generation of evaluation report for management (Factor 4) is positively associated with evaluation influence. H5: Generation of evaluation report for partnership (Factor 5) is positively associated with evaluation Influence. H11: Path model (see figure next slide) holds for groups with longer and shorter exposure to Finnish-funded programmes (incl. 5 factors and 5 controls).

  26. 4.4 Limitations Participants’ challenges to recall effects of evaluation may affect explanations of quantitative results. Lack of a comprehensive database of documents and limited accessibility to documents might influence the explanations and confirmations of quantitative results. Bias in interpretation of qualitative results due to fact that author was previously working in the Programme (1 year).

  27. 5. Analysis & findings

  28. 5.1 Quantitative data analysisSteps of analysis • Research question 1: What are the factors which may affect evaluation • influence at district and commune levels of Hai Lang district as • perceived by the staff involved? • Data analysis: Exploratory factor analysis (EFA) using SPSS • Data screening & examination: missing values, outliers, normality • Data entered into computer • Data analysis: Confirmatory factor analysis (CFA) (Reliability, validity analysis) using Amos • Data analysis: Structural path model testing, interpretation of results using Amos

  29. 5.2 Quantitative findings (1/3) • 5 factors resulted from EFA process: (1) Evaluation partners (2) Evaluation capacity (3) Evaluation plans & methods (4) Generation of evaluation report for management, and (5) Generation of evaluation report for partnership • CFA used to test multi-group measurement model, construct validity & reliability, incl. groups with longer & shorter exposure Finnish-funded programme.

  30. 5.2 Quantitative findings (2/3) • Obtained 3 statistically significant factors: (1) evaluation capacity, (2) evaluation plans & methods, and (3) generation of evaluation report for partnership. • No significant difference between groups with longer and shorter exposure to Finnish-funded programme. • None of the control factors (gender, age, years of education, level of responsibility and level of administration) confounds the relationships specified in the model.

  31. 5.2 The path model (3/3)

  32. 5.3 Reliability & construct validity • Criteria: • Construct reliability (CR) > .70 • Construct validity: • Results: • 24 out of 28 items satisfied construct validity and reliability criteria

  33. 5.1 Qualitative data analysis • Research question 2: How do the identifiedfactors impact on • evaluation influence at district and commune levels of Hai Lang district? • Interview data wasanalysedusingNvivo 8 • A visualoverview of qualitative data analysis:

  34. 5.2 Qualitative findings QUAL findings confirm the QUAN findings that both evaluation capacity, and evaluation plans and methods positively affect evaluation influence regardless of the 12 characteristics of interviewees. Generation of evaluation report for partnership positively affects evaluation influence regardless of their college level, level of administration, sectors, groups with longer & shorter exposure to programme. A group of managers perceived no effect of generation of evaluation report for partnership on evaluation influence.

  35. 6. Research and policy implications

  36. 6.1 Contributions to theory The studied conceptual framework found valid in explaining evaluation influence in Hai Lang public sector. The study’s results contributed construct measures capturing evaluation influence in Vietnam district public sector. The study’s results provided new affirmation that evaluation capacity, evaluation plans and methods, and generation of evaluation report for partnership positively affect evaluation influence in Hai Lang. Qualitative data contributed to an understanding of evaluation influence in Hai Lang.

  37. 6.2 Implications for future research (1/2) Replicating this study in other districts with other characteristics than Hai Lang might provide a more comprehensive research setting for better insights of evaluation influence. Research at interpersonal and collective levels might provide a more complete picture of evaluation influence in SEDP implementation in Hai Lang. Longitudinal design might be used to capture perceived changes at different stages of evaluation processes.

  38. 6.2 Implications for policy (2/2) Maximize significant factors by using programme staff as motivators. Balance accountability and learning purposes in design and reporting of inspection and evaluative exercises. Involve non-governmental actors in M&E processes of SEDP implementation. Enhance evaluation capacity of Ministries of Natural Resources & Environment, and Planning & Investment climate change mitigation to improve use of M&E information.

  39. Thank you for your attention!

More Related