Concepts and Application

1. Impact Monitoring in Value Chain Promotion Concepts and Application First sequence of thoughts, prepared for the Training Course „Promoting Promoting Value Chains for Agribusiness Development in Africa“ held in Nairobi, 4th - 7th April 2005 Heike Höffler, GTZ Kenya Deutsche Gesellschaft für Technische Zusammenarbeit (GTZ) GmbH

2. Heike Höffler Kenya 1 A Concept for Impact Monitoring 1. What to monitor? What are impacts? 2. GTZ Impact Model 3. Theory: Six steps to set up a monitoring system 2 Measuring Impacts in Value Chain Promotion 1. Impact Hypotheses – chain of changes 2. Scope of influence 3. Benchmark Data, Milestones and Indicators 4. Methods of Measuring Impact 5. Using Monitoring Information 3 Practice: PSDA Impact Monitoring System Impact Monitoring: Concepts and Application

3. Heike Höffler Kenya 1 A Concept for Impact Monitoring 1. What to monitor? What are impacts? 2. GTZ Impact Model 3. Theory: Six steps to set up a monitoring system 2 Measuring Impacts in Value Chain Promotion 1. Impact Hypotheses – chain of changes 2. Scope of influence 3. Benchmark Data, Milestones and Indicators 4. Methods of Measuring Impact 5. Using Monitoring Information 3 Practice: PSDA Impact Monitoring System Impact Monitoring: Concepts and Application

4. Projects and Programmes need to follow-up their progress Successes and failures need to be looked at while projects and programmes are running Monitoring is an internalised process of team communication, continuously undertaken while implementing, whereas Evaluation is an act of stopping implementation to reflect past activities (but drawing from information from monitoring). Heike Höffler Kenya Monitoring is an ongoing activity! 1

5. Heike Höffler Kenya Impact Chain Impact Model Impact Indicators Outputs Outcome Outreach Benefits Performance Monitoring Input Monitoring Activity Monitoring Output Monitoring Implementation Monitoring Impact Monitoring Impact Assessment Process Monitoring Result-based Monitoring Evaluation The World of Impact Monitoring

6. Heike Höffler Kenya Impact Chain Impact Model Impact Indicators Outputs Outcome Outreach Benefits Performance Monitoring Input Monitoring Activity Monitoring Output Monitoring Implementation Monitoring Impact Monitoring Impact Assessment Process Monitoring Result-based Monitoring Evaluation The World of Impact Monitoring

7. Heike Höffler Kenya Performance Monitoring Impact Monitoring Impact Assessment Input Monitoring Activity Monitoring Implementation Monitoring Process Monitoring Output Monitoring Impact Chain Impact Model Impact Indicators Result-based Monitoring Outputs Impacts Benefits Outreach Evaluation Outcome The World of Impact Monitoring

8. Heike Höffler Kenya What to monitor in projects – Performance & Impact Monitoring 1 • Performance • Observing outputs against planned activities • Providing information for project management • Day-to-day activity •  To trigger short-term adjustments in operation • Impact • Focussing on effect of outputs: impacts! • Observing the direct benefit of outputs • Strategic steering of implementation •  To self-evaluate whether activities contribute to objectives

9. ... changes that have a causal - or at least a plausible - link to a project/programme ... a change of circumstances as a consequence of an intervention, it can be intended or unintended, positive or negative. ... there: from the first moment of intervention and they continue to occur all the time. ... rather the result of social interaction than a straight-forward interventions ... the result of complex interactions and thus, a complex matter to deal with! Heike Höffler Kenya Impacts are ... 1

10. Broad international discussion late 90ies Criticism about the efficiency of development cooperation Criticism about the Monitoring and Evaluation system of GTZ Heike Höffler Kenya AURA • BMZ is a contractor, GTZ is an agent • New structure of the project documents • Changes in political dialogue • Changes in Report obligations Why concentrating on Impacts? 1

11. Heike Höffler Kenya The GTZ Impact Model (I) 1 • Impact Orientation has become a principle of GTZ‘s corporate development. • „quality at entry“  „quality at exit“ • „what has been done“  „what has changed“

12. Project Use of Output Heike Höffler Kenya Indirect Benefit Direct Benefit Model of Interaction 1 Engel, P., The Social Organization of Innovation, 1997

13. Heike Höffler Kenya Indirect Benefit Direct Benefit(Goal) Use of Outputs Poverty Alleviation Outputs Activities GTZ Input Partner The GTZ Impact Model (II) 1 Attribution? Observation !

14. Illustration: which of the 100 Bricks did we donate?

15. Heike Höffler Kenya Indirect Benefit Direct Benefit(Goal) Use of Outputs diff. Impact Levels Poverty Alleviation Outputs Activities GTZ Input Partner The GTZ Impact Model (II) 1

16. Heike Höffler Kenya Pro-poor Rural Economomic Growth Indirect Benefits Attribution Gap Direct Benefit (Goal) Use of Output Output Activities Inputs The GTZ Impact Model (III) 1 Increase in rural employment & income Producers can access international markets Market information is used to change production standards Presentation of Research to stakeholders and publishing Market research for a strategic agricultural product Advisory Services in the Ministry of Agriculture

17. Impact Models form the essential methodological part in project design! Project reports are reduced to the information relevant for political intervention, i.e. Changes in risk Changes in assumptions and: IMPACTS! i.e. Which changes can be observed? Which impacts can be plausibly attributed to the project? Are there unintended impacts? Heike Höffler Kenya Changes in Monitoring! Consequences for GTZ Projects 1

19. Heike Höffler Kenya Theory: Six Steps (GTZ) 1 Step 1: Identify the System Boundaries Step 2: Agree on Purpose and Procedures for Results-based Monitoring Step 3: Agree on Results Hypotheses Step 4: Review Indicators and Define Milestones Step 5: Conduct Data Survey Step 6: Using Monitoring Results GTZ 2004: Result-based Monitoring: Guidelines for Technical Cooperation Projects and Programmes

20. Heike Höffler Kenya Theory: Six Steps (Herweg/Steiner) 1 Step 1: Involvement of Stakeholders and Information Management Step 2: Review of Problem Analysis Step 3: Formulation of Impact Hypotheses Step 4: Selection of impact Indicators Step 5: Development and Application of Impact Monitoring Methods Step 6: Impact Assessment & Follow-up Herweg, K & Steiner, K. 2002: Impact Monitoring and Assessment, Vol. I & II

21. Heike Höffler Kenya Theory: Seven Steps („MAPP“, GDI) 1 Step 1: Preparing a Life Line Step 2: Preparing a Trend Analysis Step 3: Cross-checking with other sources Step 4: Compiling an Intervention List Step 5: Developing the Influence Matrix (connecting trends and interventions) Step 6: Developing the Impact Profile Step 7: Attribution of Impacts to MDGs DIE 2004: Briefing Paper: Impact Analysis of Development Cooperation is Feasible

22. Heike Höffler Kenya 1 A Concept for Impact Monitoring 1. What to monitor? What are impacts? 2. GTZ Impact Model 3. Theory: Six steps to set up a monitoring system 2 Measuring Impacts in Value Chain Promotion 1. Impact Hypotheses – chain of changes 2. Scope of influence 3. Benchmark Data, Milestones and Indicators 4. Methods of Measuring Impact 5. Using Monitoring Information 3 Practice: PSDA Impact Monitoring System Impact Monitoring: Concepts and Application

23. What do we expect to happen in value chain promotion? Heike Höffler Kenya Intervention; i.e. extension Specific Input providers Primary producers Logistics centres, Industry Final Con- sumers Traders Specific Inputs Specific Inputs Trans- formation Trans- formation Production Trade Consumption Trade Consumption C H A N G E ! Impact Models: A chain of changes 2

24. Heike Höffler Kenya Defining the Impact Hypothesis (I) Direct Benefit (Goal) Small-scale producers access international markets Use of Output Farmer group gets EuroGap certified Specific Input providers Primary producers Logistics centres, Industry Final Con- sumers Traders 2 Specific Inputs Trans- formation Production Trade Consumption Output Farmer Groups are setting up an Internal Control System Activities Training on EuropGap Compliance and Certification Inputs Advisory Services for Extension Providers

25. Heike Höffler Kenya Defining the Impact Hypothesis (II) Direct Benefit (Goal) Use of Output Specific Input providers Primary producers Logistics centres, Industry Final Con- sumers Traders 2 Specific Inputs Trans- formation Production Trade Consumption Output Activities Inputs

26. Heike Höffler Kenya Defining the Impact Hypothesis (III) Direct Benefit (Goal) Use of Output Specific Input providers Primary producers Logistics centres, Industry Final Con- sumers Traders 2 Specific Inputs Trans- formation Production Trade Consumption Output Activities Inputs

27. Heike Höffler Kenya Defining the Impact Hypothesis (IV) Direct Benefit (Goal) Use of Output Specific Input providers Primary producers Logistics centres, Industry Final Con- sumers Traders 2 Specific Inputs Trans- formation Production Trade Consumption Output Activities Inputs

28. Heike Höffler Kenya Direct Benefit (Goal) Use of Output 2 Output Activities Inputs Defining the Impact Hypothesis (V) • Each intervention undertaken along a value chain should follow an impact model; • thus: each value chain to be promoted needs a concept of impact hypothesis. • All impact hypotheses should be derived from the project goal; • thus need to be derived from the impact indicators!

29. Heike Höffler Kenya Danger of mono-causality ! Impact Hypotheses – Food for Thought: • Pre-formulation of impact chains can lead to mono-causal conclusions. • Linear following of impact chains can fade out the complex interaction of interventions and impacts. • The higher the impact level, the more a context oriented approach is needed, taking into account development trends. 2

30. Specific single interventions Heike Höffler Kenya Specific Input providers Primary producers Logistics centres, Industry Final Con- sumers Traders Specific Inputs Trans- formation Production Trade Consumption Defining the Scope of Influence (I) 2

31. Specific single interventions Heike Höffler Kenya Specific Input providers Primary producers Logistics centres, Industry Final Con- sumers Traders Specific Inputs Trans- formation Production Trade Consumption Defining the Scope of Influence (I) 2

32. Chain promotion at all stages Heike Höffler Kenya Specific Input providers Primary producers Logistics centres, Industry Final Con- sumers Traders Specific Inputs Trans- formation Production Trade Consumption Defining the Scope of Influence (II) 2

33. Full chain promotion, all stages, all levels Heike Höffler Kenya micro Specific Inputs Trans- formation Specific Input providers Primary producers Logistics centres, Industry Production Trade Consumption Final Con- sumers Traders Associations meso Subsector-specific BDS providers Subsector-specific Technical Agencies Local Government, Providers of Utilities / Infrastructure macro National Government (Line Ministries) & Public Agencies Defining the Scope of Influence (III) 2

34. Heike Höffler Kenya Specific Inputs Trans- formation Specific Input providers Primary producers Logistics centres, Industry Production Trade Consumption Final Con- sumers Traders Associations Subsector-specific BDS providers Subsector-specific Technical Agencies Local Government, Providers of Utilities / Infrastructure National Government (Line Ministries) & Public Agencies Impact Hypothesis - Conclusion 2 2

35. Heike Höffler Kenya Benchmarks, Milestones & Indicators Benchmark Data & Information The state of the art at given point in time (i.e. programme start) Milestones Desired output at a given point in time for (programme) achievements Indicators Determinants for checking whether activities led to the expected output (performance) or expected implications (impact). 2

36. Heike Höffler Kenya Programme level Value chain level t 0 Milestones Progress Reports Benchmark Data Evaluation Benchmarks, Milestones & Indicators(I) Indicators 2

37. Heike Höffler Kenya Usage of Inputs Usage of services Outreach: How many are involved? Employment Gross margins Regional distribution Number of processors Degree of vertical concentra-tion and integration Market Concentra-tion Entry Barriers Price Volatility Seasonality Trade margins Quality Food Safety Product Diversity Benchmarks in Value Chains Specific Inputs Trans- formation Production Trade Consumption 2

38. Heike Höffler Kenya In 18 Month: 2 more firms established At least 2 certified under ISO xxx Cost-effectiveness of at least 2 firms improved by x % 3 processing firms Low process quality High costs of processing Output Activities Inputs From Benchmarks to Milestones Specific Inputs Trans- formation Production Trade Consumption 2 Milestones serve as process indicators. They are directly derived from programme indicators and the impact hypothesis.

39. Heike Höffler Kenya Benchmarks & Milestones in Value Chains Specific Inputs Trans- formation Production Trade Consumption Try to build these „couples“ per stage in a chain that you are targeting with your project. Not all Benchmarks need to be quantitative data; qualitative description also serves the purpose. Market Analyses need to take the need for benchmarks into account (into the TORs!) We might not be the first to undertake this – keep looking for partners in gathering data! 2

40. Heike Höffler Kenya Programme Indicator(s) Direct Benefit (Goal) Use of Output Value chain Indicator(s) Several Milestones in a chain Output Single Milestone Activities Inputs From Milestones to Indicators – or was it the other way round? 2 Production

41. Heike Höffler Kenya The number of profitably operating private processors doubles (2002: 6) Impact Indicator, Component 3, Phase 2 Output Activities Inputs In 18 Months: 2 more firms established At least 2 certified under ISO xxx Cost-effectiveness of at least 2 firms improved by x % 3 processing firms Low process quality High costs of procesing Benchmarks, Milestones & Indicators (II) Direct Benefit (Goal) Use of Output Specific Inputs Trans- formation Production Trade Consumption 2 ...

43. Heike Höffler Kenya The number of primary producers doubles. Example: Employment 3 Customers Market 1 Primary Producers 1 (Research Institute) 1 Large processing companies n = 20 n = 400 n = 450 SM Primary Producers 2 2 Retailers A n = 20 Industrial SMEs n = 6000 Customers Market 2 n = 2400 3 Importers Retailers B n = 200 n = 40

44. Heike Höffler Kenya The value added at producer level increases from 25% to 35%. 35 % 40 % 25 % 50 % 40 % 10 % Example: Value Added Specific Inputs Trans- formation Con- sumption Production Trade Sales Prices / unit 25,00 60,00 100,00 3 Cost / unit Income / unit 23,00 27,00 30,00 2,00 8,00 10,00 Distribution in % - of value-added - of income Specific Input providers Primary producers Logistics centres, Industry Final Con- sumers Traders

45. Heike Höffler Kenya Interviews show that small cale farmers have improved relationship to traders and trust them. Key informant interviews show that wholesale traders are more satisfied with the quality/quanity/reliability of produce delivered by small scale farmers. Example: Better Business Relations Specific Inputs Trans- formation Con- sumption Production Trade Specific Input providers Primary producers Final Con- sumers Traders

46. Heike Höffler Kenya Indicators in Value Chains „There is increasing interest in, and use of, private sector tools for performance and impact measurement; i.e. consumer market research tools for measuring changes in markets and private sector business tools for measuring service provider performance.“ 2 • Some examples: • Customer satisfaction • Repeated customers • Number of enterprises demanding a service • Satisfaction with last service purchased • Percentage of women-owned enterprises

47. Heike Höffler Kenya Where to measure in the chain? What data – qualitative or quantitative? Which degree of participation? Methods of Measuring Impacts Specific Inputs Trans- formation Con- sumption Production Trade • Market Studies • Key Informant Interviews • Point of Leverage – comparisons (before – after) • Participatory Monitoring • Time-series • etc... 2

48. Heike Höffler Kenya Example: Spider Web Diagrammes 2

49. Heike Höffler Kenya E-Val GTZ tool to catch different perceptions of projects and programmes. There are only subjective opinions to changes and impacts Triangulation of opinions: GTZ, partners, and target group 2

50. Heike Höffler Kenya Using Impact Monitoring Systems • Focussing on effect of outputs: impacts! • Observing the direct benefit of outputs • Strategic steering of implementation •  To self-evaluate whether activities contribute to objectives 2 The objective of Impact Monitoring was ....