P4F Evaluative Learning Framework

1. P4F Evaluative Learning Framework Evaluation Manager Presentation to P4F and DFID 15th November 2017 Morning

2. Session 1: Overall Approach & Key Elements

3. Evaluative learning questions • What is the relevance of the P4F programme in responding to the challenges of deforestation, reforestation and poverty reduction? • What is the additionality of the P4F programme, per region and overall? • How effective has the programme been and which interventions are most effective? • Identifying new business ideas and incubating them to mobilize private sector investment and increasing land areas under sustainable management? • Which types of P4F interventions are likely to be most effective in achieving desired outcomes in the immediate post-funnel process and what is the contribution of P4F? Do P4F-supported pilots indicate that achieving desired outcomes is feasible? • How effective are the EC and DSM measures supported by P4F and what is the contribution of P4F? • What kinds of scaling are anticipated and most likely to be achieved by the end of the programme and beyond? • What has been the contribution of the P4F programme in achieving transformational change (i.e. via Scaling, Sustainable benefits, & more Resilience)? What signs are there of changes in the attitudes, knowledge and practices of key actors, particularly private sector actors, which could indicate plausible changes will occur with respect to sustainable forestry and land use? • What lessons can be elicited from secondary sources and from P4F practice to inform P4F adaptive management and DFID policies/programming, from selected themes and deep dives?

4. Collaborative approach • Highly complex programme (internal variation; challenges being addressed; timescales envisaged ) – has implications for EL approach • Generation of robust evidence to test the theory of change (from pre-funnel, through funnel and post-funnel) to generate lessons on what works, under what conditions, & with effective identification and packaging of lessons. • Taking a co-learning approach between P4F and EM, EM creating space for the programme to identify lessons from own experience, and challenging the programme (critical friend), drawing in experience from secondary data and specialists, conducting studies on FPs. • Evidence and lessons will be elicited and fed back into a) P4F’s adaptive management decision-making; b) DFID’s future policies and programmes. Feedback loops are necessary and planned.

5. P4F Programme level Theory of Change Supergoal: Reduced greenhouse gas emissions, reduced poverty and conserved biodiversity by 2025 Impact: Reduced deforestation Timeline Adoption & Adaptation FP partners within sphere of control/influence of project adopt & adapt successful models Expansion Practice change beyond sphere of control/influence of project - among private sector actors (copying / crowding- in) in target sectors/landscapes Response Practice change beyond sphere of control/influence of project – among non-competing actors responding on rules/regulations in target sectors/landscapes Changes/Benefits for partners (e.g. companies) & intended beneficiaries in pilot-related sectors and/or landscapes Proof of concept demonstrated Knowledge Transfer facilitated by P4F Change in practices of project partners, especially companies, and associated actors for uptake of SBM practices 2020-25 Change in practices of project partners, especially companies in pilots observed Capacity of project partners especially companies in direct scaling strengthened (e.g. access to finance, new knowledge etc) Increased global / regional demand for sustainable products Enabling conditions made more favourable globally Capacity of project partners especially companies in pilots strengthened (e.g. access to finance, new knowledge etc) Global / Regional Demand Side Measures Global / Regional Enabling Conditions measures Learning Pilots FPs Learning Direct to Scale FPs 2017-2020 Grants & Technical Assistance Funnel – MOU, Business Plans, Pilots, Readiness for Commercial Scale up Development of Regional Strategies Global & Regional DSMs & ECs identified by P4F / DFID, Or facilitated via the region - integrated within FPs (programme is evolving) Pre-Funnel: Ideas identified/selected

6. Legend: Regional peer review facilitated by EM EM evaluative studies EM Cross-Cutting Thematic Studies Combined Interventions Proposed overall approach by EM Annual Evaluative Learning Cycle Co-generation & review of evidence/lessons, defining priorities for next annual cycle,on-going communications & engagement Timeline Data Collection & Analysis Partner & P4F project monitoring & FP case studies Year 1: Thematic Studies (topics to be confirmed) Year 1 Yr 1: Annual Cycle Year 2: Thematic Studies (topics to be confirmed) Year 2 P4F Adaptive Management Yr 2: Annual Cycle Year 3: Thematic Studies (topics to be confirmed) SE Asia evaluative studies Year 3 East Africa evaluative studies West Africa evaluative studies DFID Policy & Programming Decision-Making Yr 3: Annual Cycle Assessment of P4F contribution to change & lessons on most effective approaches

7. Key elements of proposed approach • Theory based evaluation. • 2 key types of studies at regional level • Deep dives: Individual FP/combinations. P4F flagship projects (most impact anticipated). Tailored design with multiple methods • Thematic studies on cross-cutting issues to facilitate lesson learning, especially to inform P4F adaptive management (including EC/DSM measures) • At the programme level • Annual Cycle of Review of Evidence & Lessons • Combining evidence from a) deep dives of FPs/combined FPs, b) thematic studies, c) (thematic) evaluation of DSM/ECs; d) P4F case studies, d) an analysis of P4F M&E data • Contribution Analysis – Analysing all the evidence and lessons to test the Theory of Change & evaluate contribution of P4F

8. Deep Dives • Deep-dive(s) (FPs or combinations) • Baseline, monitoring and final evaluation • Drawing conclusions on plausible proof of concept, P4F contribution and potential scaling • Includes some comparative analysis, unexpected effects, link to value chain actors, landscapes Deep Dive Selections by P4F Regional Teams & priority selection by EM • West & Central Africa • 1 **The Cocoa & Forestry Initiative (CFI) & Touton Cocoa, West Africa [also linked to RA-Olam and CEMOI] • 2 Unilever, Africa Palm Oil Initiative (APOI) and BOPP /Unilever • 3 FORM, Ghana (timber, charcoal) DSM or EC measures, either integrated in FP or separate DSM/EC global or regional • East Africa • 1 **East Africa Wild Coffee (EWC) • 2 ISLA • 3 Climate Smart Lending Programme (CSLP) South East Asia TBC during regional workshop Landscape/sector focus Global context Regional context National context

9. …

11. Thematic studies • Cross-cutting themes • Using secondary sources, specialists, P4F experience • Flexibility to respond to emerging issues & generate rapid insights • Tailored designs • (EM will undertake approx. 2 per year) • Regional priorities as follows

13. Data collection methods and analysis • Each deep dive will have a tailored set of mixed methods to gather required data (e.g. secondary data analysis; tracking performance of corporate partners and comparison groups; perception surveys of value chain and landscape actors to capture capacity and practice changes, benefits, scaling; surveys of producers/workers; policy document and narrative analysis; sector scorecards) • Each thematic study will have a tailored design – e.g. literature review, engagement with specialists, field research • Different data sets will be analysed using appropriate analytical tools e.g. statistical methods, qualitative data tools such as N-Vivo. • Meta analysis – Contribution Analysis – drawing together data sets to test the programme theory of change and elicit lessons

14. Session 2: Discussion & P4F priorities & inputs

15. Comments and questions… • Overall approach: • Feedback, Comments, Questions • Deep dives • Feedback, Comments, Agree priorities • Thematic Studies • Feedback, Comments, Suggestions/Priorities & Agreement on thematic studies

16. For completion during the meeting… = priorities;

17. Session 3: Theory of Change, including Scaling

18. A framework for understanding scaling pathways

19. P4F Programme Theory of Change Supergoal: Reduced greenhouse gas emissions, reduced poverty and conserved biodiversity by 2025 Impact: Reduced deforestation Timeline Adoption & Adaptation FP partners within sphere of control/influence of project adopt & adapt successful models Expansion Practice change beyond sphere of control/influence of project - among private sector actors (copying / crowding- in) in target sectors/landscapes Response Practice change beyond sphere of control/influence of project – among non-competing actors responding on rules/regulations in target sectors/landscapes Changes/Benefits for partners (e.g. companies) & intended beneficiaries in pilot-related sectors and/or landscapes Proof of concept demonstrated Knowledge Transfer facilitated by P4F Change in practices of project partners, especially companies, and associated actors for uptake of SBM practices 2020-25 Change in practices of project partners, especially companies in pilots observed Capacity of project partners especially companies in direct scaling strengthened (e.g. access to finance, new knowledge etc) Increased global / regional demand for sustainable products Enabling conditions made more favourable globally Capacity of project partners especially companies in pilots strengthened (e.g. access to finance, new knowledge etc) Global / Regional Demand Side Measures Global / Regional Enabling Conditions measures Learning Pilots FPs Learning Direct to Scale FPs 2017-2020 Grants & Technical Assistance Funnel – MOU, Business Plans, Pilots, Readiness for Commercial Scale up Development of Regional Strategies Global & Regional DSMs & ECs identified by P4F / DFID, Or facilitated via the region - integrated within FPs (programme is evolving) Pre-Funnel: Ideas identified/selected

20. P4F Programme level Theory of Change (1/2) Timeline Global/Regional Enabling Conditions measures Global/Regional Demand Side Measures Learning Learning Direct to Scale FPs Pilots FPs 2017-2020 Funnel – MOU, Business Plans, Pilots, Readiness for Commercial Scale up Development of Regional Strategies Global & Regional DSMs & ECs identified by P4F / DFID, Or facilitated via the region - integrated within FPs (programme is evolving) Grants & Technical Assistance Pre-Funnel: Ideas identified/selected

21. P4F Programme level Theory of Change (2/2) Supergoal: Reduced greenhouse gas emissions, reduced poverty and conserved biodiversity by 2025 Impact: Reduced deforestation Timeline Adoption & Adaptation FP partners within sphere of control/influence of project adopt & adapt successful models Expansion Practice change beyond sphere of control/influence of project - among private sector actors (copying / crowding- in) in target sectors/landscapes Response Practice change beyond sphere of control/influence of project – among non-competing actors responding on rules/regulations in target sectors/landscapes Changes/Benefits for partners (e.g. companies) & intended beneficiaries in pilot-related sectors and/or landscapes Proof of concept demonstrated Change in practices of project partners, especially companies, and associated actors for uptake of SBM practices Knowledge Transfer facilitated by P4F 2020-25 Capacity of project partners especially companies in direct scaling strengthened (e.g. access to finance, new knowledge etc) Increased global / regional demand for sustainable products Enabling conditions made more favourable globally Change in practices of project partners, especially companies in pilots observed Capacity of project partners especially companies in pilots strengthened (e.g. access to finance, new knowledge etc)

22. Session 4: Early Lessons & Observations

23. Early observations / lessons • Workshops successful: ToC Development and portfolio analysis could contribute to creation of stronger business plans. • Regional strategies highly relevant given different starting point contexts (forest transition, types of key players, drivers of deforestation). • WCA: Sector transformation model explains focus on ECs & FPs focused upon scaling/institutionalization • EA: Fewer large corporate actors. Smallholder-related deforestation drivers. Strategy focus on FPs tackling smallholder practices. Scaling pathways less obvious. Balancing innovative & target oriented projects – which is more transformative? • Many FPs have integrated DSM and EC measures. Can more synergies be created in this regard?

24. Early observations / lessons (2) • Incubation process complex - Emerging lessons • Differences between the theory (of pre-funnel and funnel processes) and reality on the ground; • Unclear when an FP is really being implemented and so what is the real scope for evaluation; • The importance of clear reporting on the incubation process trajectory for each FP; • Implications for developing relationships with large corporate actors (transaction costs in building trust); • Is more flexibility and room for innovation needed in the incubation process? Is risk aversion leading to missed opportunities and frustration? How do incubation risks compare with assumptions in projections? • Thematic Studies proposed on incubation process and transformative approaches

25. Key Modified Forest Partnerships Maturity Funnel (FPMF) Funnel Pre-Funnel Post-Funnel Entry Entry Exit Exit Decision gate Developing Strategy, Identifying opportunities, Readying for Funnel MoU and full proposal Business plan Pilot or trial Commercial scaleup & investment finance P4F supported Development and commercial funding • Identifying ideas (e.g. via open calls, direct solicitation, website portal) • Receiving & reviewing idea concept notes Scaling Project Implementation & AAER Pathways Forest Partnerships Portfolio Grants Technical Assistance P4F Staff Support

26. Early observations / lessons (3) • Project M&E should be embedded in Forest Partnerships – in business plans, including a well-developed ToC. Data is necessary for successful implementation so partners should be willing to collect. • Implications for EM evaluative learning design: Map trajectories of FPs; Uncertainty & Timescales – spread risk, deliver learning through thematic studies; aim to contribute to understanding assumptions in projected outcomes

27. EL workplan for Y1 *Depending on P4F progress

28. Thank you! Comments & Questions?

29. Results framework indicators – areas for improvement? • Comments and proposed response from EM team • How best to respond to proposed new indicator “The number of such new FP Sustainable Business Models for which proof of concept demonstrated” • Results framework vs. log frame changes • Update on bi-annual indicators review meeting • No changes prior to forthcoming annual review • Proposed changes drafted for consideration post annual review • To be reviewed 12 December meeting • New verifiers and implications for monitoring and reporting?

30. Feedback from Evaluation Manager – slide from indicator review mtg • No significant changes are proposed • Changes that are proposed could be accommodated by revising the existing indicators slightly or by the inclusion of some additional intermediate indicators – suggest the latter within the results framework but not in the log frame for simplicity of administration. • Possible example – around RFI-026 (pilot stage); • The success rate of new SBMs (FPs) piloted by P4F, showing • The no. of new SBMs piloted; • The number of such new SBMs for which proof of concept demonstrated • The number of such new SBMs which have failed and • The number for which proof of concept is pending • These indicators are all available from our internal systems and processes. A challenge will be with setting milestones but perhaps that is one we need to face? • Note: alignment of terminology by EM to P4F terminology is required (SBM = FP)