Measuring Impact of Certification: Towards a Focus for Future Research ISEAL Conference, June 8, 2011

1. Measuring Impact of Certification: Towards a Focus for Future ResearchISEAL Conference, June 8, 2011

2. Documented Research Strategy Standard Indicators Demonstrating Impact: Now and Tomorrow Myth  Anecdote  Observation  Comparison  Replicated  RRC Now Academic Practitioners Academic Future Practitioners

3. Business Model of Certification Consumer behavior (values) Retailers Brands Manufacturers Traders Processors Business Practices in Value Chain Production Unit (Best Management Practices) Sustainability feedback: Demand for sustainable markets Sustainability feedback: Evidence of Impact Measureable Ecological, Social, Economic benefits

4. What is the hypothesis to test? All study designs must first start with a question or set of questions to be answered in the evaluation. For example….. • What are the monetary costs and benefits of implementing the SAN standard (including price premiums), and how do they change over the lifetime of a certificate? • How do farm productivity, crop quality and soil health compare between farms that do and do not implement best management practices we assume change these conditions? What are the independent variables (Controlled and varied by study) and dependent variables (determined by independent variable)?

5. How to develop Evaluation and Research Model What is the meaningful Independent Variable? Consumer behavior (values) Retailers Brands Manufacturers Traders Processors Business Practices in Value Chain What is the meaningful Independent Variable? Production Unit (Best Management Practices) Sustainability feedback: Demand for sustainable markets Sustainability feedback: Evidence of Impact Measureable Ecological, Social, Economic benefits

6. Evaluation of interventions not business models Unbundling certification to measure adoption of best management practices (BMPs). Operation 4 Operation 3 Operation 1 Operation 2 NON-CERTIFIED NON-CERTIFIED NON-CERTIFIED CERTIFIED BMP1 BMP4 No BMPs BMP1 BMP2 BMP3 BMP4 BMP1 BMP2 BMP3 BMP4 Gains us a more nuanced understanding of what production and social and environmental management practices drive what economic, social or ecological returns, and under what conditions. Informs learning and refinement of standards.

7. DETERMINING HYPOTHESIS BASED ON COST, LENGTH OF STUDY AND LEVEL OF CREDIBILITY NECESSARY Dependent variables. Determined by control and variation of independent variable Reduced encroachment Independent variable. Controlled and varied by study Productivity Level of confidence Desired Impact? Time to see an impact Indirect Adoption of RA best management practices Action Costs of measuring change

8. Hypothesis TESTING: Research Strategy Before and after implementation of BMPs/Certification is Research that tests hypothesis by reaching conclusions about relationship between dependent and independent variables attempt to discover new information will uses uses demonstrate that what we say we do is true Control group Intervention (experimental) group Dependent variable Independent variable examine the validity of a project’s hypothesis are are is is Exposed to the intervention (experimental) variable Exposed to all elements of the research, except the intervention (experimental) variable Determined by independent variable Control and varied 8

9. Hypothesis TESTING: Research Strategy Before and after implementation of BMPs/Certification is Research that tests hypothesis by reaching conclusions about relationship between BMPs and farm income variables attempt to discover new information will uses uses demonstrate that what we say we do is true Farms with no BMPs Farms with BMPs Real Farm Income BMP(s) examine the validity of a project’s hypothesis are are is is Exposed to elements such as international prices for coffee, advantageous weather conditions or improvements in processing and marketing,. Exposed to all elements + BMPs Determined by control and variation of BMPS Controlled and varied by RA 9

10. Contribution vs. attribution DEMONSTRATING CONTRIBUTION OF RA PRACTICES • Measure before and after implementation of BMPto see changes in social, economic and environmental conditions. • The implicit assumption is that had farms not applied the BMP, their outcomes, on average, would have stayed the same. • Results will show impact of BMPbut will not isolate the influence of these practices from the impact of factors unrelated to BMP implementation, e.g. international prices also influences income, advantageous weather conditions also influences productivity, a government intervention also improves working conditions. • BMP implementationmust be presented as just ONE of a possible number of factors influencing desired outcomes.

11. Contribution vs. attribution DEMONSTRATING ATTRIBUTION OF RA PRACTICES • Measure before and after implementationto see changes in social, economic and environmental conditions. • At the same time also measure changes in social, economic and environmental on farms that DO NOT implement BMP as the counterfactual outcomes (comparison group). • The implicit assumption is that had farms not implemented BMP , their outcomes would be the same, on average, as those of farms in the comparison group. • With a comparison group, the impact of BMP implementationcan be defined as the difference between the observed intervention outcome and the counterfactual outcome. • BMPscan be attributed to desired outcomes because counterfactual outcome accounts for changes as a result of influencing factors, except BMP implementation

12. Constructing a credible counterfactual • A credible counterfactual must address selection bias, a problem that will violate assumptions when farms select themselves – or are selected by NGOs/traders – into certification. • 2 approaches to eliminate selection bias • Experimental • Quasi-experimental. • Each varies in feasibility, cost, the degree of clarity and scientific validity of results.

13. Experimental Design (optimal approach) • Involves gathering a set of farmers equally eligible and willing to participate in certification and randomly dividing them into two groups: those who receive the technical assistance (treatment group) and those from whom the intervention is withheld (control / counterfactual group).

14. Quasi-Experimental Design • Consists of constructing a comparison group using matching comparisons. This involves identifying non-certified farms comparable in essential characteristics to certified farms. Both groups should be matched on the basis of very similar observable characteristics that plausibly affect outcomes • Ideally matched comparison groups should be selected before project implementation, not afterwards.

15. Meta Analysis of Literature • Looks at set of specific BMPs across standard systems (Ag, Forestry, Fisheries) as list of independent variables • Categorize by dependent variables examined • Method • Statistical analysis applied • Result - Statistical significance • Geographic scope / focus • Time frame adequate - based on dependent variable (Oak tree versus Sunflower)

16. The Rainforest Alliance works to conserve biodiversity and ensure sustainable livelihoodsby transforming land-use practices, business practices and consumer behavior.