Evidence and causality in the sciences University of Kent, Centre for Reasoning 5-7 September 2012

1. Evidence and causality in the sciences University of Kent, Centre for Reasoning 5-7 September 2012 Evidence aware policies, causality and plurality of sciences Catherine Laurent * • * Institut National de la Recherche Agronomique • Department « Science for Action and Development » • UMR 1048, Paris, France

2. 1. Quality of evidence, plurality of sciences, policy making 2. When theoretical plurality is deliberately ignored. A case study (E. Duflo et al.) 3. Consequences for knowledge and policy 4. Causality analysis as a policy instrument

3. 1. Quality of evidence, plurality of sciences, policy making 2. When theoretical plurality is deliberately ignored. A case study (E. Duflo et al.) 3. Consequences for knowledge and policy 4. Causality analysis as a policy instrument

4. Evidence, plurality of sciences and policy making. A presentation embedded in on-going exchanges between scientists (economics, ecology, agronomy, political science…), philosophers of science (Anne Fagot-Largeault, Daniel Andler, Marc Kirsch, Vincent Guillin…), students, enterprises and policy makers (mainly from the Ministries in charge of agriculture and environment). Research projects with international collaborations, seminars, annual 3 days training session for Scientists, PhD students, and policy makers.

5. Evidence, plurality of sciences and policy making. Overall objectives: - To reach a certain agreement on what is an evidence and how the quality of evidence should be assessed (types of evidence, level of proof, social relevance…) • To make the most of the plurality of theories and methodologies for both research and decision making. To produce metaknowledge on this plurality for decision makers. • To import and test the heuristic potential of these discussions on evidence in several areas of applied research and policy making.

6. Evidence, plurality of sciences and policy making. Two major conclusions: 1) the huge heuristic value of the debate on evidence for many areas of research and policy making 2) the danger of a discussion on evidence that would not consider seriously the plurality of sciences.

7. 1. Quality of evidence, plurality of sciences, policy making 2. When theoretical plurality is deliberately ignored. A case study (E. Duflo et al.) 3. Consequences for knowledge and policy 4. Causality analysis as a policy instrument

8. Case study 1 (Duflo et al.) A work presented by the authors as emblematic of their approach. Mentioned in many papers. An experimental framework regarding the purchase of fertilizers by small scale farmers in Kenya (Busia district). The final aim was to orientate public decision regarding access to fertilizers (subsidies, extension, market regulation) The results are mentioned in many papers, interviews, etc. They are presented as high level evidence, that justify a change of public policies [4, 5]. They are also presented as an example that the J-PAL produces evidence of causality with a high degree of generality [2,p.357]. [1] Duflo E. 2003. Poor but rational? MIT/NBER/CPER/ 12 p. [2] Duflo E. 2004. Scaling up and evaluation. Annual World bank conference for reconstruction and development. 341-369. [3] BanerjeeA., Duflo E. 2008. The experimental approach to development economics. NBER WP n°14467. [4] Duflo E., Kremer M., Robinson J., 2008. How high are rate of return to fertilizers? Evidence from field experiment in Kenya. AEA. 21 p. [5] 2009. Nudging farmers to use fertilizers. Evidence from Kenya. Working Paper 53 p. [6] 2011. Theory and experimental evidence from Kenya. American Economic review. 2350-2390.

9. Case study 1 (Duflo et al.) Duflo et al. have worked with NGOs in the Busia district. They feel that public policies should not subsidize access to fertilizers. Duflo et al. have themselves verified in small plots (30 square meters) in 60 farms that the use of fertilizers results in a surplus yield whose value exceeds the price of these fertilizers (1) when the production is sold on the market. Then farmers –who are willing to maximize their profit- will use them if they are aware of these results. There is a need for information, not subsidies. The results of the regional agronomic research (KARI) are only mentioned to support the "agronomic experiments" of Duflo et al. The authors do not consider the results of decades of agronomic and economic research/ extension in the regions, They do not consider the diversity of production systems, soils, labor organization, etc.). Duflo E., Kremer M., Robinson J., 2008. How high are rates of return to fertilizers? Evidence from field experiment in Kenya. American Economic Review Paper, 482-488

11. Recommendations for economic policies and public action. Access to fertilizers should be regulated by the market. No subsidies.

14. There are several ways to deal with the anomalies of a research programme To reject the theory. To acknowledge the negative heuristic of the theory, its imperfections. To build auxiliary hypotheses to protect the hard core. This can be done in different ways. 1) By giving any factual interpretation of the anomaly.

16. There are several ways to deal with the anomalies of a research programme To reject the theory. To acknowledge the negative heuristic of the theory, its imperfections. To build auxiliary hypotheses to protect the hard core. This can be done in different ways. 1) By giving any factual interpretation of the anomaly. 2) By introducing evidence of causalities from other theories, within the same discipline, or from other disciplines.

18. There are several ways to deal with the anomalies of a research programme To reject the theory. To acknowledge the negative heuristic of the theory, its imperfections. To build auxiliary hypotheses to protect the hard core. This can be done in different ways. 1) By giving any factual interpretation of the anomaly. 2) By introducing evidence of causalities from other theories, within the same discipline, or from other disciplines. To transform the world in order to make it consistent with the theory. For instance, by designing experiments where individuals will be very strongly encouraged to behave as predicted by the theory.Duflo E., Kremer M., Robinson J., 2009. Nudging farmers to use fertilizers. Evidence from Kenya. Working Paper 53 p.2011. Theory and experimental evidence from Kenya. American Economic review. 2350-2390

19. *

20. An experimental framework that prevent people from thinking • "By requiring an immediate decision during the field officer's visit and offering a simple option the programme may have reduced time spent thinking through which type of fertilizer to use and in what quantity" (Duflo et al 2009, p.18) (This slide is dedicated to the agronomists, and specially to Mama Lyimo, S.Deckers, M.LeBail)

22. A Here no relevant evidence of causality that could helpful for development objectives. The theory does not progress. Rather high level of evidence of effectiveness (pseudo RCT) but measured in a way that is widely considered as non relevant (no link with the global performances and objectives of the farm household, etc).

23. 1. Quality of evidence, plurality of sciences, policy making 2. When theoretical plurality is deliberately ignored. A case study (E. Duflo et al.) 3. Consequences for knowledge and policy making 4. Causality analysis as a policy instrument

24. Consequences – knowledge- • Reliability. The research programme does not progress. It is maintained thanks to an auxiliary hypothesis that consider that 90% of the population has a pathological behavior. In spite of this importance of the negative heuristic of the research programme, the authors make policy recommendations. The complexity of causal structures is denied. Theoretical imperialism; pluralism, interdisciplinarity are rejected. • Relevance. Some high level evidence is produced on the effectiveness of the action proposed in the experiment. But such evidence is not relevant for most of the actors (farmers [type of measurement of the final performances…], extension service [lack of reliable technical knowledge…], local authorities [do not consider subsistence farms], etc.) . • Type of evidence. In the absence of evidence of causality, evidence of effectiveness of an action is useless for designing a programme in an other context.

25. Consequences –power- Ethical and political consequences - Representations of the society An hypothesis of pathological behavior, that is despising for the farmers involved in the survey. Such classifications are interactive kinds and may influence the behavior of the people who are classified, once they are aware of it (Merton, Hacking) (also for instance the two categories "sophisticated farmers / Absent minded farmers") - Democracy The authors argue that the farmers have "behavioral biases" [5, p.3], thus it is legitimate to reduce their decision margin (less time to make a decision = less procrastination). • Accountability Adverse effects, lack of efficiency, result from an inadequate behavior of the farmers.

26. 1. Quality of evidence, plurality of sciences, policy making 2. When theoretical plurality is deliberately ignored. A case study (E. Duflo et al.) 3. Consequences for knowledge and policy making 4. Causality analysis as a policy instrument

27. Causality analysis as a policy instrument? • (Salamon 2002, Lascoumes Le Galès 2007) A public policy instrument constitutes a device that is both technical and social, that organizes specific social relations between the state and those it is addressed to, according to the representations and meanings it carries. It is a particular type of institution, a technical device with the generic purpose of carrying a concrete concept of the politics/society relationship and sustained by a concept of regulation (Lascoumes, Le Galès 2007).

28. Causality analysis as a policy instrument? • (Salamon 2002, Lascoumes Le Galès 2007) A test case. Farmers are using pesticides. Some pesticides are dangerous and may harm farmers' health. A label on the packaging of pesticides mentions the danger of the products and gives some indications to use it with no risk for the applicator's health. In spite of the danger, many farmers are not following these indication and may get intoxicated.

29. Causality analysis, as a policy instrument? Causality analysis 1. Farmers are homo oeconomicus, maximizing profit. They evaluate the risk and make their decision accordingly. They do not put a protection equipment because they prefer to save time (and money) on this item. to improve information on danger (standard economics, supported by the pesticide industry) Causality analysis 2. Farmers are rational, but they do not give priority to income. In depth investigation on protection equipment show that those are not adapted to agriculture, dangerous when it is hot (risk of heart attack). This is one of the causes of their behavior protection equipment design (but what if no possible protection???) (ergonomy, farmers associations) Causality analysis 3. Farmers' behavior results from both conscious and unconscious processes. They are scared of pesticides danger and do not consider that they can protect themselves and their family. They deny the danger and adopt inconsistent behavior (C. Nicourt 2011). • to accept / organize debates on the necessity to reduce the danger and not only the risk (social psychology, women)

30. Causality analysis, as a policy instrument Each of this analysis correspond to a specific research programme. Each programme permits to produce a certain level of evidence (RCT are possible for few of them only). There are contradictions between the hypotheses of the hard cores of these research programme. But each of them provides results that illuminates a piece of world; for practical issues these results may complement each others. For policy makers, two possibilities: • To focus on one type of analysis (and favor the associated interests) • To combine learning from various approaches. In that case, to design evidence aware-policies, there is a huge need for metaknowledge on sciences plurality, to help decision makers to assess the limits of each approach. The danger of a discussion on evidence that would not consider seriously the plurality of sciences / The difficulties to fully introduce that dimension in the practice.

36. Des preuves, aux décisions fondées sur (ou éclairées par) des preuves • 3 recommandations - développer un certain accord sur ce qui constitue une preuve … des avancées conceptuelles permettent de mieux élucider situations d'usages des preuves mais, au cœur de la pratique, la question de la concurrence des preuves • Souligner le pluralisme théorique et méthodologique plutôt que de poursuivre les antagonismes paradigmatiques …possible mais tous les acteurs (y compris chercheurs) n'y ont pas intérêt. • Rassembler les différents acteurs concernés (réfléchir à diverses formes de délibérations et d'accès aux connaissances…) pour que les résultats des recherches aient un impact élargi, au-delà des camps constitués …nécessite des dispositifs matériels et organisationnels spécifiques à créer et dont le fonctionnement démocratique est à inventer

37. Diversité des approches en termes • d'"evidence-based ou evidence-aware policy" Ce qui est en jeu: la relation à la pratique Les débats et controverses sont nombreux. Des points sensibles • Concurrence des preuves (par ex. ne pas favoriser l'usage de "preuves" de haut niveau au détriment de "preuves" pertinentes) • La nécessité de tenir compte de l'incomplétude des savoirs produits par une théorie scientifiques et les moyens de le faire (par ex. tenir compte de la pluralité des théories, réfléchir à la combinaison de différentes formes de savoir pour l'action, etc.).

38. Types de preuves Preuve de présence : mise en évidence de la présence d'une chose, par exemple inventaires biologiques pour la biodiversité Preuve de causalité : mise en évidence d'une relation de cause à effet, toutes choses égales par ailleurs. EX1. : des molécules de la famille des triazoles (prothioconazole…) inhibent un mécanisme de synthèses des stérols nécessaire au développement de fusarium, champignon des cultures. Preuves d'efficacité : mise en évidence de l’efficacité (plus ou moins grande) d’une action (action d’une molécule, d’une personne, d’une politique…) indépendamment de la connaissance des raisons de cette efficacité. Ex1. Au Mexique, augmentation avérée des performances scolaires d'une catégorie d'élèves suite à crédit fait aux parents. EX2. des molécules ont un effet fongicide sans que l’on connaisse leur mécanisme d’action Preuve de dangerosité (/innocuité). Mise en évidence de la dangerosité d’un produit, d'une action, par exemple –dans un sens élargi- impact négatif d'un conseil agricole inadéquat sur la survie d'exploitations de petite dimension.

39. Niveau de “preuve” • Opinions d’autorités respectées fondés sur l’expérience, études descriptives, ou rapports de comités d’experts • 2. “Preuves” obtenues à partir de comparaisons historiques ou géographiques. • 3. “Preuves” obtenues à partir d’études de cas réalisées dans un dispositif contrôlé • 4. “Preuves” avec un recueil de données pour un échantillon représentatif permettant des tests statistiques pour valider la robustesse des résultats, voire ERC • Economie du développement Opinions d’autorités respectées fondées sur l’expérience clinique, des études descriptives ou des rapports de comités d’experts. “Preuves” à partir de comparaisons historiques “Preuves” à partir d’une cohorte ou d’une étude cas-témoins. “Preuves” obtenues à partir d’au moins un Essai randomisé contrôlé. Médecine

40. Pluralité des sciences: pluralité théorique intra-discipline

41. Pertinence des preuves Une preuve est pertinente quand elle correspond pleinement au phénomène à considérer (pour comprendre un aspect particulier de réel ou atteindre un objectif).Mais précisément il n’y a pas forcément accord sur le type de phénomène qu’il faut considérer. Diffère selon Acteurs et à leurs intérêts(par ex. preuves à partir d'un groupe social particulier) [importance de tenir compte de la diversité sociale] Le contexte culturel, social, économique(ex. divergences sur les fonctions environnementales de l'agriculture selon les pays [importance des approches comparatives] Les théories mobilisées et les hypothèses qui les sous-tendent(par exemple importance analyse des régulations institutionnelles versus processus auto-organisation) [importance de tenir compte de la pluralité des théories intra- et inter-discipline] Laurent & Trouvé, 2011. The question of « evidence » in the emergence of evidence-based or evidence-aware policies in agriculture. EAAE Ancona

43. Du rêve à la réalité Dans l’idéal des décisions informées par des types preuves adéquats à leur objet, de haut niveau, pertinentes Dans la réalité des arbitrages, des contradictions et des complémentarités à éclairer  Pour éviter usage de preuve de haut niveau mais peu pertinentes, au détriment de preuves pertinentes de moindre niveau au sein des disciplines, entre disciplines, entre acteurs Pour expliciter types de preuves mobilisées par les acteurs dans le débat et mieux maîtriser leur complémentarité Pour expliciter les positions des professionnels de la production de preuves et se protéger des arguments d'autorité

44. Etude de cas 1. type de preuve, évaluation des politiques publiques et conception de nouvelles politiques publiques Pour évaluer l'efficacité d'une mesure de développement, nécessité de produire des preuves d'efficacité (par exemple augmentation avérée des performances scolaires d'une catégorie d'élèves suite à crédit fait aux parents) Comment généraliser? Une réponse peu satisfaisante et la plus fréquente pour certains promoteurs des ERC en économie du développement: dupliquer. (Business de l'ERC) Esther Duflo 2004. Scaling up and evaluation.

45. Etude de cas 1. type de preuve, évaluation des politiques publiques et conception de nouvelles politiques publiques L'autre possibilité, combiner la mesure de l'efficacité et la compréhension des relations causales qui expliquent succès ou échec de l'action. Une démarche qui est inégalement mise en oeuvre "Theory provides some guidance about what programs are likely to work and, in turn, the evaluation of these programs forms a test of the theory prediction" (Duflo 2004, Scaling up and evaluation, p.357). "Theory« , singular….

47. When tude de cas 1. type de preuve, évaluation des politiques publiques et conception de nouvelles politiques publiques L'autre possibilité, combiner la mesure de l'efficacité et la compréhension des relations causales qui expliquent succès ou échec de l'action. Une démarche qui est inégalement mise en oeuvre "Theory provides some guidance about what programs are likely to work and, in turn, the evaluation of these programs forms a test of the theory prediction" (Duflo 2004, Scaling up and evaluation, p.357). "Theory« , singular….

48. De l'anomalie épistémique à la pathologie mentale • "Pathological" modes of individual and group behaviors: procrastination in decision making, undue obedience to authority (…).In each case, individuals choose a serie of current actions without fully appreciating how those actions will affect future perception and behaviour. The standard assumption of rational, forward-looking, utility maximizing is violated" (p.1) Akerlof G., 1991. Procrastination and obedience. The American Economic Review, Vol. 81, n°2, 1-19

49. Décrire la pluralité théorique intra-discipline • Lorsqu'ils se réfèrent à la pluralité des théories existantes les scientifiques se tournent régulièrement vers des concepts philosophiques. Les notions de "paradigme" de T.Kuhn et de "programme de recherche" de Lakatos sont parmi les plus prisées. • La notion de "programme de recherche" paraît plus appropriée pour décrire la variété des théories qui coexistent à un moment t. • Elle permet de considérer toutes les théories au même niveau sans privilégier celles qui sont socialement dominantes propose un cadre de description plus précis que celle de paradigme • Elle décompose la description en plusieurs éléments

50. Projet: décrire la pluralité théorique intra-discipline