Ballin Marco, Carbini Riccardo, Loporcaro Maria Francesca, Lori Massimo,

1. The use of information from experts for agricultural official statistics Ballin Marco, Carbini Riccardo, Loporcaro Maria Francesca, Lori Massimo, Moro Roberto, Olivieri Valeria, Scanu Mauro Istituto Nazionale di Statistica Roma, 10 July 2008

2. Aim of the work and summary of the talk Q 2008 • The main aim of this work is to investigate if we can assess the quality of statistics based on experts opinion and if these statistics can be used as official statistics • The talk • The Italian experience • Some reminders on elicitation and a proposal for an elicitation scheme • Some results belonging to an experimental elicitation • Quality report and next steps in the research Roma, 10 June 2008

3. Introduction Q 2008 Up to now, expert opinion has been widely used to produce short term statistics on crops. Main users: • Eurostat • System of National Accounts • Economic Operators Summarizing, local authorities supply an evaluation of area and yield on the different crops (Jannuary: estimates on areas for winter cereals, May: estimates on area for maize, June estimates on yields for winter crops ,….). Roma, 10 June 2008

4. Example Q 2008 Example: common wheat area (national level)according to the Farm Structure Survey (sample survey) and aggregated evaluations from local authorities Roma, 10 June 2008

5. Example Q 2008 Example 2: maize area (national level) according to the Farm Structure Survey (sample survey) and aggregated evaluations from local authorities Roma, 10 June 2008

6. Why a survey on expert opinions? Q 2008 • …a first judgement • Up to now in Italy these “statistics” seem to reproduce quite well estimates produced by sample surveys for many crops • …some other important strengths: • Timeliness (results available before traditional survey estimates) • Analytic data (estimates with a high geographical detail) • Inexpensive Roma, 10 June 2008

7. Why a survey on expert opinions? Q 2008 • … but also some weaknesses: • Heterogeneity of the process at local level • Lacking of accountability • Problems in assessing data quality • In other words • difficulties in defining and filling up a quality report Roma, 10 June 2008

8. Proposal Q 2008 From evaluation to structured elicitation To overcome these difficulties and to transform experts opinion inan additional tool for official statisticians useful to investigate phenomena difficult to observe with traditional surveys we propose to adopt a formal expert elicitation “Expert elicitation in the context of uncertainty quantification aims at a credible and traceable account of specifying probabilistic information regarding uncertainty in a structured and documented way.” (Hora, 1992) Roma, 10 June 2008

9. Proposal Q 2008 From evaluation to structured elicitation Elicitation is currently and successfully applied in many experimental situations, as weather forecasting, biomedical applications, nuclear risks assessment, attributing foodborne pathogen illness to food consumption … Protocols on the elicitation process have already been introduced in many research fields. These allow to evaluate the quality of the elicitation process, making the users aware about the use of the statistical results …….a proposal of elicitation in the official statistics context Roma, 10 June 2008

10. Proposal Q 2008 • 1. Organization • Definition of the problem (and the questionnaire) • Finding out at least one expert and one facilitator • To train the experts and the facilitators • 2. Elicitation • Carry out the interviews • Transform expert opinions into probability distributions • Combine opinions of different experts (if available) • Feedback • Production of final results Roma, 10 June 2008

11. Proposal Q 2008 • 3. Evaluation of the elicitation results (quality report) • Timeliness, Coherence, Relevance, Comparability, Accessibility (as usual statistics) • Accuracy • Variability of elicited distributions (if more then one distribution have been elicited; this indicator should replace the usual variance) • Effect of feedback (this should replace the indicators on effect of re-interview) • Fiducial interval (this should replace the confidence interval) • Assessment of expert knowledge by means of control/seed variables (this indicator should replace indicators on bias) • Furthermore • Self assessment of expert knowledge and of his/her sources of information (by expert) • Assessment of how the questions in the questionnaire are phrased (by expert) • meta-information on the interview (by facilitator) Roma, 10 June 2008

12. An experience: the saffron case Q 2008 • Step 1. Organization • Problem definition: structure of saffron sector in Italy (Saffron is a rare crop, not observable by sample surveys) • Questions on the following phenomena: production and total area of saffron in Abruzzo, in Italy, in the world; export and import quantities; number of planted bulbs in the last year; forecast on production, number of operators,… • Selection of experts: president of the consortium of saffron producers in one of the two most important areas for the production of saffron in Italy (Navelli county) • Selection of facilitators: Istat personnel with strong agricultural background belonging to regional office • Training: document on how the elicitation process is conducted and on the main biases that can affect the elicitation process. Mail and phone contacts Roma, 10 June 2008

13. An experience: the saffron case Q 2008 • Step 2. Elicitation • face to faceInterview (about one hour) on • Minimum, Maximum, Mode (most probable value) for each phenomenon of interest • Distribution shape using fixed interval method (Ten “X” to be put in five intervals of equal length) • Check of data and first results • Check of data (compatibility, units measure, etc.,…) • Fitted distribution (Rectangular or Beta) • Point estimates • Fiducial interval • Feedback from expert and facilitator • Update of elicitation by expert and facilitator • Final results Roma, 10 June 2008

14. An experience: the saffron case Q 2008 Final result (concerning production of saffron in 2007) 120 Kg is the point estimation (mode) (105 kg -130 kg)is the range within the production of saffron in 2007 lies with probability one for the expert 111-126 Kg is the 95% fiducial interval according to the expert distribution Expert info Point estimate Fitted distribution Fiducial interval Roma, 10 June 2008

15. An experience: the saffron case Q 2008 • Step 3. Evaluation of elicitation results • Some indicators among those proposed before, • Effect of feedback (expert confirmed his point of view after the production of the first report) • Fiducial intervals (e.g. 111-126 Kg for the production of saffron in Italy) • Self assessment of expert knowledge and of his/her sources of information Index (sum of points/max) 81,2% Roma, 10 June 2008

16. An experience: the saffron case Q 2008 • Step 3. Evaluation of elicitation results • Assessment of how the questions are phrased (by expert) • Results on number of planted bulbs could produce misunderstanding if they are not integrated with information about type and quality of bulbs • Meta-information on the interview (by facilitator): • Expert declared and seemed to be honoured to collaborate with Istat • He appeared well trained and showed a true mastery of the topics • He collaborated during the feedback too • He has years of experience and has written many publications on the topic Roma, 10 June 2008

17. Final summaryand Future developments Q 2008 • Summary • The elicitation process formalizes the uncertainty of one or more experts on particular phenomena • Statistics based on experts have some good properties • In some cases expert opinion is the only available sources of information • It is possible to define protocols and quality reports to upgrade “numerical evaluations” to statistics • some indicators have been proposed for accuracy • A small experience has been (partially) illustrated • Future developments • Combination of elicited distribution belonging to two or more experts (by feedback, mathematical aggregation, …) • Combination of elicited distributions and data belonging to surveys (linear combination, bayesian framework, ….., used as early estimates to be replaced when data are available). • Development of protocols for fields different from agriculture • ….. Roma, 10 June 2008

18. Q 2008 Bibliography Anthony O'Hagan, Caitlin E. Buck, Alireza Daneshkhah, J. Richard Eiser, Paul H. Garthwaite, David J. Jenkinson, Jeremy E. Oakley, Tim Rakow (2006) Uncertain Judgements: Eliciting Experts' Probabilities. Wiley Di Bacco (1990). l’aggregazione di valutazioni diprobabilità: una rassegna…non imparziale , 8, ed. Pitagora. Bologna Garthwaite P. H., Kadane J. B., and O'Hagan A. (2005): Statistical methods for eliciting probability distributions, Journal of the American Statistical Association, 100, 680-701. ISTAT (1993): Manuale delle statistiche agricole rilevate con le tecniche estimative. Note e relazioni, 1993, 1. Roma: ISTAT. Mortera J.(1990). “Aggregazione delle opinioni: una panoramica” Rassegna di metodi statistici ed applicazioni, 8, ed. Pitagora. Bologna SHELF (Sheffield Elicitation Framework): http://www.tonyohagan.co.uk/shelf J.P. van der Sluijs, P.H.M. Janssen, A.C. Petersen, P. Kloprogge, J.S. Risbey, W. Tuinstra, J.R. Ravetz (2004):Tool Catalogue of the RIVM/MNP Guidance for Uncertainty Assessment and Communication Roma, 10 June 2008

19. Q 2008 Saffron: a flower – many recipies Saffron is obtained from the yellow “core” of the flower Crocus. It is mainly produced in Iran and India, with minor productions in Spain, Greece, and Italy. Italy produces about 120 kg of saffron per year. It is used in many regional recipes, as the Risotto alla Milanese Roma, 10 June 2008