Evaluators’ perspectives on evaluation – perspectives for the future

1. Evaluators’ perspectives on evaluation – perspectives for the future F. Dubois-Arber, Institut universitaire de médecine sociale et préventive, Lausanne Evaluation in public health – lessons learned and future directions

2. Conflicting expectations regarding evaluation • Decrease in the resources available for public policies • Increasing demand for evaluations, especially for policy evaluations • Demand for accountability (politicians) • Demand for « high level of evidence » (policy makers) • Demand for guidance, consensus building (programme managers) • Demand for immediate answers on long term issues

3. Consequences • Nature of evaluations • Approaches to evaluation and tender procedures • Methods • Profession

4. Less evaluations- higher quality, comprehensive, policy/ programmes evaluations Issues at stakes for increased relevance to various stakeholders : • Earlier in the programme/ policy process • More time for the preparation of the evaluation • Adequate financing • Adequate duration of evaluation • Evolutions in the design, in the expertise brought in the evaluation • Flexibility in responding to multiple evaluation needs

5. More user-focused evaluations and more appropriate procedures of call for tender • This approach should be – from the very beginning – embodied in the direct exchanges and the discussions between programme managers/policy makers and evaluators, with the commissioners as facilitators • In their current format, calls for tender are not the best way of choosing the evaluators who will be able to take into account the various needs in evaluation • The process should be more open with the possibility of discussing evaluation questions and possible options for evaluation, scenarii and trade-offs, etc. in direct interaction before proposing protocols • Dedicated resources (time and money) would allow a better appropriation of the matter and of the issues at stakes: investment in evaluation proposal

7. Evaluating policies, evaluating systems • Need to acknowledge the « systemic » nature of policies/ programmes and evaluate them as such • Better understanding and measurement of the evaluation object (policy/programme) • Appreciation of the possibilities of obtaining results (policy/programme theory) • Appreciation of the outcomes timing • Assessment and measurement of the intensity of interventions • Assessment and measurement of the combination and complementarity/opposition of interventions • Assessment and measurement of interventions coverage • Assessment and measurement of relevant elements of the environment (political will, convergence/ divergence between stakeholders, norms, etc.)

8. Setting up and using monitoring systems for outcome/impact measurement • Repeated, consistent over time and valid data collections • Avoid the multiplication of specific data collections • Better use of existing statistics and reporting systems • Sharing the information (coordinating bodies) • Consensus building on the information needed, trade offs • Long term thinking

9. Professionalisation of evaluation : necessary not sufficient • Tension between the profession of evaluator and the original discipline(s) • Tension between the craftsman(woman) and the academic • Teams and multidisciplinarity as a solution