PD Evaluation Methodology Reference Group Workshop

1. Phase 2 Evaluation of the PD Evaluation Methodology Reference Group Workshop/Meeting 11 – 13 February 2009

2. Methodological Issues The broad choice of evaluation methodology, as we discussed yesterday, follows on from the evaluation questions chosen. However there are more specific methods-related issues that need to be thought about at this stage.

3. Methodological Issues One way of focussing on methodology is to consider what methodological ‘standards’ or ‘qualities’ we expect this evaluation to have. (You will notice a certain overlap with the previous workshop on ‘quality standards’!)

4. Methodological Issues For example, it can be argued that this evaluation should end up with the following qualities: • A balanced & sufficient sample of countries • Sufficient coverage of sectors and themes • Information of good quality • Offering the possibility of explanation and attribution

5. Methodological Issues If an evaluation has these qualities, it will allow us to generalise to some extent across different PD settings (‘external validity’); be confident that our measurements or descriptions are consistent (‘reliability’) and have confidence in the strength or ‘power’ of findings – that they are sufficiently supported by the evidence collected and analysed

6. Methodological Issues Making the right decisions about methodology at the beginning will make an evaluation defensible, able to withstand criticism when it eventually reports Lets take each of the ‘attributes’ in turn

7. Methodological Issues ‘A balanced & sufficient sample of countries’ When policy makers ask about the PD they want to know whether we are able to say something about countries in different geographical regions; those that are more and less aid dependent; and both those that have strong institutions and governance and others that have some elements of ‘fragility’ perhaps because they are still recovering from wars or conflicts. We also need enough countries to be able to support conclusions for the PD as a whole

8. Methodological Issues ‘Sufficient coverage of sectors and themes’ Countries selected need to cover the main policy areas and sectors recognised as important for development – for example healthcare, encouraging small businesses, education, progress towards the MDGs, international trade support. They also need to include important themes such as: capacity development, civil society participation, donor harmonisation, improving governance & reducing fragility. This will allow sensible comparisons to be made across cases

9. Methodological Issues ‘Information of good quality’ This requires: • Available information – one rationale for selecting some sectors/themes • Willingness to use innovative data sources • Ensuring that all pre-existing sources are ‘synthesised’, reviewed and exploited • Cross-checking (‘triangulating’) across multiple sources of information • Expending more time relatively on data collection

10. Methodological Issues ‘Offers the possibility of explanation & attribution’ There are two ‘classic’ ways we can attempt to explain. First through longitudinal analyses that follow a causal chain over time – this is the basis of time-series data and panel studies as well as causal modelling, tracker-studies & ‘theory-based evaluations’. Second we can compare across places, settings or time periods – including before & after studies; comparison groups; quasi experiments and full controlled experiments

11. Methodological Issues On this basis, we can put together a possible starting list of methods that could be used in Phase 2. The list would include:

12. Methodological Issues • Synthesis reviews of existing evaluations, research and indicator systems • Comparative in-depth case studies (of country partnerships) which are chosen to contain a good cross-section of common themes/sectors • Longitudinal studies – either forward looking (‘theory-based’ mapping of plausible directions of travel) or backward looking tracking back to PD-like, longer established policies • Targeted comparative studies to ‘supplement’ country based case comparisons

13. Methodological Issues Methodologies have to be understood as more than analytic tools. How they are resourced and ‘steered’ will determine their value as much as their technical sophistication. To take two examples:

14. Methodological Issues Putting together the best team of experts to undertake the evaluation at country & central levels will be challenging. It may require bringing together public sector and civil society expertise; national and possibly regional resources & skills.

15. Methodological Issues National ‘reference groups’ will need to open up access for information and cooperation institutionally and across government; to safeguard the independence and credibility of the evaluation; & build bridges so as to make it more likely that evaluation outputs will be used and useful

16. Methodological Issues These are some of the issues that needs more discussion before the Terms of Reference for this evaluation are prepared – and will therefore be taken up in the group discussion session that follows ………..after any points of clarification