Formulating Indicators for Assessing Scenarios

MRC-MDBC STRATEGIC LIAISON PROGRAMBASIN DEVELOPMENT PLANNINGTRAINING MODULE 3SCENARIO-BASED PLANNING for the MEKONG BASINNapakuang, Lao PDR8-11 December 2003 Formulating Indicators for Assessing Scenarios

DEVELOPING INDICATORSA process for this part of basin planning

DEVELOPING INDICATORSSummary of steps needed to prepare for assessment of scenarios Derive BDP Development Objectives (derived from 95 Agreement) Select Assessment Criteria (Tests of how well scenarios will meet objectives) Select Indicators (Measurements to be used in assessment tests)

DEVELOPING INDICATORSStep 1 - Defining development objectives • First and foremost: Objectives must be derived from the Basin Agreement. This is the mandate of the Mekong River Commission. • But the objectives in the Agreement are broad and not easy to quantify. • An agreed set of more precise and/or detailed BDPdevelopment objectives needs to be developed through consultative processes. • These derived development objectives should cover all aspects of each of the objectives (whether explicit or implicit) of the Agreement.

DEVELOPING INDICATORSExample of BDP Development Objective Start point: Chap III of Agreement - Objectives and Principles of Cooperation. For example: Article 3. Protection of the Environment and Ecological Balance Suggested BDP Development Objectives derived from Article 3 could include: • “To achieve and maintain acceptable water quality in the Mekong River” • “To maintain fisheries production.” and so on...

DEVELOPING INDICATORSStep 2 - Select Assessment Criteria • Assessment criteria are the tests (for achievement of objectives) to be applied to the anticipated outcome of development scenarios. • At least one such criterion (or test) is needed per BDP Development Objective. • The tests (criteria) should preferably be capable of being measured in a quantifiable manner and linked to water flow regime. • Selection of the criteria should be done in a consultative way - especially with relevant experts (MRC programs + expert line agencies)

DEVELOPING INDICATORSExample of Assessment Criteria For the objective “To maintain fisheries production” some criteria could be: • Changes in dry season flow regime • Changes in wet season flood flows • Changes in selected flooded areas • Changes in connectivity for fish migration These are examples of factors that support or affect fisheries production. Note that they have all been related back to flow. Expert advice is needed for this.

DEVELOPING INDICATORSStep 3 - Selection of Indicators • Indicators are the specific measurements to be used as part of the assessment tests. • At least one indicator is needed per criterion. • To the maximum extent possible the indicators should be capable of being generated by the DSF - either as direct numerical output or as spatial data for interpretation. • Again, the indicators should be selected in a consultative way - especially with relevant experts (MRC programs + expert line agencies) - and with modellers.

DEVELOPING INDICATORSExamples of Indicators For the fisheries maintenance assessment criterion “Changes in wet season flood flows”, possible indicators are: • Annual flood frequency (of selected flood sizes) • Rate of rise of wet season flood • Rate of fall of wet season flood Note these are all numerical indicators and can be computed through DSF simulations.

DEVELOPING INDICATORSAssessment - It is done by TEAMWORK Derive BDP Development Objectives (derived from 95 Agreement) Select Assessment Criteria (Tests of how well scenarios will meet objectives) Select Indicators (Measurements to be used in assessment tests) BDP Team (MRCS + NMCs) BDP Team + Experts BDP Team + Experts + Modellers

DEVELOPING INDICATORSSome ideas for shortcuts • MRC has done a lot of work already. • Capitalise on it! For example: • Transboundary issues are thoroughly identified and listed. Good start point to derive indicators. • WUP has already prepared a comprehensive list of suggested indicators. See DFR 640 Technical Reference Report: Impact Analysis Tools. Start with these. • Vital to take a structured approach, with all steps documented.

DEVELOPING INDICATORSBuilding on work already done Transb’d’y Issues 1995 Agreement Agreed Indicators WUP suggested indicators Anything else you can find

DEVELOPING INDICATORSA Structured Format Any format will do. Important thing is to have one.

DEVELOPING INDICATORSFinding out if the indicators work • The only method is to actually apply the indicators to some scenario outcomes generated by the DSF, or otherwise. Revise them and repeat. An adaptive, iterative process. Ask the questions: • Do we have the minimum set of indicators? • Are any redundant? • Are any not sensitive enough to show variations? - even when we ‘know’ there must be change. • Have we missed any vital indicators? (Go back to experts with results and check that they are satisfied their issues are being assessed.)

MRC-MDBC STRATEGIC LIAISON PROGRAMBASIN DEVELOPMENT PLANNINGTRAINING MODULE 3SCENARIO-BASED PLANNING for the MEKONG BASINNapakuang, Lao PDR8-11 December 2003 Multi-criteria Analysis for Ranking Development Scenarios

MULTI-CRITERIA ANALYSISScenario-Based Planning - How do we do this bit?

MULTI-CRITERIA ANALYSISThe need for MCA • The ideal situation is to be able to measure the outcomes of alternative developments in the same units such as dollars. • Comparison is then a simple numerical exercise. • And in such a situation it doe not matter that for any particular scenario it may meet some criteria extremely well, and others extremely poorly. • The alternative developments can simply be optimised to maximise net dollar outcomes. • However, work already done on outcome indicators shows that this ideal will rarely occur in practice.

MULTI-CRITERIA ANALYSISExample of simulation results Which scenario is best?

MULTI-CRITERIA ANALYSISSome questions • Are all criteria of equal importance? • Who says so? Who are the decision makers? • Should the evaluation be passed over to the community or kept for ‘professionals’ to do? • Some indicators do not show much change between scenarios - should they be dropped from the analysis? • Is there some other indicator(s) that might be of more help in the scenario evaluation? • etc

MULTI-CRITERIA ANALYSISSome answers • Experience shows that stakeholders need to be involved if the evaluation is to have credibility. (At basin level this can be high level and broad) • A structured process is the only successful approach to dealing with complexity. • At the same time, every effort should be made to have the absolute minimum complexity. • There are a number of recognised methodologies and proprietary MCA software available. • However, it is important to remember that the decisions will be socio-political, not technical.

MULTI-CRITERIA ANALYSISSome Structured Approaches • Criteria weighting. This requires joint discussions by stakeholders and ‘experts’ to agree on weighting. Numerical techniques can then be used to score the scenarios. • DELPHI - a way of combining expert/informed opinions (not especially useful in scenario evaluation, but can help develop weightings) • AHP (Analytical Hierarchy Process) - one of many pair-wise preference processes - plenty of software available. • etc

MULTI-CRITERIA ANALYSISWhich is the best approach? • In basin planning, MDBC experience is that the outcomes of any unclear or partially invisible process will frequently be rejected by the stakeholders (and community). • MDBC and its member States tend not to use MCA software - both for the above reason, and because none seems to work very well for the complex systems of basin natural resources. (Research into MCA continues however.) • Instead, the common approach is to use stakeholder reference panels and an independent facilitator to work towards consensus.

Evaluating Development ChoicesScenarios, Indicators , MCA and all that. Thank you GOOD LUCK! Questions?

Formulating Indicators for Assessing Scenarios