Can We Avoid Biases in Environmental Decision Analysis ?

1. Can We Avoid Biases in Environmental Decision Analysis ? Raimo P. Hämäläinen Helsinki University of Technology Systems Analysis Laboratory raimo@hut.fi www.paijanne.hut.fi

2. Structure of the presentation • Background & decision analysis interviews • Goals of the study • Case: Regulation of Lake Päijänne • Splitting bias & swapping of levels • Description of the experiment • Results of the experiment • Conclusions ?

3. Environmental decision analysis • Parliamentary nuclear power decision (Hämäläinen et. al) • Decision analysis interviews (Marttunen & Hämäläinen) • Spontaneous decision conferencing in nuclear emergency management (Hämäläinen & Sinkko)

4. Cognitive biases • Splitting bias • attribute receives more weight if it is split • origins: subjects give rank information only (Pöyhönen & Hämäläinen) • Not observable in hierarchical weighting

5. Decision analysis interviews • Opinions of large groups of people traditionally collected through questionnaires • Decision analysis interviews may provide a more reliable way to collect these opinions • Idea: • one value tree for all = common terminology • emphasis on finding the viewpoints of different stakeholder groups • interactive, computer supported

6. Research interest • Existence of biases in a real case • Can biases can be avoided through training and proper instructing ? • Identify what can go wrong in the Lake Päijänne case • Compare the well trained university students’ and spontaneous stakeholders’ responses

7. The Lake Päijänne case • Regulation started 1964 • Main aims were to improve hydroelectricity production and to reduce damages caused by flooding • Environmental values & increase in free time • need for an improved regulation policy

8. Splitting bias • When an attribute is split, the weight it receives increases 0.4 0.1 0.4 0.3 0.3 0.3 0.3 0.3

9. Swapping of levels • Does the order of the levels affect the resulting weights? • Important question in environmental decision analysis: • stakeholder groups may vary regionally • Not studied before

10. Example of swapping of levels Attribute 1 Lake Päijänne Attribute 1 Lake Päijänne Attribute 2 River Kymijoki Attribute 3 Lake Päijänne Attribute 2 River Kymijoki Attribute 1 Lake Päijänne River Kymijoki Attribute 2 Attribute 3 Attribute 3 River Kymijoki

11. Earlier experiments on biases • Structure of the decision model affects the results • Previous experiments typically: • subjects: university students • problems: artificial • results: taken from group averages • Lake Päijänne-case: a real problem with real stakeholders

12. Important new features • Realistic case • Decision analysis interviews instead of passive decision support or survey • Interactive computer support (resulting weights shown immediately) • Instructions and training before the weighting

13. Subjects: • University students attending a course on decision analysis (N = 30) • held during a tutorial session, not mandatory • Habitants of Asikkala (N = 40) • 3 groups of students • 1 group of adults (volunteers) • 3 experts from the Finnish Environment Institute & 2 summer residence owners

14. Experimental setting • Weighting done with the SWING method using a tailored Excel interface • Subjects entered the numbers themselves, two assistants were present to help • Resulting weights shown as bars • Order of value trees partly randomized

15. Sessions • A short introduction to: • Lake Päijänne case • value trees & weighting • different structures of the value tree • In HUT the avoidance of biases was emphasized more • Duration: 60 - 90 minutes

16. SWING method • Easy to use • Attribute ranges clearly presented • Idea: • choose the attribute you would first like to move to its best level • assign it 100 points • assign other attributes points less than 100 in respect to the first attribute

17. Flat-weighting Rantojen käytettävyys Virkistys Virkistyskalastus Kalojen lisääntyminen Ympäristö Lahtien Luonto umpeenkasvu Rantakasvillisuus Vesivoima Vesivoima Tulvat, maatalous ja Muu talous ??? teollisuus Talous Tulvat, loma-asutus Muu talous Vesiliikenne Ammattikalastus

18. Upper level weights: Rantojen käytettävyys Virkistys Virkistyskalastus Kalojen lisääntyminen Ympäristö Lahtien Luonto umpeenkasvu Rantakasvillisuus Vesivoima Vesivoima Tulvat, maatalous ja Muu talous ??? teollisuus Talous Tulvat, loma-asutus Muu talous Vesiliikenne Ammattikalastus

19. ENV5-tree: Rantojen käytettävyys Virkistys Virkistyskalastus Kalojen lisääntyminen Ympäristö Lahtien umpeenkasvu Luonto Rantakasvillisuus Talous

20. ENV2-tree: Virkistys Ympäristö Luonto Talous

21. EC5-tree: Ympäristö Vesivoima Vesivoima Tulvat, maatalous ja Muu talous ??? teollisuus Talous Tulvat, loma-asutus Muu talous Vesiliikenne Ammattikalastus

22. EC2-tree: Ympäristö Vesivoima Vesivoima Muu talous ??? Muu talous ??? Talous Talous Muu talous Muu talous

23. Swapping of levels: Päijänne Tulvavahingot Tulvavahingot Päijänne Muu talous ??? Muu talous ??? Kymijoki ja muut Rantakasvillisuus Päijänne Tulvavahingot Rantakasvillisuus Kymijoki ja muut Kymijoki ja muut Rantakasvillisuus

24. Flat weights vs. upper level weights • Both in group averages and in results of individuals the total weights for the environment and economy were similar with both methods • One explanation: symmetric value tree

25. Splitting bias

26. A typical resident in Asikkala ENVIRONMENT ECONOMY 5 1 5 2 1 1 5 1 1 1 5 2

27. Example from HUT(one of the best ones) ENVIRONMENT ECONOMY 5 1 5 2 1 1 5 1 1 1 5 2

28. Why even weights ? • Some students: none of the attributes seemed to be important • Asikkala: all of the attributes were important even weights for all attributes

29. What caused the bias ? • Similar points for all attributes in one branch regardless of the structure of the value tree

30. Effect of instructions • Students had good instructions • only some had bias in their results • In the spontaneous stakeholders’ sessions the information load was too high and thus the instructions were not adopted as well • nearly all had systematically consistent bias

31. Adjusted / not adjusted weights STUDENTS STAKEHOLDERS

32. Examples STUDENTS STAKEHOLDERS

33. Observation • The students and the experts from FEI could nearly avoid the splitting bias • good background education + instructions did reduce the bias • What did the students think? - Arithmetics or real avoidance of biases

34. Avoiding the splitting bias ? • Good instruction can eliminate it • When the economical attributes were split, the magnitude of the bias was slightly larger • Graphical feedback did not eliminate • Hierarchical weighting

35. Swapping of attribute levels If the order of the levels would not affect the weigts, the pairs of weights should be equal (as in the first picture)

36. Conclusions about swapping of levels ? • Only a few had clearly differing weights with the two trees • No systematic pattern was found • Less differences residents of Asikkala and students than with the splitting bias • A simple scale lead to similar weights with both trees (100, 70 for example) • Neither tree gained clear support

37. Solutions to reduce biases ? • Hierarchical weighting • Models should be tested on real decision makers • Interactiveness of weighting (= possibility to return to change the points given earlier ) • Well balanced trees

38. Other observations in Asikkala • Concept of weight seemed to be difficult for most subjects in Asikkala • Information load was high • Facilitators role becomes important when the DM’s are uncertain

39. Problems related to the Lake Päijänne case • Current regulation policy cannot be improved very significantly • no big differences between the alternatives • unrealistic hopes and false information are probably larger problems than the regulation itself • ‘money is not money’ • strong feelings against the power companies and regulation (shape of value function ?)

40. Suggestions for future research • Hierarchical weighting • Encouragement to reconsider and readjust the statements iterate • Decision Analyst must supervise!

41. References R.P. Hämäläinen, E. Kettunen, M. Marttunen and H. Ehtamo: Evaluating a framework for multi-stakeholder decision support in water resources management, Group Decision and Negotiation, 2001. (to appear) M. Pöyhönen, Hans C.J. Vrolijk and R.P. Hämäläinen: Behavioral and procedural consequences of structural variation in value trees. European Journal of Operational Research, 2001. (to appear) M. Pöyhönen and R.P. Hämäläinen: There is hope in attribute weighting, Journal of Information Systems and Operational Research (INFOR), vol. 38, no. 3, Aug. 2000, pp. 272-282. Abstract R.P. Hämäläinen, M. Lindstedt and K. Sinkko: Multi-attribute risk analysis in nuclear emergency management, Risk Analysis, Vol. 20, No 4, 2000, pp. 455-467. M. Pöyhönen and R.P. Hämäläinen: Notes on the weighting biases in value trees, Journal of Behavioral Decision Making, Vol. 11, 1998, pp. 139-150. Susanna Alaja: Structuring effects in environmental decision models, Helsinki University of Technology, Systems Analysis Laboratory, Theses, 1998.

42. M. Pöyhönen, R.P. Hämäläinen and A. A. Salo: An experiment on the numerical modeling of verbal ratio statements, Journal of Multi-Criteria Decision Analysis, Vol. 6, 1997, pp. 1-10. R.P. Hämäläinen and M. Pöyhönen: On-line group decision support by preference programming in traffic planning, Group Decision and Negotiation, Vol. 5, 1996, pp.485-50. M. Marttunen and R.P. Hämäläinen: Decision analysis interviews in environmental impact assessment, European Journal of Operational Research, Vol. 87, No. 3, 1995, pp. 551-563. R.P. Hämäläinen, A.A. Salo and K. Pöysti: Observations about consensus seeking in a multiple criteria environment, in: Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences, Vol. IV, 1991, IEEE Computer Society Press, Hawaii, pp. 190-198. R.P. Hämäläinen: Computer assisted energy policy analysis in the parliament of Finland, Interfaces, Vol. 18, No. 4, 1988, pp. 12-23. Also in: Case and Readings in Management Science, 2nd edition, M. Render, R.M. Stair Jr. and I. Greenberg (eds.), Allyn & Bacon, Massachusetts 1990 pp. 278-288.