Observations from computer-supported Even Swaps experiments using the Smart-Swaps software

1. Observations from computer-supported Even Swaps experiments using the Smart-Swaps software Jyri Mustajoki Raimo P. Hämäläinen Petri Lievonen Systems Analysis Laboratory Helsinki University of Technology www.sal.hut.fi

2. Introduction Even Swaps method • Hammond, Keeney and Raiffa (1998, 1999) • Easy-to-use multi-criteria decision analytical (MCDA) method based on value tradeoffs Web-based Smart-Swaps software • Procedural support to carry out even swaps In this study, observations of students using the method with the help of the software • Do they like and understand it? • How laborious it is felt to be? • Role and importance of the computer support?

3. Structure of this presentation • Introduction to the Even Swaps process • and the Smart-Swaps software • Observations from computer-supported Even Swaps experiments: • Research questions and experimental procedure • Results and evidence • Conclusions and discussion

4. PrOACT-modelSmart Choices (1999) Problem Define your decision problem tosolve the right problem. Clarify what you’re really trying to achieve with your decision. Make smarter choices by creating better alternatives to choose from. Describe how well each alternative meets your objectives. Make tough compromises when you can’t achieve all your objectives at once. Objectives Alternatives Consequences Tradeoffs Smart- Swaps software - this experiment + Uncertainty, Risk profiles, Linked decisions • Introduction to the Even Swaps process • and the Smart-Swaps software

5. Even Swaps elimination process Carry out even swaps that make • Alternatives dominated (attribute-wise) • There is another alternative, which is equal or better than this in every attribute, and better at least in one attribute • Attributes irrelevant • Each alternative has the same value on this attribute • These can be eliminated Process continues until one alternative, i.e. the best one, remains • Introduction to the Even Swaps process • and the Smart-Swaps software

6. Smart-Swaps softwarewww.smart-swaps.hut.fi • Introduction to the Even Swaps process • and the Smart-Swaps software

7. 25 78 Practically dominated by Montana Dominated by Lombard Commute time removed as irrelevant (Slightly better in Monthly Cost, but equal or worse in all other attributes) Example Office selection problem (Hammond et al. 1999) An even swap • Introduction to the Even Swaps process • and the Smart-Swaps software

8. Supporting Even Swaps with Preference Programming Support for • Finding candidates for the next even swap • Identifying practically (i.e. almost) dominated alternatives Both tasks need comprehensive technical screening Idea: supporting the process – not automating it • Introduction to the Even Swaps process • and the Smart-Swaps software

9. Even Swaps Updating of the model Problem initialization Initial statements about the attributes Practical dominance candidates Eliminate dominated alternatives Eliminate irrelevant attributes No More than one remaining alternative Yes Even swap suggestions Make an even swap Trade-off information The most preferred alternative is found Decision support in Smart-Swaps Preference Programming • Introduction to the Even Swaps process • and the Smart-Swaps software

10. Previous studies of Even Swaps Comparison of Even Swaps and MAVT • Belton et al. (2005): “MCDA in E-democracy. Why weight? Comparing Even Swaps and MAVT.”, TED Workshop, May 19-22, Helsinki Environmental planning • Gregory et al. (2001): “Bringing stakeholder values into environmental policy choices: a community-based estuary case study.” Ecological Economics 39, 37-52. Strategy selection in a rural enterprise • Kajanus et al. (2001): “Application of even swaps for strategy selection in a rural enterprise.” Management Decision 39(5), 394-402. Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure

11. Thoughts about Even Swaps process From a cognitive point of view, Even Swaps process has several characteristics that are of a special interest, e.g. • What kind of swaps the DMs tend to carry out? • How the DMs understand the alternatives with the revised consequences? • Does the DM end up with the same result following different paths of even swaps? Smart-Swaps software available • Easy to study how the DMs carry out the process in practice Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure

12. Questions of interest in our experiment In this study, the focus is on: • Issues related to carrying out the process with the Smart-Swaps software or (manually) with Microsoft Excel? • Issues related to the size of the problem? • What are the benefits of using the Preference Programming functionality of Smart-Swaps software, if any? Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure

13. Material and methods • Subjects consisting of engineering studentsas DMs • Controlled experiments included in courses • Assignments in applied mathematics • Decision making and problem solving • Questionnaires with open and scaled questions • Decision process logs Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure

14. Experimental procedure After a brief introduction to the Even Swaps process each DM carried out two decision analytical assignments: • The Even Swaps process on a small introductory problem. On this assignment, • half of the DMs used Excel manually • the other half conducted the process with the Smart-Swaps software • The Even Swaps process on a much larger problem. Every DM used the Smart-Swaps software but • half of the DMs were instructed to ignore the Preference Programming functionality (‘recommender’) • the other half was instructed to utilize it Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure

15. 1st: a small introductory problem You are choosing an apartment. After preliminary elimination there are three alternatives: Lombard, Baranov and Montana. You are in a happy position, as all want you as a tenant. It all boils down to what do you want. You have ended up with four criteria to base your decision on. You gather up a consequences table: Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure

16. 2nd: a large primary problem Your company is about to choose the facility for a new office.After serious thinking you consider eight attributes relevant: a1: size of the office a2: rental costs a3: renovation need a4: car park opportunities a5: means of commuting a6: distance to city center a7: other facilities in the neighborhood a8: habitability After searching for a while you find 12 alternatives to choose from (X1-X12). The consequences are given in a table. Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure

17. Overall sample sizes Small but sufficient for our purposes to get preliminary results. Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure

18. Hypotheses on the experiment H1.1: The DMs using the Smart-Swaps software evaluate the results more positively than the DMs using Excel. H1.2: The DMs using the Smart-Swaps require less time to get the result than the DMs using Excel. H2.1: The support by Smart-Swaps is evaluated to be more useful in large problems than in small ones. H2.2: The DMs tend to eliminate rather dominated alternatives than irrelevant attributes. H3.1: The even swap recommender is evaluated as useful. H3.2: Utilizing the recommender results in fewer swaps. H3.3: Practical dominance propositions are seldom neglected. Observations from computer-supported Even Swaps experiments: Research questions and experimental procedure

19. DMs arrived to various decisions Decisions in the 1st (small) and 2nd (large) problems. Areas indicate frequencies of subjects that chose certain alternative. • Results and evidence

20. H1: Smart-Swaps software vs. Excel? • The DMs using Smart-Swaps were faster • No significant difference between no. of swaps • Note that the problem was small • No significant difference between decisions made • No significant difference between the opinions on the result • Results and evidence

21. Smart-Swaps software vs. Excel: The DMs using Smart-Swaps were faster Decision time comparison in the 1st (small) problem. Areas indicate frequencies of subjects that completed the task in certain time. • Results and evidence

22. Smart-Swaps software vs. Excel: No significant difference between no. of swapsNumber of swaps on the 1st problem. Areas indicate frequencies of subjects that completed the task with certain amount of swaps. • Results and evidence

23. H2: Issues related to the size of the problem • In large problems, it may be very difficult to manually carry out the even swaps process • Smart-Swaps is regarded as helpful in applying the Even Swaps process also in large problems • DMs tend to eliminate more dominated alternatives than irrelevant attributes • Results and evidence

24. Size of the problem: The Even Swaps process is seen to apply best to small problems Areas indicate frequencies of subjects that gave certain answer on Osgood-scale. • Results and evidence

25. Size of the problem: Elimination by dominance was used more than irrelevance Number of all eliminations in the 2nd problem studies are presented in the table. • Results and evidence

26. H3: Benefits of using Smart-Swaps • The DMs using the Preference Programming functionality (the recommender) were faster • The DMs using the recommender made fewer swaps • The recommender was rated high • The DMs using the recommender gave more positive opinions on the method • Results and evidence

27. Benefits of Smart-Swaps: DMs using the proposer were faster Decision time comparison in the 2nd (large) problem. Areas indicate frequencies of subjects that completed the task in certain time. • Results and evidence

28. Benefits of Smart-Swaps: DMs using the proposer made fewer swaps Number of swaps on the 2nd problem. Areas indicate frequencies of subjects that completed the task with certain amount of swaps. • Results and evidence

29. Benefits of Smart-Swaps: Preference programming functionality was rated high Opinions on the recommender functionality of the Smart-Swaps software. Areas indicate frequencies of subjects that gave certain answer on Osgood-scale. • Results and evidence

30. Benefits of Smart-Swaps: DMswithout proposer vs. DMs using proposer Differences in opinions between DMs ignoring the recommender and DMs utilizing it. Areas indicate frequencies of subjects that gave certain answer on Osgood-scale. • Results and evidence

31. Benefits of Smart-Swaps: DMswithout proposer vs. DMs using proposer • Results and evidence

32. Conclusions Our first experiments on the Smart-Swaps software suggest: • The support provided by the Smart-Swaps software is perceived to be useful • Preference Programming approach provided considerable help • A significant difference between the opinions of the group utilizing the recommender and the group instructed to ignore it • Further studies needed to get a deeper insight of how the DMs perceive the approach in practice • In addition, we got a lot of usability feedback to help improve the Smart-Swaps software • Conclusions and discussion

33. References V. Belton, G. Wright and G. Montibeller (2005). “MCDA in E-democracy. Why weight? Comparing Even Swaps and MAVT.” Presentation at the TED Workshop on e-Participation in Environmental Decision Making, May 2005, Helsinki. (Downloadable at http://www.ted.tkk.fi/presentations/Belton-TED.ppt) Hammond, J.S., Keeney, R.L., Raiffa, H., 1998. “Even swaps: A rational method for making trade-offs.” Harvard Business Review 76(2), 137-149. Hammond, J.S., Keeney, R.L., Raiffa, H., 1999. Smart choices. A practical guide to making better decisions. Harvard Business School Press, Boston. Mustajoki, J., Hämäläinen, R.P., 2005. “A Preference Programming Approach to Make the Even Swaps Method Even Easier.” Decision Analysis. (to appear) (Downloadable at www.sal.hut.fi/Publications/pdf-files/mmus04.pdf) Salo, A., Hämäläinen, R.P., 1992. “Preference assessment by imprecise ratio statements.” Operations Research 40(6), 1053-1061. A. Salo and R.P. Hämäläinen (1995). ”Preference programming through approximate ratio comparisons.” European Journal of Operational Research 82(3), 458-475. Applications of Even Swaps: Gregory, R., Wellman, K., 2001. “Bringing stakeholder values into environmental policy choices: a community-based estuary case study.” Ecological Economics 39, 37-52. Kajanus, M., Ahola, J., Kurttila, M., Pesonen, M., 2001. “Application of even swaps for strategy selection in a rural enterprise.” Management Decision 39(5), 394-402.