Rank-Based Sensitivity Analysis of Multiattribute Value Models

1. Rank-Based Sensitivity Analysis of Multiattribute Value Models Antti Punkka and Ahti Salo Systems Analysis Laboratory Helsinki University of Technology P.O. Box 1100, 02015 TKK, Finland http://www.sal.tkk.fi/ forename.surname@tkk.fi

2. Additive Multiattribute Value Model • Provides a complete rank-ordering for the alternatives • Selection of the best alternative • Rank-ordering of e.g. universities (Liu and Cheng 2005) or graduate programs (Keeney et al. 2006) • Prioritization of project proposals or innovation ideas (e.g. Könnölä et al. 2007) • Methods for global sensitivity analysis on weights and scores • Focus only on the selection of the best alternative • Ex post: Sensitivity of the decision recommendation to parameter variation • Ex ante: Computation of viable decision candidates subject to incomplete information about the parameter values (e.g., Rios Insua and French 1991, Butler et al. 1997, Mustajoki et al. 2006)

3. Sensitivity Analysis of Rankings • Consider the full rank-ordering instead of the most preferred alternative • How ’sensitive’ is the rank-ordering • How to compare two rank-orderings? How to communicate differences? • We compute the attainable rankings for each alternative subject to global variation in weights and scores • How sensitive is the ranking of an alternative subject to parameter variation? • Is the ranking of university X sensitive to the attribute weights applied? • What is the best / worst attainable ranking of project proposal Y?

4. Incomplete Information • Model parameter uncertainty before computation • Relax complete specification of parameters • ”Error coefficients” on the statements, e.g. weight ratios • E.g. Mustajoki et al. (2006) • Directly elicit and apply incomplete information • Incompletely defined weight ratios: 2 ≤ w3/ w2 ≤ 3 • Ordinal information about weights: w1≤ w3 • Score intervals: 0.4≤ v1(x12) ≤ 0.6 • E.g., Kirkwood and Sarin (1985), Salo and Hämäläinen (1992), Liesiö et al. (2007) • Set of feasible weights and scores (S)

5. Attainable Rankings • Existing output concepts of ex ante sensitivity analysis do not consider the full rank-ordering of alternative set X • Value intervals focus on 1 alternative at a time • Dominance relations are essentially pairwise comparisons • Potential optimality focuses on the ranking 1 • Alternative xk can attain ranking r, if exists feasible parameters such that the number of alternatives with higher value is r-1

6. ranking 1 is attainable for x3 ranking 4 is attainable for x1 w1 ranking 1 is attainable for x2 ranking 3 is attainable for x3 w2 Attainable rankings Attainable Rankings: Example • 2 attributes, 4 alternatives with fixed scores, w1 [0.4, 0.7] V x1 x2 x3 x4 0.4 0.7 0.6 0.3

7. Computation of Attainable Rankings • Application of incomplete information  set of feasible weights and scores (S) • If S is convex, all rankings between the best and the worst attainable rankings are attainable • Best ranking of xk: • Worst ranking of xk: • MILP model to obtain the best / worst ranking of each xk • V(x) expressed in non-normalized form (linear in w and v) • # of binary variables = |X| - 1

8. Example: Shangai Rank-Ordering of Universities • Shanghai Jiao Tong University ranks the world universities annually • Example data from 2007 • http://ed.sjtu.edu.cn/ranking2007.htm • 508 universities • Additive model for rank-ordering of the universities

9. Attributes Table adopted from http://ed.sjtu.edu.cn/ranking2007.htm

10. Data

11. Sensitivity Analysis • How sensitive are the rankings to weight variation? • What if different weights were applied? • Relax point estimate weighting 1. Relative intervals around the point estimates • E.g. =20 %, wi*=0.20: 2. Incomplete ordinal information • Attributes with wi*=0.20 cannot be less important than those with wi*=0.10 • All weights lower-bounded by 0.02

12. Unsensitive rankings ”Different weighting would likely yield a better ranking” Results: Rank-Sensitivity of Top Universities exact weights 20 % interval 30 % interval University incompl. ordinal no information 10th 442nd Ranking

13. Conclusion • A model to compute attainable rankings • Sufficiently efficient even with hundreds of alternatives and several attributes • Attainable rankings communicate sensitivity of rank-orderings • Conceptually easy to understand • Holistic view of global sensitivity at a glance independently of the # of attributes • Applicable output in Preference Programming framework • Additional information leads to fewer attainable rankings • Connections to project prioritization • Initial screening of project proposals for e.g. portfolio-level analysis • Supports identification of ’clear decisions’ (cf. Liesiö et al. 2007) • ”Select the ones ’surely’ in top 50” • ”Discard the ones ’surely’ outside top 50”

14. References • Butler, J., Jia, J., Dyer, J. (1997). Simulation Techniques for the Sensitivity Analysis of Multi-Criteria Decision Models. EJOR 103, 531-546. • Keeney, R.L., See, K.E., von Winterfeldt, D. (2006). Evaluating Academic Programs: With Applications to U.S. Graduate Decision Science Programs. Oper. Res. 54, 813-828. • Kirkwood, G., Sarin R. (1985). Ranking with Partial Information: A Method and an Application. Oper. Res. 33, 38-48 • Könnölä, T., Brummer, V., Salo A. (2007). Diversity in Foresight: Insights from the Fostering of Innovation Ideas. Technologial Forecasting & Social Change 74, 608-626. • Liesiö, J., Mild, P., Salo, A., (2007). Preference Programming for Robust Portfolio Modeling and Project Selection. EJOR 181, 1488-1505. • Liu, N.C., Cheng, Y. (2005). The Academic Ranking of World Universities. Higher Education in Europe 30, 127-136 • Mustajoki, J., Hämäläinen, R.P., Lindstedt, M.R.K. (2006). Using intervals for Global Sensitivity and Worst Case Analyses in Multiattribute Value Trees. EJOR 174, 278-292. • Rios Insua, D., French, S. (1991). A Framework for Sensitivity Analysis in Discrete Multi-Objective Decision-Making. EJOR 54, 176-190. • Salo, A., Hämäläinen R.P. (1992). Preference assessment by imprecise ratio statements. Oper. Res. 40, 1053-1061.