Rank Robustness of Composite Indices: Dominance and Ambiguity

1. Rank Robustness of Composite Indices: Dominance and Ambiguity James E. Foster George Washington University & Oxford Mark McGillivray Ausaid Suman Seth Vanderbilt University

2. Composite Indices Many multidimensional indices take the form C(x;w) = w·x where x = (x1,x2,…,xD) is a vector of dimensional achievements and w = (w1,w2,…,wD) is a vector of weights satisfying w1,…,wD 0 and w1+···+wD = 1 “Composite index”

3. Examples • Human Development Index or HDI (UNDP) • Multidimensional Povery Index Mα (Alkire/Foster) • Gender Empowerment Index (UNDP) • Global Peace Index (Inst. Econ Peace) • Environmental Sustainability Index (Yale) • Child Well-being Index (Inst. Child Dev.) • US News College Ranking Why this Form? • Natural • Easy to understand • Statistical properties Key challenge…how to choose weights?

4. Choice of Weights Methods • Normative • Statistical e.g. Principal component analysis • Equal weights “Our argument for equal index weights is based on the premise that no objective mechanism exists to determine the relative importance of the different aspects of environmental sustainability.” Environmental Sustainability Index “…we have no reliable basis for doing [otherwise]” Mayer and Jencks, 1989 Note: Analogous to “Principle of insufficient reason”

5. Choice of Weights Note Inevitable arbitrariness in choice of weights Arrow’s critique of HDI’s weights w = (1/3,1/3,1/3) Why important? Comparison could be ambiguous or robust

6. Example: Ambiguous Comparison C(x;w0) > C(y;w0) at initial w0 and yet C(x;w) < C(y;w) at some other reasonable w “Ambiguous” w0 = (1/3, 1/3, 1/3) w = (1/2, 1/4, 1/4)

7. Example: Robust Comparison C(x;w0) > C(y;w0) at initial w0 and C(x;w) > C(y;w) at all reasonable w “Robust” Same ranking for all w

8. How to discern? Clearly, Ranking C(x;w0) > C(y;w0) reveals nothing about the robustness or ambiguity of the comparison The following look the same – but are different Goal: Provide intuitive criteria for distinguishing between robust and ambiguous comparisons

9. Our Approach Definitions DSimplex of dimension D w0Initial weighting vector WSet of “reasonable” weighting vectors around w0 W is non-empty (1,0,0) w0 (0,1,0) (0,0,1)

10. Our Approach • NotationFor any a, b in D, ab denotes adbd for all d; a>b denotes adbd with a ≠ b for all d; and a >> b denotes ad > bd for all d • X  D: non-empty set of alternatives to be ranked • Define weak robustness relation RW on X by x RW y if and only if C(x;w)  C(y;w) for all w W • Characterize RW among all binary relation R on X • Based on Bewley (1986)

11. Our ApproachCharacterization of RW • Axioms of R • Quasiordering (Q): R is transitive and reflexive • Monotonicity (M): (i) If x > y then x R y; (ii) if x>> y then y R x cannot hold. • Independence (I): Let x, y, z, y', z'  X where y' = x +(1–)y and z' = x + (1–)y' for 0 <  < 1. Then y R z if and only if y' R z'. • Continuity (C): The sets {x  X | x R z} and {x  X | z R x} are closed for all z  X.

12. Our ApproachCharacterization of RW • Theorem 1Suppose that X is closed, convex, and has a nonempty interior. Then a binary relation R on X satisfies axioms Q, M, I, and C if and only if there exist a non-empty, closed, and convex set W such that R = RW Interesting interpretationMaximin criterion of Gilboa and Schmeidler (1989) for multiple priors

13. Our Approach Robustness of ranking x CW y if and only if C(x;w0) > C(y;w0) and x RWy Analogous constructs Knightian uncertainty (Bewley, 1986) Partial comparability (Sen, 1970) Poverty ordering (Foster-Shorrocks, 1988) Q/ Which W? A/ We use nested sets De where e[0,1] Epsilon-Contamination model of ambiguity (Ellsburg, 1961)

14. The C0 Relation Suppose W = {w0} = D0 Denote resulting relation by C0 so that x C0 y if and only if C(x;w0) > C(y;w0) Original complete ordering of the composite index Interpretation of D0 Supremely confident in choice of initial weightings - offers no robustness test at all Implausible

15. HDI Top Ten According to C0 Human Development Report 2006, UNDP

16. Complete Ordering Column Dominates Row

17. The C1 Relation Suppose W = S = D1 Denote resulting ranking by C1 so that x C1 y if and only if C(x;w0) > C(y;w0) and x R1y where x R1y denotes C(x;w)  C(y;w) for all w  S Interpretation of D1 No confidence in choice of initial weightings – offers full robustness test Stringent requirement

18. Characterization of C1 e1 = (1,0,0) Simplex D Let C(x;ei) = xi; i = 1,2,3 w0 e2 = (0,1,0) (0,0,1) = e3 Theorem 2Let x, y  X. Then (i) x R1 y if and only if x ≥ y and (ii) x C1 y if and only if x > y

19. Example: Robust Comparison Aus HDI : 0.957 Aus x1 : 0.925 Aus x3 : 0.954 Swe HDI : 0.951 Aus x2 : 0.993 Swe x3: 0.949 Swe x1 : 0.922 Aus C1Swe Swe x2 : 0.982 x C1 y if and only if x > y e1 = (1,0,0) e3= (0,0,1) w0 e2 = (0,1,0)

20. Example: Ambiguous Comparison Ire x3 : 0.995 Can x1 : 0.919 Ire x1 : 0.882 Ire HDI : 0.956 Can x3: 0.959 Ire x2 : 0.990 Can HDI : 0.950 Ireland Canada ranking is not fully robust. e1 = (1,0,0) Can x2 : 0.970 Can e1 = (1,0,0) Ire e3= (0,0,1) w0 w0 e2 = (0,1,0) e2 = (0,1,0) (0,0,1) = e3

21. Recall C0 Ranking Column Dominates Row

22. Fully Robust Ranking C1 Column Dominates Row

23. Partial Ordering Ce Consider any e satisfying 0 e 1 Define • De = {w' S : w' = (1 – e)w0 + ew for some w  S} • Interpretation • 1 – e degree of confidence in initial weighting w0 • Ellsburg (1961) • e “size” of resulting set for checking robustness • Ex • D1 = Dlowest degree of confidence in w0, largest set • D0 = {w0} highest degree of confidence, smallest set • De = intermediate

24. Partial Ordering Ce e1 1-e (1-e)w0 + ee1 = ve1 e De w0

25. Partial Ordering Ce Suppose W = De Denote resulting ranking by Ce so that x Ce y if and only if C(x;w0) > C(y;w0) and x Rey where x Rey denotes C(x;w)  C(y;w) for all w De e1 De w0 e2 e3

26. Characterization of Ce Denote ved = (1- e)w0 + eed xed = vedx Value of x at ved xe = (xe1, …,xeD) Vector of these values e1 ve1 De ve2 w0 ve3 e2 e3

27. Characterization of Ce Theorem 3Let x, y X. Then (i) x R y if and only if x ≥ y and (ii) x C y if and only if x > y e1 ve1 De ve2 w0 ve3 e2 e3

28. Partial Ordering Ce Recall Canada/Ireland example e1= (1,0,0) Can Ire e2= (0,1,0) e3= (0,0,1)

29. Recall Fully Robust Ranking C1 Column Dominates Row

30. Ce for e = 1/4 Column Dominates Row

31. Measure of Robustness Idea How robust is a given comparison? Denote A = C(x,w0) - C(y,w0) difference in HDI’s B = maxwS{C(y,w) - C(x,w)} max dimensional departure Define r = A/(A+B) Measure of robustness

32. Measure of Robustness Theorem4Suppose that x C0 y forx, y  X and let rbe the robustness level associated with this comparison. Then the -robustness relation x C y holds if and only if  ≤ r.

33. Measure of Robustness e1= (1,0,0) Can Ire e2= (0,1,0) e3= (0,0,1) Size of the largest sub-simplex is the measuure of robustness r

34. Robustness Calculation A = 0.006 B = 0.037 r = 0.006/(0.006+ 0.037) = 0.139

35. Robustness Calculation • A = 0.006 • B = - 0.003 • r = 1 Note that both comparisons have same A Yet very different robustness levels!

36. Measure of Robustness (%) Column Dominates Row

37. Conclusion • Propose a tool for measuring robustness of ranking • The idea is motivated by partial orderings, epsilon-contamination, and Knightian uncertainty • The tool can be extended to the case of general means (e.g., HPI) • A measure of robusteness is proposed • Robustness Vs. Redundancy (McGillivray 1991)

38. Thank you