Efficiency and Sensitivity Analyses in the Evaluation of University Departments

1. Efficiency and Sensitivity Analyses in the Evaluation of University Departments Ahti Salo and Antti Punkka Helsinki University of Technology Systems Analysis Laboratory02015 TKK, Finland firstname.lastname@tkk.fi

2. Context • National efforts to increase the efficiency of universities • ”Productivity programme” ~1500 positions to be slashed in 2007-10 • Efficiency studies commissioned by the Ministry of Finance • ”Measurable Productivity in Universities” by Gov’t Econ. Research Cntr in 09/06 • Developments at Helsinki University of Technology (TKK) • Rector asked us to produce comments to the above report • TKK has been using various resource allocations models over the years • Considerable dissatisfaction with many of these models • Resources Committee requested to develop different principles • Tasks • Develop value efficiency models in support of resource allocation • Explore methodological extensions in view of decision making needs

4. Efficiency of University Departments • Departments consume inputs in order to produce outputs • Valuation of inputs and outputs involves subjective preferences y1 (Master’s Theses) University / Department x1 (Budget funding) y2 (Doctor’s Theses) x2 (Project funding) y3 (Int’l publications)

5. Data Envelopment Analysis (Charnes et al., 1978) • Approach • Multiple inputs xi and outputs yi of decision making units (DMU) aggregated by non-negative multipliers (’weights’) • Efficiency ratio of each DMU is maximed, subject to the condition that this ratio does not exceed one for any DMUs • Observations • Extending the set of inputs cannot worsen the efficiency of any DMU • In Value Efficiency Analysis, the DMs’ preferences are explicitly modelled (VEA, Halme et al., 1999; Korhonen and Syrjänen, 1998)

6. Valuation of Inputs and Outputs • Preferences elicited from the Resources Committee • How valuable are the different outputs in relative terms? • What is the value of an MSc degree relative to a PhD degree etc? • 44 outputs from the reporting system using 3-year annual averages • Degrees granted – publications activity – international activities • Mitigation of impacts due to large annual fluctuations • How important are  budget funding and  project funding in terms of producing this output?

9. Feasible Valuations and Efficiencies • Feasible valuations • Responses by individual respondents plus convex combinations thereof • Efficient departments (efficiency = 1) • For some feasible valuation of inputs and outputs, the efficiency ratio of a this Dept is either greater than or equal to that of all other Depts • Inefficient deparments (efficiency < 1) • For all feasible valuations, the efficiency ratio of some other Dept is strictly greater • If the aim is  to maximize overall efficiency and  Depts increase their outputs in proportion to the use of inputs, resources should be shifted from inefficient to efficient Depts

10. Efficiencies of departments • Very significant differences in departmental efficiencies • Results still in alignment with resource allocation models

11. Motivations for Methodological Extensions • Results of Value Efficiency Analysis may not be robust • Introduction of an outlier may produce radical changes in efficiency results • Hence the results may appear counterintuitive to DMs • Pairwise dominances among DMUs • It may be of interest to enable comparisons among all DMUs • Efficient DMUs need not be greatest relevance for very inefficient DMUs • Rank-based information about relative efficiencies • Ranking lists (e.g., Shanghai Jian Tao University) have been influential • Yet this list (and many others) do not account for the value of inputs • Hence the interest to examine efficiencies in terms of rankings, too

12. Pairwise Efficiency Dominance of DMUs • For any feasible input and output valuations , the efficiency of DMU s is defined as • Definition: If DMU s and DMU t are such that for all feasible input and output valuations (with strict inequality for some feasible valuations), then DMU sdominates DMU t.

13. Pairwise Efficiency Dominance of DMUs • Definition: If the efficiency ratio of DMU s is greater than or equal to that of DMU t for all , (with strict inequality for some feasible valuations), then DMU sdominates DMU t. • This dominance holds if the minimum is positive

14. DMU t DMU s Pairwise Dominance • This minimization problem gives a lower bound on how much more efficient DMU s is incomparison with DMU t

15. Ranking of DMUs’ Efficiencies • Definition: Let ,  be a feasible valuation. The ranking of DMU t among DMUs S is  The ranking of the most efficient DMU is 1  If several DMUs have the same efficiency ratio, they have a tie with the same ranking • Different feasible valuations assign different rankings to DMUs • Best and worst possible of rankings computed with an MILP model

16. Ranking Ranges of Rankings for TKK Departments

17. Ranking List of Shanghai Jia Tong-University

18. Weight Sensitivity of Rankings

19. Conclusions • Lessons learned • Different models complement each other • Thinking about the value of intangibles is useful - does our data matter? • Efficiency analysis alone does not suggest strategic changes • Useful methodological extensions • Inter-departmental comparisons supported by pairwise comparisons • Ranges of rankings show sensitivities in the relative efficiency of Depts • Possible extensions • Development of analyses to account for intermediate inputs/outputs • Explicit linkages to resource allocation through goal-setting • Interactive decision support tools with Internet-based user interfaces