What’s In a Name? Measuring the Impact of the Maclean’s Reputational Survey

1. What’s In a Name? Measuring the Impact of the Maclean’s Reputational Survey Presented at the 2011 Conference of the Canadian Institutional Research and Planning Association Fredericton, NB October 24, 2011 Garry Hansen Director of Institutional Research and Planning St. Thomas University

2. The Maclean’s Reputation Indicator • Reputation has a greater impact on overall rank than any other single indicator • Currently 20% of overall rank • Prone to bias and gaming • Least transparent indicator: no raw data published

3. Topics • The reputational survey instrument and methodology • How to calculate actual pre-reputation indicator scores • How to estimate reputational scores from ordinal ranks • The relationship between reputation and ranking • Actual magnitude of the differences between the rank intervals • Exploring the relationship of reputation and ranking through longitudinal ordinal data

4. Limitations • Not a rigorous statistical analysis (cf. Bastedo and Bowman) • Based solely on the Primarily Undergraduate category. Previous research (cf. Michael and Drewes 2006) has determined that institution size and type have an impact on how rankings affect choice of university.

5. Description of the Survey • 24 University officials at each ranked institution • High school principals • High school guidance counsellors • Heads of a wide variety of national and regional organizations • CEOs • Corporate recruiters In 2010, “more than 11,000” distributed to:

6. Maclean’s Reputational Survey Response Rates

7. Inferring the Impact of University Officials In 2010, 24 mailouts per university X 49 ranked universities = 1176 mailouts Response rate for university officials was 27%, so 318 surveys were returned by university officials. Overall response rate is 7.1% of “more than 11,000” (say, 11,500) so approximately 817 total responses. Responses from university officials make up approximately 39% of all responses.

8. The Survey Instrument • Two surveys of three questions each • National survey • Regional surveys • Atlantic • Quebec • Ontario • Western Provinces • Everyone completes the national survey, but only university officials and high school principals and guidance counselors complete regional survey.

9. In terms of overall quality, please place the following universities in one of four quartiles, circling the appropriate figure. In terms of innovation, both institutional and social, please place the following universities in one of four quartiles. As well, please choose three universities that, in your opinion, are emerging as the leaders of tomorrow. These three should be checked off in the final column.

10. Estimating the Magnitude of the Reputational Score • Ordinal scores obscure actual magnitude of difference • We can estimate a reputational score with reasonable confidence because: • We know the range • We can calculate a pre-reputational score using the raw data provided in the magazine • We know the overall rank and the reputational rank, which serve as tests for an estimated total score composed of the calculated pre-reputational score and the estimated reputational score

11. Calculating the Actual Pre-reputation Score Mean & SD Z-scores Percentages Weighted Scores

12. Step 1: For each indicator, use the raw values provided in the magazine to calculate a mean and standard deviation. Excel formulae AVERAGE() and STDEVP()

14. Step 2: Z-scores Step 2: For each indicator and institution, calculate the z-score by subtracting the mean from the raw value and dividing by the standard deviation. Excel formula: STANDARDIZE()

15. NOTE: Since smaller is better for the Student/Faculty Ratio indicator, reverse the formula (mean – raw score / standard deviation).

16. Step 3: Calculate Percentages Step 3: For each institution and indicator, calculate a percentage from the z-score based on the normal distribution curve. Excel function NORMSDIST()

18. Steps 4 and 5: Weighted Scores Step 4: Calculate the actual indicator score by multiplying the weighted value of the indicator by the percentage calculated in Step 3. Step 5: For each institution, sum the indicator scores to calculate a total pre-reputation score out of 80. (Note: the score for St. Thomas University is actually out of 74 because it is excluded from the Medical/Science Research indicator.)

20. This is what we now know:

21. Estimating Reputational Scores • As a starting point, equally distribute the known range (0-20 points in 2010) across the rank intervals (22 in 2010). • Add these estimated reputation scores to the previously calculated pre-reputation score to produce an estimated overall score. • From these estimated scores, calculate rank for reputation and rank for overall. • Compare the estimated rank with the actual rank in each case. • If the estimated overall rank is not equal to the actual overall rank, add or subtract as necessary from the reputation score estimates • Repeat until all estimated ranks match the actual ranks.

22. Example

23. Exploring the Data

24. Pre-Reputation and Estimated Reputation Scores (Maclean’s 2010)

26. Positive or Neutral Impact (2010)

27. Negative Impact (2010)

28. Is reputation a self-fulfilling prophecy? • If the reputational indicator is truly independent of the rankings, then the reputational indicator should affect the rankings (since it comprises 20% of the overall rank), but the rankings shouldn’t affect the reputational indicator. • Research on U.S. News and World Report and the Times Higher Education Supplement rankings has established that “overall rankings have a causal impact on future peer assessments of reputation” (Bastedo and Bowman 2010).

29. Bastedo and Bowman [I]t is difficult to maintain the fantasy that reputational scores are independent from the rankings themselves. It would take a massive, discontinuous change in academic quality to notably influence reputation scores in any given year. Nearly always, the causal chain is that rankings change in response to shifts in their particular indicators (e.g. faculty-student ratio), and reputations shift in response to rankings. But clearly, rankings drive reputation, not the other way around. (Bastedo and Bowman 2011)

30. The Lag Effect • Possible evidence of overall Maclean’s rank determining reputation found in “lag effect” – after a sudden change in overall rank, reputation changes after a year or more

33. What’s happening here?

34. Final Thoughts • The reputational survey has a disproportionate impact on overall rankings. • The various types of respondents to the reputational survey may have vastly different conceptions of “quality” and “innovation.” Some institutions, by virtue of location and programme offerings, will have a higher profile among certain types of respondents (eg. Corporate or labour market). • There is some evidence of circularity – for some institutions, the reputation score may be largely driven by the very rankings they inform.

35. Questions? Garry Hansen St. Thomas University hansen@stu.ca