June 2008

1. EXPLORING THE HINTERLANDS: MAPPING AN AGENDA FOR INSTITUTIONAL RESEARCH IN THE UK June 2008 Institutional Research Conference 25 – 26 June 2008 “What can institutional research learn from the wider world of market research?” Presented by Steve King, Research Director

2. Controversial? In January 2007, The Times Higher Education Supplement included an article entitled: Mystery callers will shop staff The article begins… Universities are using consultants posing as would-be students to test the 'customer care' skills of academics. Phil Baty reports Groups of fake students and their parents are being unleashed on universities to test how well staff handle "customer relations" it emerged this week. In a move more commonly found in the retail and catering industries, universities are embracing the controversial practice of "mystery shopping" exercises, where consultants are paid to pose as customers to check the level of service given. Some companies even offer to carry out tests using hidden cameras. Throughout a number of industries Mystery Shopping is an established means of examining service delivery and it is certainly not considered controversial within the wider world of Market Research.

3. Why is Mystery Shopping Important? Mystery shopping bridges the gap between perception and reality. How many people here have ever seen research projects where they know that improvements have been made but satisfaction has decreased, or at least not reflected this, or where no changes have been made but satisfaction goes up? Mystery Shopping is an attempt to identify performance in a scientific way without being influenced by the vagaries of human nature. One of the issues that exist over asking people what they think about anything is the amount of background noise that can influence them. “We cannot make a science more exact merely by our wishing it to be so” R.Thomas Malthus

4. As exact as it can be Mystery shopping attempts to deal only in reality. For example: A University has a helpline where students can leave their contact details and be called back (the University has set targets to call back students within half an hour). You could ask a proportion of students who used the service if they were satisfied with the time taken to call them back. But if they say “no” why is this? - it could be they had unrealistic expectations - it could be that they weren’t happy with the advice and down rate anything connected with the helpline as a result. A Mystery Shopper will record the exact time the message was left and the exact time a response was received. The University can then judge whether it is meeting its target (whether it has set the right target in the first place is another matter). The way mystery shopping data is used can be controversial, the practice itself is simply a common sense way of ensuring you are providing the service you want to provide.

5. If something as widely accepted as Mystery Shopping can be seen as controversial this begs the question… What else can institutions learn from other sectors?

6. A Number of Techniques you may (or may not) know I would like to run through a number of very powerful market research techniques that you may or may not have heard of and give some ideas about how these could be applied by academic institutions. • Conjoint analysis (paired and full concept) • Optimum pricing • Brand Price Trade off • Customer Segmentation • Key drivers analysis These are all commonly used techniques in other sectors (and often they are put together in certain combinations, presented as ground breaking packages and given fancy titles!).

7. Conjoint analysis- paired

8. Conjoint Analysis In its simplest terms Conjoint Analysis allows you to examine the relative ‘importance’ or attraction a number of factors or attributes have relative to each other. For example you may be building a new block of student accommodation. Obviously you have space and budget constraints and want to know which factors to include in order to make the accommodation as attractive as possible. You may therefore have five things you are considering including but can only afford to include two of them, the question is which two to include? - Wi-fi capability in all rooms - University network points in all rooms - Air conditioning in all rooms - Satellite TV connections in all rooms - Landline telephone points in all rooms

9. How do you find out which is more important? (I) In order to find out which of the five factors is more important you could just ask! Q1. How important is it that your room has the following?

10. How do you find out which is more important? (II) However, you run the risk of receiving the following response: Q1. How important is it that your room has the following? This is not particularly helpful!

11. How do you find out which is more important? (III) An alternative questioning technique is to ask respondents to rank the the factors in importance from 1st to 5th. Q1. Please rank the following statements from one to five, where one is the most important attribute for your accommodation to have and five is the least important. However, this is something which respondents often struggle with and experience suggests that many will fail to complete it correctly.

12. How do you find out which is more important? (IV) Conjoint analysis allows us to conduct a very complicated trade off analysis using a number of fairly straightforward questions. Each respondent is asked a series of questions in which they chose which, out of a pair of factors, was most important to them. e.g. Each respondent would be asked five questions like this.

13. How do you find out which is more important? (V) The conjoint analysis will produce an overall hierarchy of importance which gives a clear indication of the relative importance of individual factors to respondents (see example below) Hierarchy of Importance Relative Importance CAUTION : DUMMY DATA

14. Conjoint analysis- Full Concept ? + + =

15. Full Concept Conjoint Analysis (I) The example we have just looked at is called ‘paired’ conjoint analysis and is ideal for looking at the relative importance of different discrete factors. A more powerful tool, called full concept conjoint, can also be used to not only look at the relative importance of individual factors, but also the relative importance of different attributes of these factors. For example, you may wish to use full concept conjoint analysis to look at three specific attributes of a course: - Price - Contact hours - Class sizes

16. Full Concept Conjoint Analysis (II) However, within each of these there are a number of levels. For example, price may overall be more important than contact hours but.. how much more appealing is a cost of £3,000 pa compared with £5,000 pa? And is a £5,000 pa course with lots of contact hours and small class sizes more appealing than a £3,000 pa course with limited contact hours and large class sizes? - Price - Contact hours - Class sizes

17. Full Concept Conjoint Analysis (III) In order to look at these issues, a number of different ‘levels’ are attributed to each of the overall factors. e.g.: Price Contact Hours Class Sizes Course costs £2,000 per year Course costs £3,000 per year Course costs £4,000 per year Course costs £5,000 per year 4 hrs spent in tutorials per week 6 hrs spent in tutorials per week 8 hrs spent in tutorials per week 10 hrs spent in tutorials per week Average tutorial size of 5 students Average tutorial size of 10 students Average tutorial size of 15 students Average tutorial size of 20 students

18. Full Concept Conjoint Analysis (IV) Respondents are again asked a number of very straight forward questions which provide exceptionally strong data. Respondents are asked to choose the course that they would be most likely to apply to from three different options e.g. Respondents will be asked a series of such questions and asked to choose their preference from each one. 1 Option 1 2 Option 2 3 Option 3 4 Course costs £2,000 per year 6 hrs spent in tutorials per week Average tutorial size of 20 students Course costs £5,000 per year 10 hrs spent in tutorials per week Average tutorial size of 5 students Course costs £3,000 per year 8 hrs spent in tutorials per week Average tutorial size of 10 students None of these

19. Full Concept Conjoint Analysis (V) Full concept conjoint analysis provides: - An overall hierarchy of importance e.g. price vs class size vs contact hours (similar to that produced by paired conjoint). - An overall hierarchy of importance within each showing the relative strength of different levels e.g. how important is £2,000 pa vs £3,000 pa vs £4,000 pa vs £5,000 pa. - It also allows you to compare scenarios by producing ‘share of preference’ data for specific options.

20. Full Concept Conjoint Analysis (VI) For example, using the three scenarios below, the share of preference may suggest that overall Scenario 2 has the highest appeal (even though price may be the most important factor and is higher than in the other Scenarios): CAUTION : DUMMY DATA Scenario 1 Scenario 2 Scenario 3 Course costs £2,000 per year 6 hrs spent in tutorials per week Average tutorial size of 20 students Share of preference = 15 Course costs £5,000 per year 10 hrs spent in tutorials per week Average tutorial size of 5 students Share of preference = 18 Course costs £3,000 per year 8 hrs spent in tutorials per week Average tutorial size of 10 students Share of preference = 10 You can also put competitor Universities offer into this as a scenario and look to see what you would need to offer to have a greater share of preference!

21. Optimum Pricing

22. Optimum Pricing (I) The price of a course is becoming a greater determining factor in university and course choice, for full-time and part-time students. Optimum pricing is a method used in FMCG research to examine the price consumers will pay for products and it can be applied to University courses. We can ask students at what price they would consider a course to be: ‘Too Cheap’ such they they would question the quality of the course ‘Cheap’ ‘Expensive’ such that they would consider not applying for it’ ‘Too Expensive’ so that they would definitely not apply Optimum pricing can also be conducted using ‘price differentials’ between universities to take into account variances in bursary schemes etc.

23. Optimum Pricing (II) Please move down this list until you get to a price where you would consider the course to be so Cheap that you would question the quality of the course -Select One – More than £5,000 £4,500 £4,000 £3,500 £3,000 £2,500 £2,000 £1,500 £1,000 Less than £1,000 Similar questions would be asked for Cheap Expensive and Too expensive £2,000

24. Optimum Pricing (III) The data provides an optimum price (where % saying cheap crosses % saying expensive): ALL STUDENTS Optimum Price % Course Fee CAUTION : DUMMY DATA

25. Optimum Pricing (IV) The data can be used to provide optimum price ranges for individual courses: £ CAUTION : DUMMY DATA

26. Brand Price Trade Off

27. Brand Price Trade Off (I) Brand Price Trade Off’s are again a research technique used in FMCG which could potentially be applied to Institutional Research (currently we are not aware of it being used in this sector). Brand Price Trade Off (as its name suggests) is a technique that tries to examine that tricky relationship between brand and price. In terms of choosing somewhere to study the Universities brand and the price of the courses are all likely to have an impact on University choice. Respondents in this type of questioning would be shown a number of Universities and annual course costs and asked to choose, all other things being equal, which University they would apply to for their chosen subject. Once they have chosen a University, the price of that University course increases and they are asked to choose again. This continues until the respondent says they would not choose any at this price point or has chosen one of the universities up to the highest price level possible.

28. Brand Price Trade Off (II) At which University would you choose to study History if the annual costs for the courses were as shown below? For example, If the respondent chose Sussex the price for this course would increase and they would be asked the same question: If the respondent then chose Kent, the price for this course would increase and so on and so on:

29. Brand Price Trade Off (III) Brand Price Trade Off is again an extremely strong research technique and provides feedback on the strength of individual brands and their relationship to price. In FMCG these are often used in conjunction with other questions to help build predictive models which, under specified scenarios, look at ‘purchase rates’ (in this instance applications) given various price options etc. It also highlights where ‘steal’ is most likely to come from (a University introducing a new Engineering course might like to include their original Engineering course in such a piece of research in order to examine what the ‘steal’ is likely to be from this course).

30. Customer Segmentation

31. Customer Segmentation (I) Customer Segmentation is a technique used to examine what ‘groups’ of customers exist within a specific universe. Whilst people can be categorised based on basic demographic profiles (e.g. age, sex or social class etc) it is often useful to understand how they differ by attitude or behaviour rather than pigeon-holing based on these broad profiles. From a University’s point of view, it is often the case that your students will be examined by type e.g. Undergrad vs Post Grad, Course studying/ Faculty, Mature vs Other or Home vs Overseas students. But there may well be a proportion of Undergraduates who act and think more like Postgraduates, whilst some overseas students behave more like Home students.

32. Customer Segmentation (II) Customer Segmentation looks beyond these traditional categories to see if there are any better ways of grouping ‘customers’ together. These can be: Current students, Potential applicants or Alumni. Once segments are identified this information can then be used to assist in marketing campaigns, promotional literature or social/ alumni events can be aimed at specific segments. The strength of using attitudinal or behavioural segments is that it allows a university to trigger an emotional response (and does not just try to chase satisfaction).

33. Customer Segmentation (III) Almost any type of questioning can be used to draw up segments and often this can be done retrospectively (using projects that have already been conducted). However, ‘battery of agreement’ statements are often used to help segment respondents. For example, you may wish to segment based on attitudes towards higher education in general using questions such as: Q1. How much do you agree or disagree with the following statements?

34. Customer Segmentation (IV) This kind of research can produce the following types of segments: Vocationally Focused The Unsure Self Improvers Party People CAUTION : DUMMY DATA

35. Customer Segmentation (V) 1: The Unsure These respondents have no strong opinions on education, they tend to go into higher education because it is expected of them. Tend to be aged under 21, have high levels of academic achievement prior to going to university and have other family members who have attended higher education. 2: Party People Most likely to say that universities are about having a good time (or chose for sporting reasons) Tend to be males aged under 21, undergraduates and have fewer than 2 grade A A’levels. 3: Self Improvers Tend to have come to university to improve themselves/ prove that they were capable of doing it. Large proportions of mature and part-time students in Arts or Social Science subjects, tend to be readers of the Independent or Guardian and have environmental concerns. 4: Vocationally Focused Chose to come to university to improve their career prospects/ future earnings. Large proportions of postgraduate and overseas students. Also undergraduates from lower social backgrounds. Tend to be in science subjects (and particularly Engineering). Also Medicine/ Law students from all social classes. These segments can then be examined by other attributes to build up profiles based on demographic or other factors. CAUTION : DUMMY DATA

36. Key Drivers Analysis =

37. Key Drivers Analysis (I) Key Drivers Analysis is a way of examining the relative importance of factors relative to an overall Key Performance Indicator. For example, you may currently run a student satisfaction survey and want to know what is more important to students: the Library service, availability of computing services or sports facilities. Without asking any additional questions (such as conjoint etc.), it is often possible to look at how satisfaction ratings with these factors correlate with overall satisfaction.

38. Key Drivers Analysis (II) Hierarchy of Importance This will produce a relative importance hierarchy (usually indexed on the most important factors). Relative Importance CAUTION : DUMMY DATA This highlights the importance of Library services in terms of driving overall satisfaction with the University (in reality you would include a wide variety of factors into this kind of analysis). This analysis can then be examined by different types of students (or segments) to see if there are any significant differences (e.g. for Male, Home, Undergraduates – sporting facilities could actually be the most important).

39. Key Drivers Analysis (III) There are a number of different statistical techniques (e.g. linear regression etc) which can be used to provide these hierarchies. It is important that once they are produced, however, that they are not looked at in isolation. In order to determine key drivers it is very important to take all other information available into account. It is also important to bear in mind that current performance is a factor in correlation analysis: any attribute on which there is variance in satisfaction is most likely to be able to explain variances in overall satisfaction. Cause and effect cannot always be assumed and hygiene factors can go unnoticed as they only come into play once a threshold has been breached. For example, safety and security is a given until something happens to bring this into question.

40. Conclusions

41. Conclusions In this presentation we have very briefly touched on some of the many ways in which Institutions can use research techniques developed in other sectors to better understand their potential, current and past students. There are many more! Research is not just about asking whether students are satisfied with something and if they are not asking them why not? It can help predict future actions, test potential scenarios and identify groups with specific views or behavioural patterns. It can ensure that your institution offers students what they want and that it does so within a pricing framework that remains competitive. There is no one research technique that can answer all of your questions, but an established programme of research can go a long way in helping to focus policy making and investment to maximum benefit for the institution itself.

42. Thank you for listening Any questions? For further information please contact Steve King steve.king@tora.co.uk 5 Kings Meadow Ferry Hinksey Rd Oxford OX2 0DP (01865) 728272