Moving into the mainstream: Researching the institutional introduction of EVS, from the realm of the enthusiasts to supp

1. Moving into the mainstream: Researching the institutional introduction of EVS, from the realm of the enthusiasts to supporting the later adopters Marija Cubric Business School Amanda Jefferies School of Computer Science www.herts.ac.uk

2. Contents Background Literature review Methodology Preliminary findings Focus on staff views on technology adoption and use of EVS Conclusions

3. Background Major institutional investment by the university in 2010-11 (Assessment and Feedback project) • 3,845 students form 8 academic schools issued with a personal handset from Jan 2011 JISC EEVS project (stand B) , alongside iTEAM Strand A , started in September 2011 • EEVS = Evaluating Electronic Voting Systems for Enhancing Student Experience EEVS Objectives • To provide an up-to-date view of the student and staff experiences of EVS • To identifya set of critical success factors for introducing and maintaining the use of EVS in support of an institutional assessment and feedback strategy. • Focus on large-scale deployment and differences across multiple disciplines.

4. Literature Review: Use of EVS in HE Survey of literature on the introduction of EVS in HE • 300+ papers on published between 1998and 2011 • A wide variety of writers : • Brown, Davis, Draper, d’Inverno, Kennedy, Nichol, Oliver, White et al • Our own early adopters :- Lorimer, Hilliard, Thornton and Willis (University of Hertfordshire) Most authors reported on local use in classes: • to engage students, • to encourage interactivity, • to support constructivist approaches to teaching and learning in line with e.g. Chickeringand Gamson’s guidelines. Lack of literature on institutional deployment, with the exception of: • Twetten et al (2007) Successful Clicker Standardisation, Educause Quarterly 4 • Experience from EVS standardisation in four North American universities

5. Methodology: Data Exploratory pragmatic approach: • Gaining understanding of the meanings from various stakeholders • Focus on practical applied research, integrating different perspectives to help interpreting data (Sounderset al 2009) Mixed-method data collection: • Qualitative data: students’ blogs, staff interviews, staff reports • Quantitative data: students’ questionnaire, staff questionnaire Time horizon: Mainly cross-sectional (a ‘snapshot’ taken in 2011/12) with some elements of longitudinal study (students’ blogs over a period of 2-4 weeks). Sampling: • Self-selection (voluntary student participants for questionnaires) • Self-selection & purposive (voluntary student participants for blogs; further selection made to enable balanced representation across different schools) • Purposive (heterogeneous): staff involved with or leading EVS adoption in different Schools

6. Methodology: Students’ survey design Survey launched in February 2012 Includes 27 questionsspread across 9 sections Total number of responses N= 590 (across 11 different Schools) Response rate: 14.4%. (based on 4093 EVS handsets registered in June 2012 )

7. Methodology: Staff survey design Survey launched in June 2012 Includes 18 questions spread across 5 sections Total number of responses N= 88 (across 13 different Schools) Response rate: 5.9%. (based on 1500 academic staff registered on staffq list in June 2012)

8. Methodology: Students’ blogs: Participants Student blogs were completed in a variety of ways according to individual preferences : Word Files (46), online blog entries (14), Short video clips (11), Audio files (1) and submitted via a restricted access group area on StudyNet (UH MLE) 65 students registered for the web/ written blog of whom 27 were chosen to ensure a spread of genderand programme. One failed to complete their blog 7 Academic Schools: Business (6), Computer Science (7), Humanities (6), Law (4), Education (1), Sports management (1), Mathematics (1) Equal male and female numbers applied but ratios varied according to subjects 15M:11F All the bloggers had previously used EVS since at least January 2011 The reflection period varied between 2 and 4 weeks in the period Nov 11- Feb 12

9. Methodology: Student blogs:Original and Emerging Categories and Themes

10. Methodology: Staff interviews:Participants and Developing Categories and Themes • Semi-structured interviews with • 12 staff members incl: • Local School Champions (2) • Associate Heads and Heads of School (3) • Lecturers (6) • The iTEAM Project Director (1)

11. Preliminary Findings : Summary of students’ views (survey and blogs) • Usability or ease of use of the EVS was rated very highly by students • Very positive impact on students’ learning and satisfaction: • Responding to questions makes me think more about the course material (80%+) • EVS provides them with an immediate check of their understanding (80%+) • Viewing responses give them an idea of how they are doing in relation to the rest of the class (80%+) • Less positive impact on perceived performance and engagement : • use of EVS will increase their chances of getting a good grade (49.9%) • Spend more time discussing the subject with colleagues (47.5%) • Thinks about the subject outside of the classroom (34%) • Ask more questions in classes that use EVS (21%) • Time given for answering questions is not always sufficient Formative use of EVS has been seen as adding a value to students’ learning experience, however summative use has in some cases created “an unfortunate tension between the administrators, the tutors and the students” (MC12) All students find the EVS very easy to use, with little or no training; however, accessibility might be an issue for students with sight or dexterity impairments. Majority agree that EVS is a great tool for answering QUESTIONS, but inadequate as a TEST tool

12. Preliminary Findings : Staff Survey • There is an overall (UH-wide) highly positive attitude to technology (EVS) acceptance with 83% of surveyed staff intending to use EVS in the next 6-12 months. • A more positive attitude in schools from Social Sciences and Arts subject area than in other subject groups • There is an overall (UH-wide) high agreement on handing out EVS for free upon registration and disagreement that the use of EVS has led to immediate operational efficiencies

13. Preliminary Findings : Staff interviewsThe Head of School ‘Our biggest challenge was getting staff on board with it. I would think when I started, 25% of staff were very enthusiastic and 75% were not, now we’re probably running at 50/50. Everybody who’s used EVS and got it to work comes back and says it’s great and they’re going to use it again… the technology has got to be easy so that people can get in the swing of it’ Head of School interview June 2012

14. Preliminary Findings : Staff interviewsThe Lecturers …it was so obviously simple to generate questions and I think that has always been the main barrier in people’s minds, that because it’s something new and that it’s a technology that it has to be difficult as well ‘Once it’s embedded in the workload for staff then that should be greatly reduced in terms of marking, writing assignments, it’s really refreshing the bank of multiple choice questions [they’ve created.]’

15. Conclusions • High student enthusiasm for using EVS in class especially for formative and low-stakes summative assessments • Plenty of top-down support and leadership and local support is important to embed the technology • Academics using EVS are not yet seeing all the expected efficiencies in time saved that they had hoped for. • BUT 93% of those responding to the survey and using it for the first time in 2011 said they plan to use it in the next 6 -12 months • Patterns of previous technology adoption at UH are mirrored – • The example of the MLE saw early student enthusiasm but academics took more time to feel fully at ease with the change in practice and pedagogy.

16. References Bangor, A., Kortum, P., & Miller, J.A. (2008). The System Usability Scale (SUS): An Empirical Evaluation, International Journal of Human-Computer Interaction, 24(6). Chickering and Gamsonhttp://www.uis.edu/liberalstudies/students/documents/sevenprinciples.pdf D’Inverno, R., Davis, H., & White S. (2003), Using a personal response system for promoting student interaction, Teaching Mathematics and its Applications, Vol. 22 (No. 4): 163-169. Draper, S. W., & Brown, M. I.(2002).Use of the PRS handsets at Glasgow University, Interim Evaluation Report: March 2002 http://www.psy.gla.ac.uk/~steve/evs/interim.html accessed April 2012 Draper, S. W., & Brown, M. I. (2004). Increasing interactivity in lectures using an electronic voting system. Journal of Computer Assisted Learning, 20(2), 81-94. JISC (2004) http://www.jisc.ac.uk/media/documents/publications/effectivepracticeelearning.pdf Kanter, R.M., Stein, B.A. & Jick, T.D. (1992) The Challenge of Organizational Change. Free Press. New York Kennedy, G. E., & Cutts, Q. I. (2005). The association between students' use of an electronic voting system and their learning outcomes. Journal of Computer Assisted Learning, 21(4), 260-268 Lorimer, J., & Hilliard, A. (2009). Use of a Electronic Voting System (EVS) to Facilitate Teaching and Assessment of Decision Making Skills in Undergraduate Radiography Education. Paper presented at the Proceedings of 8th European Conference on e-Learning, Bari, Italy Moore, G. A. (1991). Crossing the Chasm. New York: Harper Business. Nicol, D., & Draper, S. (June 2009). A blueprint for transformational organisational change in higher education: REAP as a case study. In J. T. Mayes (Ed.), Transforming Higher Education through Technology-Enhanced Learning. Oliver, M. (2006). New pedagogies for e-learning? ALT-J: Research in Learning Technology, 14(2), 133 - 134. Robins, K., (2011) EVS in the Business School, University of Hertfordshire Internal Report Thornton, H. A. (2009). Undergraduate Physiotherapy students’ choice and use of technology in undertaking collaborative tasks. Open University, UK, Milton Keynes Twetten, J., Smith, M.K., Julius, J. and Murphy-Boyer, L. (2007) Successful Clicker Standardization EDUCAUSE QUARTERLY • Number 4 2007 Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. B. (2003). User acceptance of information technology: toward a unified view. MIS Quarterly, 27(3), 425–478. Willis, J. (2009) Using EVS in the School of Life Sciences, University of Hertfordshire Internal Report