Piloting CAA: All Aboard

1. Piloting CAA: All Aboard Gavin Sim & Phil Holifield

2. Overview • Introduction CAA at UCLAN • Key Challenges • Staff Uptake • Framework • Staff Development • Students • Other Stakeholders • Conclusions and Discussions

3. Introduction • Teaching and Learning strategy incorporated e-learning – mainly content development • First summative CAA test summer 2003 WebCT • TRIADS and Questionmark evaluated for pilot study • Questionmark - adopted felt easier for staff develop their own questions

4. Introduction • Technical Infrastructure analysed concerns • Scalability – expansion over time • Connectivity – internal and external colleges • Bandwidth – 10 Mbps available, multimedia • Purchased dedicated server host Questionmark, Internet Information Server & SQL Server • Integration with other systems concern but addressed later • Piloting Within Department of Computing

5. Key Challenges • Encouraging Staff Uptake • Staff Development • Stakeholder Acceptance – e.g. management • CAA perceived ability to test range of cognitive skills • Practical Issues - Labs

6. Methodology • Questionnaire Staff n=34 response rate 64% • Views in relation to CAA, support and training • Framework developed based on Blooms Taxonomy, 6 staff, 8 modules • Questionnaire Students n=86 response rate 94% • Acceptance of technique • Question styles • Language Used • Usability

7. Staff Uptake • Computing encompasses range of subjects technical networking, subjective HCI • CAA may readily lend itself assessment specific disciplines • Questionnaire revealed only five members of staff used CAA, 3 actively using it • Encourage uptake CAA being incorporated into department’s strategy, all level 1 formative and summative being optional

8. Staff Uptake • Five staff now using CAA within the department • Questionnaire revealed • 91% use CAA formative • 56% Summative • Difference could be attributed level lecturer teaches

9. Framework • Analysing structure of module identify how CAA could be incorporated into modules Learning Outcomes Other Assessment format Bloom’s Taxonomy CAA Syllabus

10. Number of Learning Outcomes at each level of Blooms Taxonomy Framework

11. Framework • Variations between number of Learning Outcomes from 3 – 8 • Level 1 modules at lower Cognitive Level • Level 2 Module CO2601 (Technical Solutions and Business) requires students to demonstrate similar ability found on CO3707 • Next is to identify elements of syllabus and relationship to Learning Outcomes • Prevent unrelated content being integrated into exam

12. Example for CO3707 Identify the parts of the syllabus that relate to the learning outcomes. Framework

13. Framework • Number of syllabus elements at each level of Bloom’s Taxonomy

14. Framework • Is going to be used on MSc Web Development Module • Module is all coursework • Formative test in first semester • Enable students gain early feedback • Lecturer obtain early indication of their progress • Framework shows how staff can integrate CAA into modules but further development necessary

15. Staff Development • Asked staff ‘ Would you be prepared to input the questions into the software yourself?’ • 80% Yes • May not reflect attitude staff in other departments

16. Staff Development • Lecturers need support in question design 74% • LDU organised staff development in CAA • An introduction to Computer Assisted Assessment • CIF bid for funding pay developer to work with staff develop multimedia questions • 81% more time required to write questions • Question banks and experience reduce time • 61% lecturers help invigilation (essential)

17. Staff Development • Informal Focus Groups • Discuss problems and share experiences • How accommodate students special needs • Invigilation issues • Risk issues e.g. server fails • Without this students experience may be different from module to module

18. Students • Attitude measured through series of questionnaires • Students asked ‘ Would you find this format of assessment an acceptable replacement for part of your final exam?’ • 5 Point Likert Scale, Strongly Disagree=0, Strongly Agree=4 • Mean=2.9, SD=.9, 99% Conf. Interval ± 0.26 • Indicates reasonable level of support

19. Students • Research into computer anxiety and CAA (Liu et al. 2001; Zakrzewski & Steven 2000) • Concerns, students no prior experience of QM • ‘This format of assessment is more stressful than a paper based test’ • Mean=.99, SD=.987, Conf. Interval ± 0.28 • Comments ‘I prefer completing a test in this way as it is less intimidating’ • ‘As a computer geek I feel more at ease in front of a computer.’ (final exam)

20. Students • ‘Did you have any difficulties accessing the test?’ • 14% Yes • Majority problems copying password from email with white space • Software could trim white spaces • Authentication could be achieved through LDAP process

21. Students • Questionmark used question by question delivery • Standard Templates • Question the suitability of a number of templates e.g. scrolling, navigation • Idea have a template bank

22. Students • Series of questions relating to the interface

23. Students • 81 Students completed questionnaire 3o provided qualitative feedback • Requested facility go directly back to previous question (11 times) • ‘Proceed’ button felt inappropriate near main navigation • Features incorporated into forthcoming test and further analysis will be conducted

24. Other Stakeholders • Information System Services and Management informed through steering committee • Responsibility report finding of the evaluation for institutional wide deployment • Without support of management additional resources will not be made available

25. Conclusions and Discussions • Scepticism about CAA appropriateness at level 3,4 for summative assessment • Framework showed how it may be incorporated further research required • Adopting CAA into departments strategy increased uptake but staff development necessary • Students responded positively to experience • Logging in process could be improved • Comparison of WebCT and Questionmark planned

26. Questions