Enhancing Learning through Online Assessment Tool

1. Using QuestionMark Perception for summative on-line assessment Marija Cubric May 2006

2. Context • Level: UG1 • Module: Introduction to programming • Practical “hands-on” subject that requires some amount of thinking  • Cannot be learnt from books only • Number of students: ~100 • Mix of BISF, AMIS, BS, MB, .. • Last year failure rate: 56% • Main Issues: • Low attendance (>50% never showed up!) • Low motivation (does not count towards the final grade) • Not enough contact time (1hpw tutorial – not sufficient)

3. Solution (first iteration ) • Attendance • Weekly on-line test (20% of final grade) • Motivation • Publishing “league tables” (top 10, 10 most improved etc) • Contact time (tutorials) • Increased from 1hpw to 2hpw

4. Theory • Objective questions (MCQ/MRQ) can be designed to test all levels of Bloom’s educational competencies: • Knowledge • Comprehension • Application • Analysis • Evaluation • Objective questions can be used in summative/ not only formative assessment, but need to be mixed up with other forms of assessment • Creativity, design and development, non-cognitive skills, writing style etc – cannot be tested with objective questions. • Receiving immediate feedback is important part of any constructivism*-based learning cycle such as Kolb’s learning cycle • (*Constructivism - knowledge construction is social process that occurs through collaboration with others) • Amanda Jeffreys & Andy Oliver (BLUE team) run w/shops on how to design objective tests

5. Tool • QuestionMark Perception • 3rd party sw, licensed to UH (trial) • Not part of StudyNet • http://www.questionmark.com/uk/home.htm • Tool administrator: Angus Shaw <r.a.shaw@herts.ac.uk> • To setup account for you • To create participant groups (module code)

6. Features • Roles: administrator or participant • Administrator functions: • Question management (C/E/D) • Assessment management • Schedule management • Group management • Report management • Participants functions: • Start & submit test • Review test results

7. Login • http://www.perception.herts.ac.uk/em/login.asp

8. Question database

9. Types of questions

10. Configuring assessment

11. Configuring schedule

12. Report example (full score list report) • Result index • Assessment ID • Assessment name • Assessment author • Assessment last modified • Participant • Participant group • Participant details • Hostname • IP address • Status • Date/time started • Date/time finished • Total score • Maximum score • Percentage score • Time taken • Topic percentage score • Topic result

13. Types of reports

14. Test example • Login to: • http://www.perception.herts.ac.uk/q/perception.dll

15. Feedback • Not immediate (except the score) • Run through the answers immediatelly after the test (extra 5-10 minutes)

16. Results (tutor’s perception) • Increased attendance: • Average tutorial attendance: 75% • Increased motivation: • “League tables” very popular! • Better results? • Yet to be seen • Expecting raise of 10% in pass rate • “Liminality” (Glynis Cousin) • “mimicking” rather than “thinking” (“cut&paste” strategy)

17. Results (students’ perception) • Data: anonymous questionnaire (74 respondents)

18. Results (usability)

19. Workload • Creating questions (10min/q) • Creating assessments & schedules (30min/w) • Importing reports to Excell & generating league tables (30min/w) • Beware: extra time >4hpw (incl. 20 new questions)

20. Questions? • Can questions be parameterized & parameters within question randomized • NO • Can questions database be exported • Questions-yes, not sure how to expeort answers??? • Can it be used outside of the campus • YES (with usual proxy settings for remote access)

21. URLs • Admin login: • http://www.perception.herts.ac.uk/em/login.asp • Participant login: • http://www.perception.herts.ac.uk/q/perception.dll • More on QuestionMark Perception: • http://www.questionmark.com/uk/home.htm

22. Top 10 – see StudyNet for full score table

23. Top 10 (most improved) – see StudyNet for full score table