210 likes | 339 Views
Strategies for large-scale assessment: an institutional analysis of research and practice in a virtual university. Gavin Sim, Naveed Malik, Phil Holifield. Introduction. Background of VU Infrastructure Strategy Admissions Assignments Examinations Future Strategy Conclusions.
E N D
Strategies for large-scale assessment: an institutional analysis of research and practice in a virtual university Gavin Sim, Naveed Malik, Phil Holifield
Introduction • Background of VU • Infrastructure • Strategy • Admissions • Assignments • Examinations • Future Strategy • Conclusions
Background • Virtual University Government Funded Project • Courses delivered via satellite television and a Learning Management System (Intralearn) • Utilise lead academics to develop courses • Students attend Learning Centres across Pakistan to view programmes and use the Internet • Evidence that the needs of learners in remote areas can be served via local learning centres (Mayes, 2001)
Background - Cont. • Current Cohort 1700 capabilities of 30750 • Universities in Pakistan full year course followed by exams • VU this strategy not feasible, students need to be able monitor progress and obtain feedback • Marking time would be 212 days for typical exam scripts • New Strategy required utilising the technology
Infrastructure • Guidelines highlight need to identify technical issues (Stephens, Bull et al. 1998) • VU identified, distributed cohort - connectivity, scalability, security & authenticating users • Academic Intranet • Courses and assessment delivered via the LMS • Intralearn requires a ColdFusion backend and a DBMS which Microsoft SQL Server is used • Intralearn and SQL-Server are hosted on separate servers creating a 3-Tier approach
Infrastructure - Cont. • Scalability an Issue • Max bandwidth between Learning Centres and VU Hub 2Mbps can increase to 155Mbps • The LMS needs an adaptable architecture • Custom built software has been developed for submitting the assignments (http upload). • Asynchronous assessment is required due to the variability in connectivity between centres • Students can use any software - this is then encrypted, renamed and forwarded to the tutor for grading
Infrastructure - Cont. • Delivery of exams - security issues • Security flaws in sending emails such as authentication and encryption (Hatton, Boyle et al. 2002) • Due to student numbers an asynchronous approach required • Software developed and deployed at centres • Exam sent to centres encrypted and placed on local server • Same file returned encrypted after the exam
Strategy - Admissions • Admissions scheduled for spring and autumn • Test administered to 5000 students comprising MCQ and essay questions. • Marking outsourced to National Testing Centre • Found no advantage in using this approach due to bias for those exposed to testing before • Test anxiety could adversely effect their grades (Cassady & Johnson 2002) • Admissions based on public examination results
Strategy – Assignments • Account for 15% of marks and students usually have between 5-10 per semester. • Regular testing is vital (Croft, Danson et al. 2001) • A series of bad marks can lead to a downward spiral in learning and confidence (Taras, 2002) • The assignments are structured to build student confidence • CS101 first assignment send an email and use the discussion board
Strategy – Assignments Cont. • Submission of assignments deadline plus one day • Academics are responsible for designing the assignments • Essay style formats usually used rather than objective testing • Tutors responsible for marking 250-300 assignments per week • Number of tutors is proportional to the number of students enabling scalability
Strategy – Assignments Cont. • Solutions are posted on LMS no feedback is provided • Self-reflection against a model answer enables students to gain personal insight of their understanding (Peat, Franklin et al. 2001) • Can ask questions concerning assignments on moderated discussion boards
Strategy - Exam • Two exams per module mid-term 35% and final exam 50% • Examiners are responsible for authenticating student ID at learning centres • Exams consist of both MCQ and Essay style questions • MCQ issues surrounding guessing (Harper, 2003) • Essays subjectivity with different markers
Strategy – Exam Cont. • Having varied question styles help produce fairer test • MCQ marked automatically whilst essays are marked by tutors • The students type their essays improving legibility for the tutors • This method enables the full range of cognitive skills to be tested as defined by Bloom (1956) • Debate whether objective testing can test higher cognitive skills in all subject domains
Strategy - Exams Cont. • In the majority of instances Synthesis and Evaluation promote divergent thinking and answers cannot be determined in advance (Bloom 1971) • Higher Cognitive Skills tested in Linguistics (Reid 2002) • Results are posted on the LMS • Any queries are directed to staff within the VU
Future Strategy • Group projects planned for final year • Students will work with others outside their geographical area • Tutors responsible for grading projects • Explicit guidelines will need to be drawn up • Ways to overcome gaps in face-to-face presentations and lab demonstrations are sought
Conclusions • Infrastructure and strategy need to be developed in parallel • A strategy of combining CAA and tutor marked assessments is feasible • The scalability of the strategy and infrastructure should enable successful future expansion • Future analysis and research in the field of CAA may produce solutions that overcome some of the drawbacks