The Role of e-Assessment in XXI Century Education: Opportunities and Risks Martin Ripley

Strategic objectives About WCAL Risks & Issues Three case studies The Role of e-Assessment in XXI Century Education: Opportunities and Risks Martin Ripley Managing Director, World Class Arena

Strategic objectives About WCAL Risks & Issues Three case studies website • WCAL’s History • Formed in 2007 • We specialise in the creation of learning resources for the digital age • We create assessment resources – such as school tests • Includes paper-based, on-screen and face-to-face learning events, with technology as a significant theme • We operate globally and our particular niche is talent development. “Talent development” is concerned with identifying and supporting future leaders – in social, entrepreneurial, economic, political, scientific and literary fields www.worldclassarena.net www.worldclassarena.org.uk www.worldclassarena.hk contact martin@worldclassarena.net

Strategic objectives About WCAL Risks & Issues Three case studies • WCAL’s Services • We provide the following services: • Authoring textbooks and related content • Designing and running teacher training events • Support and advice for companies considering introducing technology into their learning and assessment businesses • Design and sale of 21st century assessment and testing products • Strategic support for schools looking to provide greater challenge and to raise the aspirations of their students website www.worldclassarena.net www.worldclassarena.org.uk www.worldclassarena.hk contact martin@worldclassarena.net

Strategic objectives About WCAL Risks & Issues Three case studies website www.worldclassarena.net www.worldclassarena.org.uk www.worldclassarena.hk contact martin@worldclassarena.net

Strategic objectives About WCAL Risks & Issues Three case studies

Strategic objectives About WCAL Risks & Issues Three case studies It is estimated that 40 exabytes (4.0 x 10 19) of unique new information will be generated worldwide this year....

Strategic objectives About WCAL Risks & Issues Three case studies 2x The amount of new technical information is doubling every 2 years

Strategic objectives About WCAL Risks & Issues Three case studies For students starting a three-year university degree, this means that...

Strategic objectives About WCAL Risks & Issues Three case studies ...half of what they learn in their first year of study will be outdated by the end of their course.

Strategic objectives About WCAL Risks & Issues Three case studies • Researchers have found fairly uniform effects: • Assessments signal priorities for curriculum and instruction. • School administrators and teachers pay attention to what is tested, analyse test results, and adapt curriculum and teaching accordingly. • Teachers tend to model the pedagogical approach reflected in tests. • When high visibility assessments are composed of multiple-choice items, teachers tend to rely heavily on multiple-choice worksheets. • Curriculum developers, particularly commercial interests, respond to important tests by modifying existing textbooks and other instructional materials.

Strategic objectives About WCAL Risks & Issues Three case studies • Research shows the potential for substantial negative consequences. • Schools and teachers tend to focus on what is tested rather than underlying standards or learning goals, and to ignore what is not tested. • Focusing on the test, rather than underlying learning, may encourage a one-time performance orientation and transmission-type teaching. Schools may promote a performance orientation in students, which in turn can work against students’ engagement and persistence in learning, metacognition, and self regulation. • Instructional/teaching time is diverted to specific test preparation activities.

Strategic objectives About WCAL Risks & Issues Three case studies • Use of assessment information is the key learning • Assessment evidence must be acted upon to inform subsequent instruction. • Assessment helps to chart the learning road forward, by identifying and providing information to fill any gaps between the learners’ current status and goals for learning. • Assessments can directly support the learning process by incorporating principles of learning and cognition (Herman & Baker Bennett & Gitomer, 2009). For example • by asking students to make public their thinking • by asking students for explanations and providing practice over multiple and authentic contexts • by making learning goals explicit and involving students in self -assessment

Strategic objectives About WCAL Risks & Issues Three case studies • Quality marks of e-assessment in the 21st century • Be aligned with the development of significant, 21st century goals: • Incorporate adaptability and unpredictability: • Be largely performance-based. • Add value for teaching and learning. • Make students’ thinking visible. • Be fair. • Be technically sound. • Valid for purpose. • Generate productive and usable feedback for all intended users. • Provide productive and usable feedback for all intended users. • Build capacity for educators and students. • Be part of a comprehensive and well-aligned system of assessments designed to support the improvement of learning at all levels of the educational hierarchy.

Strategic objectives About WCAL Risks & Issues Three case studies Innovative assessment website Innovative assessment, designed to change teaching & learning Transformational strategy Technology-rich assessments Paper-based assessments Technology delivers process improvements – efficient, flexible Traditional, paper-based, low levels of innovation Migratory strategy Traditional assessment

Strategic objectives About WCAL Risks & Issues Three case studies • Top Ten Cited Benefits of Computer-Based Testing • Improved access. More administration dates. More administration sites. • Efficient collection and scoring of responses. The possibility of instant scoring. • High-quality items and test forms that reflect 21st century skills. Greater validity. • Use ICT tools. • Test administration standardised (eg timing). • Accuracy. • Improved motivation. • The ability to provide teachers and students with detailed information that can be used to improve future learning. • Reduced costs - of printing data entry, collection, aggregation, verification, and analysis. • Substantial advantages for test security. Student responses should not get lost.

Strategic objectives About WCAL Risks & Issues Three case studies • In 2001 City and Guilds set a target to provide 80% of their objective assessments on-line by 2006. By 2010 they were delivering over 700,000 exams on computer each year. • The Graduate Management Aptitude Test (GMAT) is marked twice – once by an expert and once by computer. • ACCA has announced that all its global accountancy tests will be on computer by 2015. • ExamSoft has completed a trial of 2000 law students in the US completing their bar exams on their own laptop – all sitting in a traditional, large exam hall. • The United States Medical Licensing Exam includes a computer-based simulation of a patient with a range of symptoms which, in the examination, must be diagnosed and a treatment plan must be created. • In the UK, theory driving tests are booked on-line and completed sitting at a computer. A search engine enables people to locate their nearest test centre.

Strategic objectives About WCAL Risks & Issues Three case studies website • 1. Wireless Generation • Assessment of early reading • Assesses children’s grasp of reading skills – the alphabet, phonic knowledge, reading, comprehension • Includes Progress Monitoring • Can be performed every 10 days to 2 weeks • Can be administered in 2-3 minutes • Visual charting of progress makes it easy for teachers to see if teaching methods are working • Provides a large database of assessments so that progress monitoring can be done as often as necessary without repetition of content www.wirelessgneration.com More information http://www.worldclassarena.net/research3.htm

Strategic objectives About WCAL Risks & Issues Three case studies website • 2. Primum: USA Medical Exam • In the 1960s there were concerns that the one-day practical oral assessment was susceptible to inter-rater reliability and there were concerns about resource requirements. • Developed a series of Patient Management Problems • 1999 – computer-based case studies made mandatory • Each examinee has to address nine case studies http://www.usmle.org/Examinations/step3/step3_content.html contact See also: www.worldclassarena.net /research.htm for a review report regarding Primum software

Strategic objectives About WCAL Risks & Issues Three case studies website • Software presents the candidate with a scenario and a control screen. • From here the candidate can • Request more comprehensive examinations • Order medical procedures • Advance the time • Move the patient • Several thousand tests and procedures are available. Candidates make free text entry. www.worldclassarena.net www.worldclassarena.org.uk www.worldclassarena.hk contact martin@worldclassarena.net

Strategic objectives About WCAL Risks & Issues Three case studies website • World Class Tests • In the two key areas of mathematics and problem solving) • Calibrated against the performance of the best ten percent of pupils aged 9 and13 from a range of countries • When ready model of administration • Internationally calibrated www.worldclassarena.net www.worldclassarena.org.uk www.worldclassarena.hk contact martin@worldclassarena.net

Strategic objectives About WCAL Risks & Issues Three case studies website www.worldclassarena.net www.worldclassarena.org.uk www.worldclassarena.hk contact martin@worldclassarena.net CHORTLERS

Strategic objectives About WCAL Risks & Issues Three case studies A strategic view of risks website Results Scoring Delivery Writing Design Dumbing Down Cost contact Technical Reliability Cheating Benefits for learners

Strategic objectives About WCAL Risks & Issues Three case studies website ‘Big Bang’ approaches, ie introducing everything at once Preparing schools – people and systems Providing support, guidance and training materials Communication, communication, communication Dealing with cheating, and concerns about integrity of the assessments Delivery and roll-out contact

Strategic objectives About WCAL Risks & Issues Three case studies website Finding the expertise needed Working with a technical partner, possibly a company Putting in place an effective and reliable procurement approach Implementing a controlled management approach – innovation in a controlled environment Procurement contact

Strategic objectives About WCAL Risks & Issues Three case studies website Being too ambitious Dumbing down Expecting more than the technology can support Not focusing on the use for students – what is the pedagogical model of learning Item design, reporting contact

The Role of e-Assessment in XXI Century Education: Opportunities and Risks Martin Ripley