1 / 23

SOPHISTICATED TASKS IN E-ASSESSMENT

SOPHISTICATED TASKS IN E-ASSESSMENT. WHAT ARE THEY? AND WHAT ARE THEIR BENEFITS?. Andrew Boyle Research & Statistics team QCA. Importance of e-assessment. Important in major government initiatives eLearning strategy White Paper on 14 – 19 education QCA’s e-assessment vision

craig
Download Presentation

SOPHISTICATED TASKS IN E-ASSESSMENT

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. SOPHISTICATED TASKS INE-ASSESSMENT WHAT ARE THEY?AND WHAT ARE THEIR BENEFITS? Andrew Boyle Research & Statistics team QCA

  2. Importance of e-assessment • Important in major government initiatives • eLearning strategy • White Paper on 14 – 19 education • QCA’s e-assessment vision • E-assessment commonplace by 2009 • Currently developing KS3 ICT test • CAA well established in UK HE

  3. Importance of sophisticated tasks • Substantially different task types in the near future • From pencil-and-paper tests and from early CAA • Tomlinson report/14 – 19 White Paper • E-assessment not limited to multiple-choice testing • Potential to test learners in both structured and unstructured environments • Both short and long answer questions • Ability to use techniques (sic) such as video clips • Assess a wider range of knowledge, skills and understanding than possible at present

  4. Possible e-assessment futures • ‘Intelligent measurement’ • Sophisticated assessment methods (e.g. simulations) become more practicable • ‘Reinvented’ learning and assessment • Radically different assessment tasks • Integration of learning and assessment • Highly dynamic and adaptable content

  5. Examples of sophisticated tasks • World Class tests • Key Stage 3 ICT test • TRIADs • NAEP-TRE • Concept-mapping tasks

  6. Taxonomies of e-assessment tasks • Parshall’s five-dimensional framework • Item format • Response action • Media inclusion • Level of interactivity • Scoring algorithm

  7. Towards a definition? • Parshall • Five dimensions likely to be inter-related • Each dimension acts as a continuum • No simple-to-sophisticated cut points • Difficult to find a definition • Cf Mackenzie • Possible to suggest core features

  8. Core features of sophisticated tasks • Contain media-rich stimulus material • Would rule out essays – even machine-marked • Test taker interacts with stimulus in various ways • Choosing responses (various ways) • Constructing responses (writing answers, plotting data points) • Directly demonstrating capability (simulations) • Test taker produces complex work • More complex than simple CAA, not necessarily more sophisticated than pencil-and-paper tests

  9. Comments on core features • Must contain several features to be ‘sophisticated’ • Notion of ‘sophistication’ always linked to other aspects – not just task design • Test model • Type of data produced • Marking • Reported information • Assessment purpose • Description of tasks as ‘sophisticated’ usually amounts to claim about merits of tasks/test

  10. Possible benefits of sophisticated tasks • Measuring different things • Addressing different assessment purposes

  11. Measuring different things • Simple items based on a ‘dated form of psychology’ • Learner as a ‘collector of facts and skills’; added to repertoire independently • At odds with recent understanding of learning and experts’ cognition • Conventional test models at odds with test takers’ cognitive strategies • Success in tests depends on at least two dimensions (accuracy and speed) • Test-taking strategies • Models should account for varied sources of difficulty

  12. Measuring different things (2) • Curriculum coverage • ‘Higher-order learning outcomes (HLOs)’ • Valued in curriculums, but not easily tested by simple item formats • Simple-item tests might not represent the curriculum adequately • Particularly important if modern curriculums emphasise new types of abilities, etc. • ‘Shared rhetoric in education’ • Emphasis on: Maths and Science; ICT; problem solving and communication

  13. Definition of ICT capability • Construct measured in KS3 ICT test • Technical and cognitive proficiency to access, use and communicate information using technological tools • Purposefully applying technology to solve problems, analyse information, develop ideas, create models and exchange information • Discriminating in use of information and ICT tools • Contrasted with ICT skills • Technical competences to carry out tasks using common software applications

  14. Challenge for sophisticated-task users • Sophisticated tasks depart from outdated psychological models • Need for new mental models • Significant challenge to researchers • Account for implications of sophisticated tasks • When specifying assessment model • During validation

  15. Threats to validityfrom sophisticated tasks • Construct-irrelevant variance • Sophistication and complexity in user interfaces - ‘nuisance variables’ • Detract from validity • Differential impact of new task styles • Learners of differing backgrounds • Different cognitive styles/dispositions

  16. Different assessment purposes • Substantial strand of recent research addresses assessment purposes • Formative assessment can improve learning • Formative assessment wide-reaching notion • Applies to formal test instruments and to day-to-day teaching techniques (e.g. questioning styles)

  17. Formative e-assessment • Potential of e-assessment to provide useful feedback on aspects of sophisticated tasks • Well-designed formative information more supportive of learning than simple right|wrong • Varied presentation styles • Providing feedback in range of visual and interactive modes • Might include a wider range of learning styles

  18. Limits to the optimism (1) • E-assessment is expensive to develop • Costs of e-assessment shifted towards front of cycle • Large budgets necessary for test development • Especially true of sophisticated tasks • Large organisations tend to have most resource to commit to big projects • Generally focused on summative assessment • Often conservative, risk averse • Makes use of sophisticated tasks for innovative formative assessment less likely

  19. Limits to the optimism (2) • Writing good test items is difficult. • Good quality items take a long time to produce; iterations of reviewing, editing, etc. • Teachers don’t necessarily have item writing skills • Claim that difficulty in writing good items is particularly pronounced for sophisticated tasks • Sophisticated tasks • Expensive • Slow to develop • Not easily written by non-specialist • May be difficult for teachers and learners to ‘own’ • Lack key characteristic of formative assessment • Assessment closely integrated with classroom practice

  20. Discussion • No clear-cut definition of ‘sophisticated tasks’ • Device to distinguish from alleged reactionary use of simple question styles in early CAA • Critique of benefits of sophisticated tasks • Implies a position of informed scepticism • Central assertion is accepted • Sophisticated tasks will soon be ubiquitous • They will bring real benefits • Benefits have been asserted rather than proven • Need to move to informed implementation

  21. Discussion (2) – the two benefits • Assessment of HLOs • Formative assessment • Often confounded – they are distinct • Argument stronger in case of HLOs • Empirical evidence in near future • Sophisticated tasks’ ability to facilitate formative assessment – less secure • Difficulty in authoring tasks – continue to be created by large central organisations • Formative assessment ought be to low-tech – usable in everyday classrooms • Use of day-to-day software applications for formative assessment – e.g. track changes in word processors

  22. Other work to do onsophisticated tasks • Deriving scores for each task • Measurement models • summarise scores • provide meaningful information about performance/ability, etc. • Validation of tests

  23. Mackenzie definition • Advanced Computer-Based Assessment … • goes beyond the simple multiple-choice/response item types into the area of complex question types, adaptive branching, scenarios and simulations of real life situations or problems. In formative mode, feedback may be immediate, extensive, context-sensitive and include annotation of diagrams, links to web resources or full courseware/e-learning tutorials making it an extremely powerful learning tool.

More Related