1 / 87

Tips for Providing Fair, Accessible, and Efficient Online Assessments

Tips for Providing Fair, Accessible, and Efficient Online Assessments. Jeff Place, Questionmark email: jeff@questionmark.com www.questionmark.com. Role of assessments – IN Learning and OF Learning. … OF Learning. Types of Assessments. Summative Assessments

newman
Download Presentation

Tips for Providing Fair, Accessible, and Efficient Online Assessments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Tips for Providing Fair, Accessible, and Efficient Online Assessments Jeff Place, Questionmark email: jeff@questionmark.com www.questionmark.com

  2. Role of assessments – IN Learning and OF Learning

  3. … OF Learning Types of Assessments • Summative Assessments • Measure knowledge, skills and competencies • Examples: • Exams and Certifications • Compliance tests • Pre-employment tests • Reaction Assessments • How did you feel about..? • Examples: • Course evaluations • Level 1s … IN learning Formative Assessments Strengthen memory recall Examples: Quizzes Diagnostic Assessments Route you to learning Learn what you know Learn what you need to know Examples: Learn Quizzes Tests

  4. 0 Pre-Course Formative Post-Course Assessments IN Learning Results Database – Aggregating results for all interactionsNeeds Analysis, Opinion/attitude surveys, Transcripts, Skills Gap Analysis, Certification Results AppropriateLearningEvents

  5. 0 Pre-Course Formative Post-Course Assessments IN Learning Results Database – Aggregating results for all interactionsNeeds Analysis, Opinion/attitude surveys, Transcripts, Skills Gap Analysis, Certification Results • Purpose: • Create intrigue • Route to course • Establish benchmarks AppropriateLearningEvents

  6. 0 Pre-Course Formative Post-Course Assessments IN Learning Results Database – Aggregating results for all interactionsNeeds Analysis, Opinion/attitude surveys, Transcripts, Skills Gap Analysis, Certification Results AppropriateLearningEvents • Purpose: • Stimulate and strengthen memory • Correct misconceptions

  7. 0 Pre-Course Formative Post-Course Assessments IN Learning Results Database – Aggregating results for all interactionsNeeds Analysis, Opinion/attitude surveys, Transcripts, Skills Gap Analysis, Certification Results • Purpose: • Determine if knowledge transfer occurred • Reduce Forgetting Curve AppropriateLearningEvents

  8. Learning Curve Knowledge/Skill Time Training Ouch! The Forgetting Curve Training

  9. Questions IN the Learning Process Questions…. Knowledge/Skill Time Reduce Forgetting Reduce Forgetting Strengthens Memory Creates Intrigue

  10. … OF Learning Assessments • Summative Assessments • Measure knowledge, skills and competencies • Examples: • Exams and Certifications • Compliance tests • Pre-employment tests • Reaction Assessments • How did you feel about..? • Examples: • Course evaluations • Level 1s … IN learning Formative Assessments Strengthen memory recall Examples: Quizzes Diagnostic Assessments Route you to learning Learn what you know Learn what you need to know Examples: Learn Quizzes Tests

  11. 0 Level 4 - ResultsWas it worth it? Assessments After LearningKirkpatrick’s Levels of assessment Level 3 - BehaviorKSA being used on the job? Assessment Level 2 - KnowledgeDid they learn anything Assessment Level 1 - Reaction Was the environment suitable for learning? Assessment

  12. Assessments must: • Provide the Right Stimulus to Gather Data...then • Provide the RightAnalysiswith Right Presentationto the Right Person at the Right Time • Be Valid and Reliable • Be Efficient and Cost Effective

  13. Tip: Accommodating Test Takers with Disabilities Pay particular attention to the authoring and delivery of assessments

  14. Accommodations and Accessibility • People with disabilities deserve to be assessed fairly and sometimes Accommodations must be made • Following good practice in web accessibility not only helps those with disabilities but many others • Workers not used to computers can use touch screens • Temporary injuries incapacitating participants • Increasingly important for government, academic and corporations to empower and harness the intellect and energy of those with disabilities

  15. Blindness and Partially Sighted • International definitions of blindness and low vision / partial sightedness varies, but there are tens of 1,000s of people of working age impacted • Accommodations include: • Screens to be compatible with text to speech screen readers • Allow participant to: • Select large text • Change colors and contrast

  16. Color Blindness Colors of rainbow as viewed by a person with no color vision deficiencies. • Many types of color “blindness” • People see colors differently to the norm • ~7% of men and ~0.5% of women are color blind • Accommodations include: • Not using only color to distinguish information or navigation • Not relying on color within the question or choices to get the right answer Colors of rainbow as viewed by a person with protanopia. Colors of rainbow as viewed by a person with deuteranopia. Colors of rainbow as viewed by a person with tritanopia.

  17. Dyslexia • Dyslexia challenges people to read and spell • ~10% of people are impacted; 4% severely • Most sufferers muddle through • Accommodations include: • Allow participant to: • Select larger text • Change colors and contrast • Use (text to speech) screen readers • Have extra time • Not using only color to distinguish information or navigation • Not relying on color within the question or choices to get the right answer

  18. Motor Disabilities • Some disabilities (multiple sclerosis, paralysis, muscle and joint problems) prevent use of mouse/keyboard and require special input devices • Number of people affected: • Quadriplegia (often caused by vehicle or sports injury) • ~100,000 people in US • ~25,000 people in UK • Accommodations include: • Ensure assessment system can be used by keyboard (not mouse) as such special input devices emulate keyboard

  19. Guidance and Standards • W3C (WCAG) http://www.w3.org/TR/WCAG10/ (WCAG 1, 1999) http://www.w3.org/TR/WCAG20/ (WCAG 2, Dec 2008) • Section 508 from the US government • Main document (web applications): http://www.access-board.gov/sec508/guide/1194.22.htm • Secondary document (software): http://www.access-board.gov/sec508/guide/1194.21.htm • Section 508 similar to WCAG but not identical www.jimthatcher.com/sidebyside.htm

  20. Accessibility and Accommodations • Optimize delivery for screen-reader compatibility • Provide text-sizing and contrast controls • Enable navigation without mouse • Keyboard shortcuts • Alternate pointing devices

  21. Text-to-Speech Screen readers • Market leader text-to-speech screen reader is JAWS (75% of market) • www.freedomscientific.com/jaws-hq.asp • Costs about US$1,000/seat • Demo version available at http://www.freedomscientific.com/downloads/jaws/jaws-downloads.aspallows 40 mins use before restarting

  22. Allowing Extra Time

  23. Provide Accessibility Review • Workflow feature ensure review of accessibility issues

  24. Provide scribes and proctors to help those with disabilities answer assessments

  25. Expand “Clickable” Regions Select by clickingform field or choice text region

  26. Text Sizing Controls

  27. Contrast controls

  28. Tip: Consider how you plan to score a test or exam before you begin writing Recommended reading: Criterion Referenced Test Development: Technical and Legal Guidelines for Corporate Training Sharon A. Shrock and William C. Coscarelli

  29. CRT vs. NRT

  30. Setting cut scores for Criterion Referenced Tests • Informed Judgment • Poll SMEs to determine criteria for passing (no. of correct answers, no. of attempts) • Not recommended for high-stakes assessments • Angoff Method • Poll SMEs • Consider marginal candidates and the probability of getting specific questions right (0-100%) • Make a mathematical calculation to determine cut score • Borderline Group Method • Candidates are interviewed prior to an exam and borderline cases are identified • After the exam, the scores of borderline candidates can be used to determine the cut score • Contrasting Group • Compare SME to others

  31. Angoff MethodWhat is the expected possibility of a marginal student getting question right?

  32. Norm Reference Testing (NRT): The “Normal Curve”

  33. NRT: Test designed to “spread” and rank participants within a group; so to easily identify top performers Top 10% of scorers

  34. Norm Referenced If used to determine a ‘cut’ or ‘pass/fail’, the pass/fail score is determined after test Criterion Referenced Pass/fail score determined before test Setting a Pass/Fail score? 175 people 175 people 3rdgroup 71pass 1stgroup 55pass Only 50 people can pass 2ndgroup 35pass

  35. Tip: Outline a “Test Blue Print” before you begin writing your test or exam questions Ensure your test meets your objectives Improve efficiency of test writing

  36. 0 Assessment Blue Print • A detailed, written plan for an assessment • Typically includes • descriptions of the purpose and target audience • the topics and performance areas it will cover • the types of items and number to be written for each topic or performance area • scoring • timing • the administration method (paper, browser, CD) • desired reports, distribution of results • decisions made based on results

  37. 0 Blueprint for Nursing Licensure Exam

  38. Tip: Consider options for gathering content from globally dispersed SMEs

  39. Using a distributed authoring and centralized item management with globally dispersed authors 0 Internet

  40. Can you provide multilingual interfaces to SMEs?

  41. Tip: When delivering localized/multilingual assessments, consider the workflow of your translation process.

  42. Efficient Models for Translating Items, Quizzes and Tests

  43. Two Translation Methods Define translation project Translate within an assessment authoring tool Interactive Translations 3rd-party localisation using industry-standard format On-The-Fly

  44. Multilingual Item Banking

  45. Example of localization software: Trados Studio

  46. Tip: Align delivery environment to the stakes of an assessment

  47. Low/High to High/High Stakes Tests What’s at Stake? Higher Stakes Life and Limb Promotion & Jobs & Legal Concern Educational Exams Tests Elearning & Surveys

  48. Issues to Consider • Rationalization • Opportunity • Motivation • Impersonation • Content Theft • Cheating Fraud Triangle Threats

  49. Secure Browser Regular Browser High stakes: Secure Browser • Mitigates risks of: • Cheating • Items being leaked/shared with future test participants • How it works… • No menu or icons • Cannot switch to another program • Disables printing • Can be customized to permit certain functions, i.e. printing feedback page

  50. Other risks to mitigate • Power Outage? • Loss of Internet connection? • Ensure there is an “auto save” or “save as you go” mechanism: • Answers auto-saved as the participant takes test • In the event of technical failure, participants can resume an assessment where they left off

More Related