26 th Annual Management Information Systems (MIS ) Conference February 14, 2013

1. 26th Annual Management Information Systems (MIS) ConferenceFebruary 14, 2013 National Assessment of Educational Progress (NAEP) Observable Data From Computer-Based Assessments Gregory Anderson, Ph.D. Senior Usability Expert Fulcrum IT, LLC

2. Outline • What are observable data? • What can we observe in the data? • How have we used what we’ve observed? • How can we observe more?

3. Outline • What are observable data? • What can we observe in the data? • How have we used what we’ve observed? • How can we observe more?

4. What are Observable Data? • Computer actions recorded into a database • Possible with Computer-Based Assessments (CBA) • Automatic recording of low level information • Initially • Number of key presses • Number of word processing functions (cut, copy, etc.) • Button clicks (Next, Back, etc.) • Screen coordinates (mouse click) • Recently added: • Time stamping • Extended data (keys pressed, etc.)

5. What are Observable Data? 31           Reach End of Tutorial 32           Backspace 33           Delete 34           Clear Answer 35           Mouse Cut 36           Menu Cut 37           Mouse Copy 38           Menu Copy 39           Mouse Paste 40           Menu Paste 41           Mouse Bold 42           Keyboard Bold 43           Menu Bold 44           Right Click Bold 45           Mouse Italic 46           Keyboard Italic 47           Menu Italic 48           Right Click Italic 49           Mouse Underscore 50           Keyboard Underscore 51           Menu Underscore 52           Right Click Underscore 53           Mouse Indent 54           Mouse Outdent 55           Mouse SpellCheck 56           Menu SpellCheck 57           Mouse Misspellings 58           Misspellings Identified 59           Misspellings Corrected 60           Menu Thesaurus 61           Right Click Thesaurus 62           Thesaurus Replacement 63           Item Scroll 64           Response Scroll 65           Stimulus Scroll CBA2011 (Math Computer Based Study MCBS Grade 8 – Adaptive) 1              Enter Item 2              Exit Item 3              Change Volume 4              Change Language 5              Keypress Count 6              Click Choice 7              Click Progress Navigator 8              Keyboard Cut 9              Right Click Cut 10           Keyboard Copy 11           Right Click Copy 12           Keyboard Paste 13           Right Click Paste 15           Text-To-Speech 17           Collapse Prompt 18           Expand Prompt 19           Collapse Response 20           Expand Response 21           Zoom 22           Admin Move-To-Next-Block 23           Hide Timer 24           Show Timer 25           Change Skin 26           Application Recovery 28           Swap Mouse Button 29           Play 30           Pause

6. What are Observable Data? 91           Enter Student Control Panel 92           Leave Student Control Panel 93           Menu Indent 94           Menu Outdent TEL2012 (2012 TEL Tryout Grade 8) 95           Close Word Definition CBA2013 (2013 TEL Pilot Grade 8) 96           Reset Booklet 97           Next 98           Back 99           Yes 100         No 101         OK 102         Media Interaction CBA2012 (2012 WCBA Pilot Grade 4) 66           Drop Choice 67           Keypress 68           Highlight Mode On 69           Highlight Mode Off 70           Highlight Text 71           UnHighlight Text 72           Eliminate Mode On 73           Eliminate Mode Off 74           Eliminate Choice 75           UnEliminate Choice 76           Mouse Select All 77           Keyboard Select All 78           Menu Select All 79           Mouse Undo 80           Keyboard Undo 81           Menu Undo 82           Mouse Redo 83           Keyboard Redo 84           Menu Redo 85           Right Click SpellCheck 86           Mouse Thesaurus 87           Click Word Definition 88           Display Reminder 89           Pause Workstation 90           Resume Workstation

7. Putting the “Data” in Observable Data

8. Observable Data Sources • 2011 Writing Computer-Based Assessment Pilot • Grade 8: 24,281 Students • Grade 12: 28,238 Students • 2011 Math Computer-Based Study • Grade 8: 8,427 Students • 2012 Writing Computer-Based Assessment • Grade 4: 13,005 Students

9. Outline • What are observable data? • What can we observe in the data? • How have we used what we’ve observed? • How can we observe more?

10. What Can We Observe? Easy • Frequency/Proportion (event ID, booklet ID, total number of participants) • End State (event ID, booklet ID, order of events) • Time Course (event ID, time stamp) • Duration/Intervals (event ID, 2 time stamps, math) • Qualitative (event ID, extended data) • Big Picture Qualitative (all data, other data sources) Difficult

11. What Can We Observe? Easy • Frequency/Proportion (event ID, booklet ID, total number of participants) • End State (event ID, booklet ID, order of events) • Time Course (event ID, time stamp) • Duration / Intervals (event ID, 2 time stamps, math) • Qualitative (event ID, extended data) • Big Picture Qualitative (all data, other data sources) Difficult

12. Observing Frequency/Proportion How many times did the student use the mouse to cut? How many times did the student use the [CTRL+X] cut keystroke combination? How many times did the student use the menu to cut? How many times did the student cut using the right-click context menu? How many times did the student use the mouse to copy? How many times did the student use the [CTRL+C] copy keystroke combination? How many times did the student use the menu to copy? How many times did the student copy using the right-click context menu? How many times did the student use the mouse to paste? How many times did the student use the [CTRL+V] paste keystroke combination? How many times did the student use the menu to paste? How many times did the student paste using the right-click context menu? How many times did the student use the mouse to click the bold button? How many times did the student use the [CTRL+B] bold keystroke combination? How many times did the student use the menu to boldface? How many times did the student boldface using the right-click context menu? How many times did the student use the mouse to click the italicize button? How many times did the student use the [CTRL+I] italic keystroke combination? How many times did the student use the menu to italicize? How many times did the student italicize using the right-click context menu? How many times did the student use the mouse to click the underline/underscore button? How many times did the student use the [CTRL+U] underline/underscore keystroke combination? How many times did the student use the menu to underline/underscore? How many times did the student underline/underscore using the right-click context menu? How many times did the student use the mouse to click the spell check button? How many times did the student use the menu to check spelling? How many times did the student check spelling using the right-click context menu? How many times did the student right-mouse click misspelled words? How many times did the spell check identify misspellings?

13. Observing Frequency/Proportion Available: WCBA 2010

14. Observing Frequency/Proportion What Proportion Told Us What Proportion Doesn’t Tell Us Which students used multiple tools If tools were used in some sections more than others When and for how long each tool was used The impact of tool use on performance Exploration vs. need • All tools were triggered at least once by a higher percentage of 8th graders than 12thgraders • Text-to-Speech and scrolling were used by a greater percentage of students than other system tools • Help was used by a smaller proportion of students than other system tools

15. What Can We Observe? Easy • Frequency/Proportion (event ID, booklet ID, total number of participants) • End State (event ID, booklet ID, order of events) • Time Course (event ID, time stamp) • Duration/Intervals (event ID, 2 time stamps, math) • Qualitative (event ID, extended data) • Big Picture Qualitative (all data, other data sources) Difficult

16. Observing End State Available: WCBA 2010 Grade 8 Blue–72% High Contrast–21% Gray–7%

17. Observing End State What End State Told Us What End State Doesn’t Tell Us When they changed to the final state (beginning? end?) How long they spent in the end state Why they changed How many blue users surveyed the other skins first (i.e., kept the default vs. choosing it) • Blue was the end state skin for 72% of participants • Grey was the least used (7%)

18. What Can We Observe? Easy • Frequency/Proportion (event ID, booklet ID, total number of participants) • End State (event ID, booklet ID, order of events) • Time Course (event ID, time stamp) • Duration/Intervals (event ID, 2 time stamps, math) • Qualitative (event ID, extended data) • Big Picture Qualitative (all data, other data sources) Difficult

19. Available: MCBA 2011 Observing Time Course Booklet #6037

20. Observing Time Course What Time Course Told Us What Time Course Doesn’t Tell Us Which words were misspelled Number of words that were not identified as misspelled If spellcheck was used to good effect What was happening in between spellcheck events • When Corrected Misspelling events occurred • How long between Corrected Misspelling events

21. What Can We Observe? Easy • Frequency/Proportion (event ID, booklet ID, total number of participants) • End State (event ID, booklet ID, order of events) • Time Course (event ID, time stamp) • Duration/Intervals (event ID, 2 time stamps, math) • Qualitative (event ID, extended data) • Big Picture Qualitative (all data, other data sources) Difficult

22. Duration(2 time stamps, math) Observing Duration/Intervals

23. Duration(2 time stamps, math) Observing Duration/Intervals • YOU ARE PERMITTED TO USE A CALCULATOR IN THIS SECTION. • This section has 16 questions. Respond to questions on the computer. You will have to select an answer or type your response as directed. In those questions where you type an answer, it is important that your answer be clear and complete since partial credit may be awarded. • Remember: • You are permitted to use a calculator in this section; either your own, or the one provided by NAEP. If you need help using the one provided by NAEP, read the instructions on the back of the scratch paper or look at the quick reference card on the calculator lid. • If the calculator does not work or you do not know how to use it, raise your hand and ask for help. • You will have to decide when to use the calculator. For some questions using the calculator is helpful, and may even be necessary. However, for other questions the calculator may not be helpful. • This section is 25 minutes. If you finish before time is up, you may review any question within this section. You will not be able to go to the next section until 25 minutes are up. • You may use the scratch paper provided, but be sure to enter all of your responses on the computer. • Click NEXT to continue.

24. Duration(2 time stamps, math) Observing Duration/Intervals • YOU ARE NOT PERMITTED TO USE A CALCULATOR IN THIS SECTION. • This section has 16 questions. Respond to questions on the computer. You will have to select an answer or type your response as directed. In those questions where you type an answer, it is important that your answer be clear and complete since partial credit may be awarded. • Remember: • You are NOT permitted to use a calculator in this section. • This section is 25 minutes. If you finish before time is up, you may review any question within this section. You will not be able to go to the next section until 25 minutes are up. • You may use the scratch paper provided, but be sure to enter all of your responses on the computer. • Click NEXT to continue.

25. Observing Duration/Intervals What Duration Tells Us What Duration Doesn’t Tell Us What students were doing while on that page How much of the directions they read How well they understood the directions • Students spent from 15-20 seconds on the directions for different blocks • Students on average spent more time on the directions for VE521278c16 • Senior Usability Experts are really slow readers!

26. What Can We Observe? • Frequency/Proportion (event ID, booklet ID, total number of participants) • End State (event ID, booklet ID, order of events) • Time Course (event ID, time stamp) • Duration/Intervals (event ID, 2 time stamps, math) • Qualitative (event ID, extended data) • Big Picture Qualitative (all data, other data sources) Easy Difficult

27. Observing Qualitative Information • Spelling Examples (includes punctuation) • “befor” vs. “before” • “ninjas” vs. “ninja’s” • Spacing Examples • “monkey.You” vs. “monkey. You” • “books,none” vs. “books, none” • No Correction Examples • “stonger,faster,and” (no alternate spelling suggested) • “orrhjrejfjhhrhgjshjdnewjmfnffjnendndsmsandbfnnwnnfgbnemsnfcnfnsehgfnbsjehdnbvfndbfjxdnnebndfnfbfbfbbdbdnbsnsnnnsnsnsnnssnsnnsnnsnssssssssdjejkwjennbrbbfbbfbfcbfcbcvbbcxbcxblolooooooooooooooooooooooobbxbbx” Identified Error Types

28. Observing Qualitative Information What Qualitative Told Us What It Didn’t Tell Us The impact of the modification on the final output Re-creation of the final product • What was modified • How it was modified • When it was modified

29. Outline • What are observable data? • What can we observe in the data? • How have we used what we’ve observed? • How can we observe more?

30. Major Takeaways • Students used accommodations and system and WP tools substantially during CBAs. • Scrolling: 60%-70% • Text-to-Speech: 53%-72%

31. Major Takeaways • Some accommodations and tools were not used as might be expected • Accommodation – Directions written in Spanish • Used by - 55%-56% of eligible students

32. Major Takeaways Observable data helped to identify and validate usability issues with CBAs. • Misuse of Next button • To open writing panel • To check on time remaining • Pausing tutorials (only 8%-9% of pausers didn’t finish) • Scrolling was not used by 7%-14% of participants on items where it was necessary in order to show all the information.

33. Major Takeaways • Insight into student response patterns • Number of words used for different writing purposes

34. Major Takeaways • Insight into student response patterns • Types of errors identified by spell check

35. Major Takeaways • Insight into student response patterns • Distribution of spellcheck corrections by student

36. Major Takeaways • Insight into student response patterns • Compare silent reading rates

37. Major Takeaways With Calculator Without Calculator YOU ARE NOT PERMITTED TO USE A CALCULATOR IN THIS SECTION. This section has 16 questions. Respond to questions on the computer. You will have to select an answer or type your response as directed. In those questions where you type an answer, it is important that your answer be clear and complete since partial credit may be awarded. Remember: You are NOT permitted to use a calculator in this section. This section is 25 minutes. If you finish before time is up, you may review any question within this section. You will not be able to go to the next section until 25 minutes are up. You may use the scratch paper provided, but be sure to enter all of your responses on the computer. Click NEXT to continue. • YOU ARE PERMITTED TO USE A CALCULATOR IN THIS SECTION. • This section has 16 questions. Respond to questions on the computer. You will have to select an answer or type your response as directed. In those questions where you type an answer, it is important that your answer be clear and complete since partial credit may be awarded. • Remember: • You are permitted to use a calculator in this section; either your own, or the one provided by NAEP. If you need help using the one provided by NAEP, read the instructions on the back of the scratch paper or look at the quick reference card on the calculator lid. • If the calculator does not work or you do not know how to use it, raise your hand and ask for help. • You will have to decide when to use the calculator. For some questions using the calculator is helpful, and may even be necessary. However, for other questions the calculator may not be helpful. • This section is 25 minutes. If you finish before time is up, you may review any question within this section. You will not be able to go to the next section until 25 minutes are up. • You may use the scratch paper provided, but be sure to enter all of your responses on the computer. • Click NEXT to continue.

38. Written Instructions? • Written instructions are not being read for comprehension • Possible Causes / Consequences • The students suffered for not having read the instructions • The students didn’t need the instructions • Some students didn’t need the instructions and skewed the average duration data

39. What Do We Know? • How students use system tools • How students use word processing tools • What keys they are pressing and when • What buttons they are clicking and when • What words are identified by spell check, which are corrected and how they are corrected • How much time students spend on each screen

40. Outline • What are observable data? • What can we observe in the data? • How have we used what we’ve observed? • How can we observe more?

41. Descriptive vs. Experimental • Observable Data are by definition descriptive • Don’t answer causation or other complex questions • Causation needs experimental data • Isolate variables • Manipulate conditions • Compare groups • Complex questions • Pair Observable Data with other information

42. The More You Know, The Less You Know • Simple Question: Can Students Access the Features of CBA? • Simple Answer: Yes • Complex Questions • Did the changes made to interface have any impact on their performance or comfort during assessment? • Did everyone who would have benefited use them? • Do kids even know what benefits them? • What is impact of age/grade on beneficial impact of interface changes?

43. More Questions • Why do some students not use system tools? • Why do some students not use word processing tools? • Which tools result in better performance? • Are there other tools that might prove beneficial? • Do written instructions help? Everyone?

44. Better Questions = Better Answers • Start with questions to be answered • Determine sources of information necessary to answer question (research design) • Obs. Data (OD) + Usability testing • OD + Background Question Answers • OD + Performance Scores • OD + Qualitative Analyses • Interpret interactions among data

45. Data Sources • Observable Data (behavior) • Easy to collect (now) • Every participant (not just a sample) • Many, many parameters (many, many) • Single source • Profile Data (outcomes, background) • Also easy to collect on every participant • Fewer parameters • Multiple sources • Qualitative Data (impact, usability) • Requires interpretation • Difficult to collect • Requires sampling (research design)

46. Data: The Final Frontier Level III – Meaning Multiple variables Interactions among variables Qualitative information Level II – Structure Extended information Time course / Duration Value Level I – Counting Single variable Frequency /Proportion Value

47. Fulcrum IT Services, LLC National Assessment of Educational Progress (NAEP) Observable Data From Computer-Based Assessments Gregory Anderson, Ph.D. Senior Usability Expert Fulcrum IT, LLC