Can Third Graders Type?

1. Can Third Graders Type? An examination of the fairness by grade of requiring constructed-response items on large-scale assessments administered via the computer By Steve Martin Connecticut State Department of Education Connecticut Assessment Forum 8/13/12

2. Good Afternoon Today we will • Practice using responders. • Use responders to get a sense of where the group is. • Review where SBAC is on this question and cover standards and other background on this issue. • Discuss what other states are doing. • Examine typed-response data from a recent CMT pilot test. • Discuss possible contributing factors to trend data. • Review where Connecticut is going in the short-term. • Discuss reporting and further research possibilities. • Q&A • We will finish by exploring some free sample online resources designed to help students learn to type.

3. Practice Using Responders • Start->programs->2know toolbar • To start your responder, first hit the on/off button on the bottom.

4. Practice Using Responders • Start->programs->2know toolbar • To start your responder, first hit the on/off button on the bottom. • You will know it is on when words appear in this screen.

5. Practice Using Responders • Start->programs->2know toolbar • To start your responder, first hit the on/off button on the bottom. • You will know it is on when words appear in this screen. • Wait for the 2Know Toolbar to appear on this screen before you enter the session.

6. Practice Using Responders • Start->programs->2know toolbar • To start your responder, first hit the on/off button on the bottom. • You will know it is on when words appear in this screen. • Wait for the 2Know Toolbar to appear on this screen before you enter the session. • Once you see the toolbar press ENTER. • When you see the network “Steve” on your responder, press ENTER again. • Once everyone is on we will practice this ABC question. When instructed, press A if you are MALE, and press B if you are FEMALE; and then press ENTER.

7. Question #1 Q: What age were you when you first learned to type? • Age nine or younger • Age 10 -12 • Age 13 -15 • Age 16 -17 • Age 18 or older

8. Question #2 Please answer this question only if you have children of your own. Q: At what age was your child(ren) when he/she/they first learned to type? • Age nine or younger • Age 10 -12 • Age 13 -15 • Age 16 -17 • Age 18 or older

9. Question #3 If you have a child or children who are at least 13 years of age, (or at least grade 8) please answer this question about typing speed. Q: I think that my child (or my children) can type… • Faster than I can. • About the same speed as I can. • Not as fast as I can. • I don’t know how fast my child(ren) can type.

10. Question #4 Please answer this question if you have recent experience with third graders using the computer. Q: Which of the following is the fairest assessment of typical third-grade computer skill? • They cannot use the keyboard or mouse with any reliability. • They can use the mouse to point and click reliably • They can use the mouse and keyboard reliably, but their typing is very slow. • They can use the mouse and keyboard reliably and they can type words using the computer keyboard at least as fast as they can write with a pencil.

11. Question #5 Q: Given that the statewide test will be administered to third graders via computer beginning in the spring of 2015, which of the following test environments for them would you currently support? • Third graders should only need to point and click. • Third graders should point and click AND type responses to short-answer items. • Third graders should point and click, type responses to short answer items, AND type responses to extended response items (like the CMT writing sample).

12. Where SBAC Stands Steve: What research is planned on this issue? SBAC • Cognitive lab research is planned to learn how students across a range of grades approach and interact with different types of computer-administered items. • This is part of a larger effort to eliminate or minimize factors that could distract students from the ability to demonstrate their knowledge and skills. • For more information, please see the SBAC accessibility guidelines

13. Where SBAC Stands Steve: What about keyboarding skills specifically? SBAC: The Common Core State Standards emphasize proficiency with technology. For example, the 4th grade writing standard calls for students to “demonstrate sufficient command of keyboarding skills to type a minimum of one page in a single sitting.” Administering assessments by computer is a part of ensuring that schools and teachers can measure mastery of the full range of the standards.

14. Technology Standards for Grade 3 Students What exactly are the CCSS Standards on the keyboarding issue for grades 3 and 4? CCSS: • Grade 3 Standard—W.3.6—With guidance and support from adults, use technology to produce and publish writing (using keyboarding skills) as well as to interact and collaborate with others. • Grade 4 Standard—W.4.6.—With some guidance and support from adults, use technology, including the Internet, to produce and publish writing as well as to interact and collaborate with others; demonstrate sufficient command of keyboarding skills to type a minimum of one page in a single sitting.

15. What is Age Appropriate?Differing perspectives “There is no right or wrong time to introduce your child to the family computer. Much like deciding when to buy the first bike, or when your child can go out alone with friends, the choice will be based on your parenting style, your values and, most importantly, your child’s readiness.” Source: http://familyinternet.about.com/od/introtofamilycomputing/a/growthchart.htm

16. What is Age Appropriate?Differing perspectives What about computers and Waldorf education? Source: http://www.whywaldorfworks.org/02_W_Education/faq_about.asp “Waldorf teachers feel the appropriate age for computer use in the classroom and by students is in high school. We feel it is more important for students to have the opportunity to interact with one another and with teachers in exploring the world of ideas, participating in the creative process, and developing their knowledge, skills, abilities, and inner qualities. Waldorf students have a love of learning, an ongoing curiosity, and interest in life. As older students, they quickly master computer technology, and graduates have successful careers in the computer industry.”

17. General Typing Speed Data Source: Lynda S. Hartman rev 1/20/05 Permission to use if credit is retained. lshartman1@gmail.com

18. General Typing Speed Research • Researchers suggest that 10-20 wpm is sufficient in later elementary school (grades 3 – 6). • Some teachers, though, have made lower targets: • Others made their goal to be 2-3 times the students handwriting speed. 2002-2003 Study In the Illinois School District U - 146 Source: Lynda S. Hartman rev 1/20/05 Permission to use if credit is retained. lshartman1@gmail.com

19. General Typing Speed Research • Task Force Report: • No recommendations for keyboarding speed until grade 8. • At this point, though, students are expected to type at a rate of 30 WPM with high accuracy. Buhler, Kansas USD – 313 Conducted Research of student typing speed. Source: Lynda S. Hartman rev 1/20/05 Permission to use if credit is retained. lshartman1@gmail.com

20. General Typing Speed Research Utah State Office of Education Utah provides a detailed Web site in which keyboarding speed requirements are laid out very explicitly: http://www.schools.utah.gov/cte/keyboarding_standards.html Source: Lynda S. Hartman rev 1/20/05 Permission to use if credit is retained. lshartman1@gmail.com

21. What Are Other States Doing? I asked states that were listed in a State Educational Technology Directors Association 2011 report as having computer delivered assessmentsalready the following six survey questions: • Is your state’s summative test administered mostly via computer at any grade? • Is your state’s summative test administered mostly via computer to grade 3 students? • For how many years has your state allowed for grade 3 students to be administered a large-scale exam on computer? • Are there any computerized, grade 3, open-ended items requiring a keyboarded short-answer response on the statewide exam?

22. What Are Other States Doing? Survey Questions 5 and 6: • Are there any grade 3, open-ended items requiring a keyboarded extended-answer response on the statewide exam? (For purposes of this question, consider an “extended” response as typically requiring more than two sentences to receive the highest score.) • Has the issue of keyboard competence for younger students in any way shaped your state’s decisions in the selection of computerized item-types?

23. What Are Other States Doing? • 9/12 state’s surveyed administer some portion of their summative test via computer. • For Georgia, the computer version is an option for the grade 3 reading retest only. • Oklahoma reports that there is no grade 3 computer assessment in the state at all. • Texas administers an assessment in grades 2-12 online. TELPAS (Texas English Language Proficiency Assessment System) is for LEP students only. That is the only online assessment at grade 3 in Texas. • Is your state’s summative test administered mostly via computer at any grade?

24. What Are Other States Doing? • Is your state’s summative test administered mostly via computer to grade 3 students? • 7/12 state’s surveyed administer their summative test via computer to grade 3. • For Georgia, the computer version is an option for the grade 3 reading retest only. • Texas LEP students only. • Limited to the Direct Assessment of Writing. The rest of the Westest 2 is administered via paper and pencil at this grade. • First draft can be done in pencil, but then it must be copied into the computer for a score.

25. What Are Other States Doing? • For how many years has your state allowed for grade 3 students to be administered a large-scale exam on computer? • 9/12 state’s surveyed have administered some test to grade 3 students for 2 years or more. • Maryland gave a small number of students a modified assessment online, but they no longer administer a modified exam. • Kansas has been at this the longest having six years of computer-delivered testing. • Georgia makes this an option for grade 3 retesters. • Indiana—not the summative test.

26. What Are Other States Doing? • Are there any computerized, grade 3, open-ended items requiring a keyboarded short-answer response on the statewide exam? • Only 2/12 state’s surveyed require keyboarded short-answer responses from students at grade 3. • Hawaii gave a tentative “Yes” to this question. This still needs confirmation. • Minnesota administers the MCA in Mathematics to 80% of the students. The short-answer items are only numerical. MN ran cognitive labs and decided to limit students to just the numbers. • West Virginia only gives the Writing Sample online. No short-answer open-ended.

27. What Are Other States Doing? • Are there any grade 3, open-ended items requiring a keyboarded extended-answer response on the statewide exam? • Only 2/12 state’s surveyed require keyboarded extended-answer responses from students at grade 3. • Hawaii gave a tentative “Yes” to this question. This still needs confirmation. • West Virginia direct assessment of writing is keyboarded at grades 3-11, the rough draft can be by paper/pencil but final must be keyboarded by grade 3 students. When surveyed, 85% of 3rd graders said that they preferred the online version of the test over the traditional paper and pencil test.

28. What Are Other States Doing? • Has the issue of keyboard competence for younger students in any way shaped your state’s decisions in the selection of computerized item-types? • 6/12 state’s surveyed have made decisions based, at least in part on, keyboard competence. • Hawaii gives students three opportunities to test. • Kansas has statewide standards for keyboarding that begin in kindergarten. • MD moved away from administering online. • MN’s Cognitive Labs indicated that they should limit keyboarding to just the numbers. • Nebraska does paper/pencil until grade 4. • West Virginia did a survey, 85% of 3rd graders said that they preferred the online version of the test over the traditional paper and pencil test.

29. A Question for Your Responder Given what you just learned about the experience in other states… Q: which of the following test environments for third-grade students would you currently support? • Third graders should only need to point and click. • Third graders should point and click AND type responses to short-answer items. • Third graders should point and click, type responses to short answer items, AND type responses to extended response items (like the CMT writing sample).

30. March 2012 CMT DAW Pilot Test • Over a thousand CT students per grade (range 1,381 – 1,581) participated in an online Direct Assessment of Writing pilot which required them to keyboard their responses. • These responses are not reported back to schools or students. They don’t “count.” • The purpose of the pilot test is to determine the difference in difficulty between the two modes. If the two modes can be “post-equated” then a choice of modes can be offered before the 2015 online test. • The tests were recently scored. Both the pilot scores, and the observations of the scoring leads at the various Measurement Incorporated scoring centers were reported to the state.

31. Anecdotal Evidence From the Scoring Center Mathematics: (Marianne Nasiatka, Grade 3 Scoring Director) We saw evidence that some third graders don’t know how to use the shift key. An equal sign would appear in their responses instead of an addition symbol.

32. Anecdotal Evidence From the Scoring Center Reading and Science (Steve Zrimec, project monitor) The quality of the typewritten responses appeared to be equal to the handwritten responses from past projects. Keyboard unfamiliarity presented no obvious problems. There did not appear to be any obvious advantages for the students (i.e. spelling, grammar, and even ideas weren’t improved because of the format). And actually, this seemed true across all grades, not just Grade 3.

33. Anecdotal Evidence From the Scoring Center Direct Assessment of Writing (Sharon Williams, Grade 3 scoring director) We were pleasantly surprised at the typing skills.  There were few responses that indicated that the students did not know how to type.  In other words, we saw very few "fingers on the wrong keys" gibberish responses but rather a fairly competent sample. Readers noted (as did I) that an unusually high number of responses ended abruptly, as if the writers had run of time before finishing their narratives. We wondered if this was because the third graders are unable to type as fast as they can write. We also noted that many of the responses seemed brief and undeveloped compared to the responses we saw in the spring assessment. Very few typed responses went on to the second page, and many were only a half page long.  This could be an issue with time and typing speed too.

34. A Question for Your Responder Now that you have heard of the observations from the Scoring Centers… Q: which of the following test environments for third-grade students would you currently support? • Third graders should only need to point and click. • Third graders should point and click AND type responses to short-answer items. • Third graders should point and click, type responses to short answer items, AND type responses to extended response items (like the CMT writing sample).

35. CMT Typed Response Data S c o r e P o i n t Grade

36. CMT Typed Response Data S c o r e P o i n t Grade

37. CMT Typed Response Data S c o r e P o i n t Grade

38. CMT Typed Response Data S c o r e P o i n t Grade

39. CMT Typed Response Data S c o r e P o i n t Grade

40. CMT Typed Response Data S c o r e P o i n t Grade

41. CMT Typed Response Data S c o r e P o i n t Grade

42. Possible Contributing Factors to This Trend • Were the piloted (keyboarded) prompts more difficult? • Was the novelty of computer testing a factor? • Are the scorer’s showing a bias in favor of handwritten text? • Because of a lack of clear standards on the pilot, did the time of administration of the pilot generally rob students of the full 45-minutes intended? • If time on task is a factor, will lengthening the standard time limit for a handwritten response be sufficient to overcome the differences for students who are typing? • Is keyboard familiarity (typing skill) a contributing factor?

43. A Question for Your Responder Now that you have seen the data from the CMT DAW Pilot Test… Q: which of the following test environments for third-grade students would you currently support? • Third graders should only need to point and click. • Third graders should point and click AND type responses to short-answer items. • Third graders should point and click, type responses to short answer items, AND type responses to extended response items (like the CMT writing sample).

44. Connecticut’s Short-Term Plan • The Department would like to encourage equipped districts to administer more statewide tests on the computer before 2015. • Pilot data will show which of the test sessions can be equated to take into account the different difficulties of taking tests on the computer. • The Department plans to give districts some level of choice about which mode (paper-and-pencil vs. computer submitted) for students taking the census test.

45. Connecticut’s Short-Term Plan • Because selected forms will be equated, this means that a resulting scale score for the same test may have a different raw score as a result of mode. (written vs. computer submitted) =

46. Further Research • RESPONDER TIME: Should these CT findings be reported to SBAC? Please use “TRUE” for YES and use “FALSE” for NO. • A repeated study could be done to see if the patterns are persistent over time. (Students may be improving on typing skill each year.) • It is possible to design a factor analysis to try to find out how much of the difference between the two modes is attributable to typing skill alone.

47. Q & A

48. Online Resources • http://teachersites.schoolworld.com/webpages/yollis/typing.cfm Mrs. Yollis’ blog (above) sends you to these fine resources: • Steve’s Review: • Learn to type AND Dance Mat Typing Game (same site)— By the BBC the best one here. • I learn Tech…Website—hit the drop down “typing games.” the best ones are: • typing chef • keyboard climber game • super hyper • I also got addicted to “Typing Defense” • Typer Shark game???—could not find it nor “Typing Lessons”

49. Dance Mat Typing • http://www.bbc.co.uk/schools/typing/ • Sequential lessons • Animal guides • Reward Screens • 10-minute lessons • 20 small typing tasks. • About a dozen lessons in the sequence.

50. Typing Chef • http://www.sense-lang.org/typing/games/typingchef.php • Type the word in the bubble before it pops. • In later, more challenging screens other food falls into the pan. • Game works on speed of typing target words that get increasingly complex.