Thinking about thinking: Methods for the study of online reading comprehension

1. Thinking about thinking: Methods for the study of online reading comprehension

2. Dr. Donald J. Leu Part I Introduction & Theoretical Frame

3. More than a billion people are reading online today

4. The RAND Reading Study Group concluded: • “… accessing the Internet makes large demands on individuals’ literacy skills; in some cases, this new technology requires readers to have novel literacy skills, and little is known about how to analyze or teach those skills.” (p. 4)

5. Our Strategy • A continuously, iterative method within a collective team approach. • Based on extent research in multiple areas of inquiry as well as our own surveys, interviews, observations of online readers, and other data. • Each of us, bringing our own questions, methods, and results, continuously challenge our working theoretical definition as we seek to refine and expand it.

6. Current theoretical definition of online reading comprehension • The new literacies of online reading comprehension include the skills, strategies, and dispositions necessary to … use the Internet and other ICT to (1) identify important questions, (2) locate information, (3) analyze the usefulness of that information, (4) synthesize information to answer those questions, and then (5) communicate the answers to others.

7. You will see a team approach to inquiry in this area, where all members of our research team are valued as equal colleagues in the research enterprise.

8. The important next question is: • How might we build an even more powerful, collective team approach to research in order to better understand the nature of online reading comprehension?

9. Jill Castek Part II The Changing Nature of Reading Comprehension: Examining the Acquisition of New Literacies in a 7th Grade Science Classroom

10. The Issue • How can we examine the acquisition of new literacies in a 7th grade science classroom where the Internet has been integrated into instruction for a school semester?

11. Create • Online problem-solving tasks requiring students to demonstrate their ability to: • 1) locate information • 2) critically evaluate information • 3) synthesize information from multiple sources, • 4) communicate the results efficiently to others (Leu, Kinzer, Coiro, & Cammack, 2004).

12. Post • Tasks on a secure, educationally structured blog • Typepad (http://www.typepad.com/) • Edublogs (http://edublogs.org/) • Blog interface prompts authentic communication http://newliteracies.typepad.com/science_exchange/)

13. Observe & Record • As students individually complete tasks • List observations on a specialized rubric designed to evaluate the tasks. • Observations translate into scores that determine students’ proficiency level in online reading comprehension

14. Archive Data • A real time screen capture program called Camtasia is useful for capturing process data. • This program records audio as well as students’ screen movements as they complete tasks. • Video data can be examined multiple times to achieve inter-rater reliability between researchers. • Responses and processes can be analyzed further to identify patterns in strategy use. Link to Camtasia Video

16. Video Examples and Score Sheets Snapshots of Three Online Readers http://www.newliteracies.uconn.edu/reading.html

17. Discussion • How can the collection of data in these forms enhance our understanding of online reading comprehension?

18. Julie Coiro Part III Measuring the Nature of Online Reading Comprehension with ORCA-Scenario I and II

19. The Issue • How can we scale up to reliably measure performance in two parallel online reading comprehension scenarios with larger groups of students (n=120) in an online quiz interface? • Purpose (mixed-methods): What factors contribute to performance on a scenario-based measure of online reading comprehension? • Are there new proficiencies over and above general reading comprehension and domain-specific prior knowledge? • If so, what patterns emerge among skilled and less-skilled readers?

20. Methods: Designing Test Items • Adapt the ORCA-Blog to design two parallel measures of online reading comprehension (ORCA-Scenario I and ORCA-Scenario II) as part of an “Internet Treasure Hunt” created by fictitious 7th graders requesting online resources about the respiratory system and carbon monoxide poisoning • Designing parallel items that independently measure each online reading proficiency (see handout)

21. Online Reading Comprehension Assessment (ORCA) Scenario I

22. Online Reading Comprehension Assessment (ORCA) Scenario II

24. Methods: Designing Protocols • Develop a standardized protocolfor moving multiple levels of students through the online quiz interface and recording observations. • 12 students at a time (in 3 cycles every 20 minutes) • Time limits for completing each task (15/20/20/10) • Reviewing directions individually and adjusting prompts for diverse range of students • Use field notes to track the time, what each student completed, actions (e.g., writing on paper), and difficulties (e.g., questions) • Start, stop, and reset Camtasia while clearing browser settings for each new user

25. Methods: Designing Scoring Rubrics • Develop a scoring rubricthat integrated both process and product • Rubric item scores are totaled to reflect process and product in a way that can be used in quantitative analyses (e.g., comparing means, correlations, factor analysis, predicting the role of isolated variables, etc.)

26. Methods: Designing Scoring Rubrics • Develop a scoring rubricthat integrated both process and product • Actions noted on score sheet and integrated with Camtasia process recordings and strategy interview transcripts in a way that can be used for qualitative analyses of patterns • within readers • across multiple tasks or sessions • across multiple readers

27. Methods: Capturing Reading Actions

28. Scoring locating efficiency as part of online reading comprehension

29. Scoring how deeply readers read for accuracy or how efficiently they communicate information to others

30. Methods: Integrating and Analyzing Data Sources • Develop a procedurefor integrating multiple types of data into one transcript • Student name (pseudonym) • Item and task number (similar items across tasks) • Video action (from testing session) • Researcher prompts (from follow-up stimulated recall interview) • Student think-aloud (from follow-up stimulated recall interview) • Field notes and beginning interpretations

31. Notes and Interpretations Student ID Test Item and Task Number Video Actionsone action or a series of actions Think-AloudColor Code researcher and student

32. Methods: Accomplishments • Successfully scaled up to measure more students with one researcher in online quiz interface • Developed a second valid and reliable measure of online reading comprehension (parallel forms - Scenario I and II) • Developed a standardized protocol for administrationand follow-up strategy interview • Designed a preliminary method of combining multiple data sources of process and product data for multiple analyses • Next question…What do we lose (or gain) by measuring only the “products” of online reading comprehension as compared to both product and process?

33. Laurie A. Henry Part IV Streamlining the Assessment Process for Measuring Online Reading Comprehension

34. The Issue How can we streamline the assessment of online reading comprehension skills and strategies and develop a measurement tool that is highly correlated to the previously utilized performance-based assessments?

35. Develop A Survey of Internet Use and Online Reading

36. Administer Seventh grade students in partner schools at research sites in Connecticut and South Carolina (n = 1,025)

37. Forced Response Open-ended Response Analyze Analysis of student responses to determine skill-level for Internet reading comprehension.

38. Scoring • Rubrics developed to score open-ended skill items • Inter-rater reliability established between four scorers (2 at each research location) with  = .87

39. Sample Student Responses • (0) I picked this site because it seems to tell the most information. • (1) i picked my answer because the website says i will learn about history, language, and culture of ancient Egypt • (2) you would choose the ancient egypt site because one is a site that has other peoples reports, another one is about travling to egypt, and the last one does not give you a lot of info • Less than 5% of the students reported that they check the information they read online for accuracy

40. Important Next Question Can we develop a survey instrument that is a good measure of online reading comprehension and is highly correlated with the ORCA, yet requires minimal effort for administration and scoring?

41. Lisa Zawilinski Part V Measuring Pre-service Teacher Knowledge of the New Literacies

42. Create • Survey of Online Teaching & Learninghttp://www.newliteracies.uconn.edu/carnegie/TeacherEd.html • Course Internet Survey-Written Responses (scenarios)http://www.newliteracies.uconn.edu/carnegie/TeacherEd.html

43. Administer • Fifty-two pre-service teachers (38 seniors, 14 graduate students) in an integrated bachelors/masters program at a public university in the Northeast participated in the study • Survey & scenarios administered separately, at the start then again at the end of the methods course

44. Archive & Analyze

45. Important Next Question How can we effectively integrate new literacies instruction within teacher preparation programs?

46. Dr. Douglas Hartman Part VI Open Conversation: What are the most important next questions and how might they be studied?

47. Questions You know our important next questions. What are yours? Are there other questions that we need to consider as well?

49. Visit us on the web:http://newliteracies.uconn.edu