The Problem of Comprehension:

2. The Problem of Comprehension: A Potential Barrier to Reading Curriculum-Based Measurement as a Progress Monitoring and Screening Instrument Mark R. Shinn, Ph.D. Professor and School Psychology Program National Louis University, Skokie, IL markshinn@icloud.com http://markshinn.org February 21st, 2013

3. Editor and Contributor to 2 Major Texts on CBM My Area of Expertise 1 of 6 members of Technical Review Panel, National Center for Student Progress Monitoring, USDE/OSEP 2003-2007 Author of More than 75 Refereed Journal Articles and Book Chapters on the Topic of CBM, Progress Monitoring, and Screening

4. Disclosure Disclosure Mark R. Shinn, Ph.D. Serves as a Paid Consultant for Pearson Assessment for their AIMSweb product that provides CBM assessment materials and organizes and report the information from 3 tiers, including RTI. He provides technical support and training. Mark R. Shinn, Ph.D. Serves as a Consultant for Cambium/Voyager/Sopris for their Vmath product, a remedial mathematics intervention but has no financial interests. He helped them develop their progress monitoring system. Mark R. Shinn, Ph.D. Serves as a Consultant for McGraw-Hill Publishing for their Jamestown Reading Navigator (JRN) product and receives royalties.He helped them develop their progress monitoring system. Mark R. Shinn, Ph.D. Serves as a Member of the National Advisory Board for the CORE (Consortium on Reaching Excellence) and receives a stipend for participation. He provides training and product development advice.

5. Learning to Read is Critical for School (and Life) Success

6. Frequent Formative Assessment Like Weekly Use of Reading Curriculum-Based Measurement (R-CBM) is Among our Most Powerful Tools! And the Number 1 Most Powerful TEACHING Variable Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York, NY: Routledge.

7. BUT... What about comprehension? What about “word callers?” I know this kid who...reads “fluently” but doesn’t understand a THING they read...

8. Presentation is Based on the Following White Paper in PRODUCTION • Will Be Available in • pdf format • iBook format A “glossy” and official Pearson version will be finished soon and sent to you. Shinn, M.R. (2013). The problem of comprehension: A potential barrier to Reading Curriculum-Based Measurement (R-CBM) as a progress monitoring and screening instrument. Minneapolis, MN: Pearson Assessment.

9. References on CBM Criterion Related Validity Deno, S. L., Marston, D., Shinn, M. R., & Tindal, G. (1983). Oral reading fluency: A simple datum for scaling reading disability. Topics in Learning and Learning Disability, 2, 53-59. Fuchs, L. S., Fuchs, D., Hosp, M. K., & Jenkins, J. R. (2001). Oral reading fluency as an indicator of reading competence: A theoretical, empirical, and historical analysis. Scientific Studies of Reading, 5, 239-256. Fuchs, L. S., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special Education, 9, 20-28. Hamilton, C., & Shinn, M. R. (2003). Characteristics of word callers:  An investigation of the accuracy of teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240. Miura Wayman, M., Wallace, T., Ives Wiley, H., Ticha, R., & Espin, C. (2007). Literature synthesis on curriculum-based measurement in reading. The Journal of Special Education, 41, 85-120. Shinn, M. R., Good, R. H., Knutson, N., Tilly, W. D., & Collins, V. (1992). Curriculum-Based reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21, 458-478.

10. Accessing Reading Materials markshinn.org • Click on the Downloads for Professionals Icon • Click on the Presentations and Handouts Folder • Click on AIMSweb Webinar R-CBM and the Problem of Comprehension (Sponsored by Pearson) 2013 Folder 10

11. Big Ideas • When students read passages aloud for 1 minute, we are obtaining a holistic assessment of general reading skills. • A strong empirical relation has been demonstrated between R-CBM and measures of generalreading and reading comprehension (RC). • Persuasive empirical data are not always persuasive of teacher opinion. • General reading skills are necessary, but not sufficient for comprehension. • RC is not a single “thing."  Judgments are based on: • What a student is expected to read. • How the student will be assessed. • RC is influenced by a number of important variables, including • General reading ability • Language, especially vocabulary and familiarity with cultural idioms • Knowledge, especially content knowledge and knowledge of the world • Metacognitive strategies, including motivation and interest

12. 2 Other Big Ideas We Can’t Discuss Today • If educators are concerned about specific student’s RC, then we must have a systematic assessment process to address these concerns, starting with ruling out general reading skill deficits, but also including interviews, observations, and diagnostic assessment. • See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 671-698). Bethesda, MD: National Association of School Psychologists. • If students have low general reading skills, RC strategy intervention should be time limited and the use of such strategies should be actively taught, expected, supported, and rewarded in Tier 1, general education and/or content area classes. • See: Willingham, D.T. (2006/07). The usefulness of Brief instruction in reading comprehension strategies. American Educator, Winter, 39-50.

13. 2 Other Big Ideas We Can’t Discuss Today • If educators are concerned about specific student’s RC, then we must have a systematic assessment process to address these concerns, starting with ruling out general reading skill deficits, but also including interviews, observations, and diagnostic assessment. • See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 671-698). Bethesda, MD: National Association of School Psychologists. • If students have low general reading skills, RC strategy intervention should be time limited and the use of such strategies should be actively taught, expected, supported, and rewarded in Tier 1, general education and/or content area classes. • Willingham, D.T. (2006/07). The usefulness of Brief instruction in reading comprehension strategies. American Educator, Winter, 39-50.

14. The Problem of “Fluency” Experienced--and Often Powerful--Teachers Push Back When R-CBM is (Inappropriately) Interpreted and Communicated as “Fluency” How Did This Happen? • Errors in Judgment(s), Confusing Constructs with Behaviors Tested • Tendency toward Reductionism

15. An Example of Reading Curriculum-Based Measurement (R-CBM) It was a pretty good composition. I felt proud knowing it was the best one at my school. After I’d read it five times, I was impatient to start reading it out loud. I followed the book’s directions again. First I read the composition out loud without trying to sound impressive, just to hear what the words sounded like. Billy, 4th Grader

16. ALL These Skills General Reading Skill Oversimplifying the NRP Strands are NOT BOXES- Low Scores “in the Box” Led to TEACHING the Things in the Box SEPARATELY

17. Consequences of (Mis) Interpretation as FluencyTeacher Push Back and Shallow Intervention

18. Research Evidence of “Fluency-Driven” Intervention or “Reading Faster”? Implementing repeated reading and wide reading interventions without more formative intervention is not likely to be valuable(p. 9) Wexler, J., Vaughn, S., Roberts, G., & Denton, C. A. (2010). The efficacy of repeated reading and wide reading practice for high school students with severe reading disabilities. Learning Disabilities Research & Practice, 25, 2-10. Our results indicate that repeated reading does not qualify as an evidence-based or promising practice for students with or at risk for learning disabilities (p. 276) Chard, D. J., Ketterlin-Geller, L. R., Baker, S. K., Doabler, C., & Apichatabutra, C. (2009). Repeated reading interventions for students with learning disabilities: Status of the evidence. Exceptional Children, 75, 263-281.

19. Potentially Severe Educational Need A Student with a Significant Reading Discrepancy

20. Poor Reading Quality as Well as Quantity • Reads Accurately? • Reads Efficiently with Automaticity? • Reads with Expression (Prosody)? • Effective Strategy for Unknown Words? • Errors Distort or Preserve Meaning? • Self Corrects Errors (Comprehension Self-Monitoring)? • Adjusts Pace When Text Difficulty Changes?

21. A Poor High School Reader Grade 8 Material < 10th percentile at beginning of Grade 8 55

22. It’s Not an Empirical Issue--The Evidence is Compelling and Overwhelming Fuchs, L.S., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special Education, 9, 20-28.

23. .90 .88 .85 .71 .72 It’s Not an Empirical Issue--The Evidence is Compelling and Overwhelming Shinn, M.R., Good, R.H., Knutson, N., Tilly, W.D., & Collins, V. (1992). Curriculum-Based reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21(3), 458-478.

24. But, I KNOW and HAVE SEEN Word Callers!!!!!!! The Research The idea of word callers has gained popularity despite a lack of evidence that applies “to an appreciable number of poor readers” (Stanovich, 1986, p. 372). Stanovich, K.E. (1986). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 21, 360-406.

25. But, I KNOW and HAVE SEEN Word Callers!!!!!!! The Practice 31 of 75 Third Grade Teachers responded to a postcard put in their box asking them “if you teach a third-grade student who can read fluently, but has difficulty comprehending text.” 41%! Hamilton, C., & Shinn, M.R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240.

26. But, I KNOW and HAVE SEEN Word Callers!!!!!!! Teachers Identified PAIRS of Kids in Their Classrooms: Teachers Predicted BOTH Pair’s R-CBM, Maze, and Oral Comprehension Question Answering Skills--THEN Both Pairs WERE TESTED Hamilton, C., & Shinn, M.R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240.

27. The “Word Caller”

28. Peers Peers WC WC Teachers Were Partially Right! Almost 2 Standard Deviations Lower Almost 1.3 Standard Deviations Lower WCs “Comprehended” Less Well But WRC Was ALSO MUCH Lower

29. So Where Do This Leave Us... • Reading Comprehension is Complex and We Tend to Oversimplify and Overgeneralize • Reading is Necessary, But Not Sufficient for Understanding • Reading Comprehension Comes From Reading, But is MORE Than Reading

30. “Comprehension” is Oversimplified A-Rod hit the cover off of the ball, but ended the game with a 6-4-3 double play. Inferential: Why would people from Beantown celebrate this? Why would this event mean different things in June than October? Factual: Who is A-Rod? What does “6” mean?

31. Another Common Example The procedure is actually quite simple. First, you arrange things into different groups. Of course, one pile may be sufficient depending on how much there is to do. If you have to go somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set. It is important not to overdo things. That is, it is better to do too few things at once than too many. In the short run, this may not seem important, but complications can easily arise. A mistake can be expensive as well. After the procedure is completed, one arranges the materials into different groups again. Then they can be put into their appropriate places. Eventually, they will be used once more and the whole cycle will have to be repeated. At first, the whole procedure will seem complicated. Soon, however, it will become just another fact of life. It is difficult to foresee any end the necessity for this task in the immediate future, but then one can never tell. 173-word paragraph with 14 sentences of words that are very short (about 4 letters) and rated as late Grade 6 by Flesch-Kincaid

32. Let Me Add Some Information How to Wash Clothes The procedure is actually quite simple. First, you arrange things into different groups. Of course, one pile may be sufficient depending on how much there is to do. If you have to go somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set. It is important not to overdo things. That is, it is better to do too few things at once than too many. In the short run, this may not seem important, but complications can easily arise. A mistake can be expensive as well. After the procedure is completed, one arranges the materials into different groups again. Then they can be put into their appropriate places. Eventually, they will be used once more and the whole cycle will have to be repeated. At first, the whole procedure will seem complicated. Soon, however, it will become just another fact of life. It is difficult to foresee any end the necessity for this task in the immediate future, but then one can never tell.

33. The Bigger Deficits Here • Motivation & Engagement • Active Reading Strategies • Monitoring Strategies • Fix-Up Strategies •Life Experience • Content Knowledge • Activation of Prior Knowledge • Knowledge about Texts • Oral Language Skills • Knowledge of Language Structures • Vocabulary • Cultural Influences Fluency* We Refer to It as General Reading Skills Language Metacognition Knowledge Reading Comprehension • Prosody • Automaticity/Rate • Accuracy • Decoding • Phonemic Awareness And Here Oral Reading is the EASIEST to Measure--Let’s Get This Down and Add MORE Tools And the MOST Unmotivated Here For Some, the Hardest Thing They’ll Ever Do The Easiest Thing To Teach The Longer It Takes... *modified slightly from presentations by Joe Torgesen, Ph.D. Co-Director, Florida Center for Reading Research; www.fcrr.org

34. Comes From Reading, But Is Not Reading Reading Early, Well, and Wide (EWW) is Critical for Reading Comprehension. See: Hunter, P.C. (2012). It's not complicated! What I know for sure about helping our students of color become successful readers. New York, NY: Scholastic. Cunningham, A.E., & Stanovich, K.E. (1998). What reading does for the mind. American Educator, 8-15. Denton, C.A., & Vaughn, S. (2010). Preventing and remediating reading difficulties: Perspectives from research. In M. R. Shinn & H. M. Walker (Eds.), Interventions for achievement and behavior problems in a three-tier model, including RTI (pp. 469-500). Bethesda, MD: National Association of School Psychologists. Linan-Thompson, S., & Vaughn, S. (2007). Research-based methods of reading instruction for English language learners Grades K-4. Alexandria, VA: Association for Supervision and Curriculum Development. http://www.youtube.com/watch?feature=player_embedded&v=RiP-ijdxqEc

35. Comes From Reading, But Is Not Reading Very Few Students Receive Evidence-Based or Sufficiently Explicit/Systematic Comprehension Strategy Instruction, Especially in Navigating Expository/Informational Text and Vocabulary, Learning the “Right Words” the “Right Way.” This is an INSTRUCTIONAL PROBLEM, Not An Assessment Problem! See: Feldman, K., & Kinsella, K. (2005). Narrowing the language gap: The case for explicit vocabulary instruction. In Scholastic (Ed.), Read About: . New York, NY Kamil, M.L., Borman, G.D., Dole, J., Kral, C.C., Salinger, T., & Torgesen, J. (2008). Improving Adolescent Literacy: Effective Classroom and Intervention Practices: A Practice Guide. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Educational Sciences, U.S. Department of Education. Marchand-Martella, N.E., Martella, R.C., Modderman, S.L., Petersen, H., & Pan, S. (2013). Key areas of effective adolescent literacy programs. Education and Treatment of Children, 36, 161-184. Torgesen, J., Houston, D.D., Rissman, L.M., Decker, S.M., Roberts, G., Vaughn, S., . . . Lesaux, N. (2007). Academic literacy instruction for adolescents: A guidance document from the Center on Instruction. Portsmouth, NH: RMC Research Corporation, Center on Instruction.

36. We Need to Expand Our Assessment Repertoire We Need to Add a Variety of Ways to Authentically Assess Comprehension in the Materials We Expect Students to be Able to Read and UNDERSTAND See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology IV (pp. 671-698). Bethesda, MD: National Association of School Psychologists. Howell, K.W., & Nolet, V. (1999). Curriculum-based evaluation: Teaching and decision making (3rd ed.). Atlanta, GA: Wadsworth Publishing Company.

37. Big Ideas • When students read passages aloud for 1 minute, we are obtaining a holistic assessment of general reading skills. • A strong empirical relation has been demonstrated between R-CBM and measures of generalreading and reading comprehension (RC). • Persuasive empirical data are not always persuasive of teacher opinion. • General reading skills are necessary, but not sufficient for comprehension. • RC is not a single “thing."  Judgments are based on: • What a student is expected to read. • How the student will be assessed. • RC is influenced by a number of important variables, including • General reading ability • Language, especially vocabulary and familiarity with cultural idioms • Knowledge, especially content knowledge and knowledge of the world • Metacognitive strategies, including motivation and interest