David J. Francis Texas Institute for Measurement, Evaluation, & Statistics

1. Developing an Early Diagnostic Assessment for Reading Skills: The Research Basis for the Texas Primary Reading Inventory David J. Francis Texas Institute for Measurement, Evaluation, & Statistics Department of Psychology University of Houston

2. UT-Houston HSC Barbara R. Foorman Jack M. Fletcher Angeliki Mousaki Univ. of Houston Chris Schatschneider Coleen D. Carlson Kimberly J. Wristers Dung-tsa Chen Collaborators in Initial Development of TPRICenter for Academic and Reading Skills

3. Overview • Background on Reading and Reading Problems • Overview of TPRI • Development of TPRI Screening Components • Reliability and Validity of Screening Components • Conclusions and Future Directions

4. What we know about reading, reading acquisition, and reading problems Reading is • the process of extracting meaning from print • a developmental process • dependent on language • a complex behavior that is acquired • playing the piano vs learning one’s native language

5. What we know about reading, reading acquisition, and reading problems • Reading outcomes vary across children. • some children learn almost magically (about 5%) • others learn easily no matter what method of instruction is used (about 20%-30% ) • about 30% of children struggle to learn to read (equal for boys and girls)

6. What we know about reading, reading acquisition, and reading problems • Most reading problems occur at the level of the single word • reading is characterized by slow and labored decoding • comprehension suffers due to inefficiency of decoding • Phonemic awareness is an important skill that facilitates the development of word recognition skills

7. Importance of Early Assessment and Intervention for Reading Problems • Reading problems are associated with poor educational and social outcomes • Children do not simply outgrow reading problems. • Early intervention is clearly effective (Torgesen, 1997; Vellutino et al., 1996) • Not all children respond equally well to all interventions. • Combining early identification with targeted intervention, the effectiveness of early interventions may be enhanced

8. Reading in Texas • State Curriculum Standards (TEKS) • State-wide Accountability System • TAAS (TEKS based CRT beginning in G3) • Academic Performance Indicators • 1997 - Diagnostic Reading Instruments Bill • Link to TEKS • Teacher administered to individual students • Identify reading problems in K-2  Inform instruction • Inform instruction  All children read by Grade 3 • NOT part of State-wide Accountability System

9. Overview of the TPRI • Teacher administered • Kindergarten to Grade 2 • At each grade, test comprised of two major sections: • Screening (identify children NOT at risk) • Inventory (inform instruction)

10. Kindergarten TPRI • Administered middle and end of year • Screening • Letter Names & Sounds • Phonemic Awareness (Blending Onsets & Rimes) • Inventory • Book & Print Awareness • Phonemic Awareness (Rhyming, Blending Word Parts, Blending Phonemes, Detecting Initial Sounds, Detecting Final Sounds) • Graphophonemic Knowledge (Letter Name Identification, Letter to Sound Linking) • Listening Comprehension

11. First Grade TPRI • Screening (beginning and end of year) • Letter Names & Sounds (beginning of year only) • Phonemic Awareness • Blending Word Parts (Beginning of year only) • Blending Phonemes (End of year only) • Word Reading (beginning and end of year) • Inventory • Phonemic Awareness • Graphophonemic Knowledge • Word Reading (Placement on comprehension stories) • Reading Comprehension

12. Second Grade TPRI • Screening (beginning of year only) • Word Reading • Inventory • Graphophonemic Knowledge • Word Reading (Placement on comprehension stories) • Reading Comprehension

13. Empirical Data Behind TPRIDesign of EARS Study

14. Sample Demographics • Total N=945 across the 5 cohorts • 3 schools • All children in all K, 1, and 2 invited. (Random sample of those consenting - 80+%) • Free lunch participation ranged from 13% to 30% • Boys and girls were equally represented • Caucasian (54%), African American (18%), Hispanic (15%), Asian (12%) • SES - LC (9%), WC (43%), MC (48%)

15. Measures Assessed Four Times per Year

16. End of Year Outcomes in Grades 1 and 2 • Woodcock-Johnson Reading (LW, WA, PC) • KTEA Spelling • Gray Oral Reading Test • Formal Reading Inventory • WISC • WJ Broad Reading Score was used to form criterion outcome in Grades 1 (1.4) and in Grade 2 ( 2.4), which correspond to the 18th and 35th percentiles, respectively

17. Longitudinal Sample Sizes in EARS Study

18. Steps Involved in Development of Each Screen 1) Scale Predictors Using IRT where possible and Estimate Ability for all subjects 2) Determine Criterion of Interest 3) Predict Criterion from Latent Ability Estimate 4) Manipulate Cut-Point to Achieve Desired False-Negative Rate 5) Determine Latent Ability at the Cut-Point (4) 6) Select Items to Discriminate at Latent Ability Determined in (5)

20. IRT vs. Classical Test Theory • Classical test theory defines true score as persons expected (i.e., average) score on the test • score is dependent on the items used in making up the test. If the test is made more difficult, a person’s “abilty” goes down • The IRT model is a latent trait, or strong true score model • ability is independent of the items used in the test • because we know the fundamental relation between item responses and ability, we can estimate ability on this constant metric, even though the items in the test may change

21. Advantages of IRT utilized in the TPRI • Focus the selection of screening items at the cut-point on the ability dimension • Keep screening short and still afford accurate discrimination around the cut-point • Develop new screening and inventory items through appropriate linking studies • Develop word lists linked to stories for placement of students on comprehension stories

22. Definitions of Decisions

23. Steps Involved in Development of Each Screen 1) Scale Predictors Using IRT where possible and Estimate Ability for all subjects 2) Determine Criterion of Interest 3) Predict Criterion from Latent Ability Estimate 4) Manipulate Cut-Point to Achieve Desired False-Negative Rate 5) Determine Latent Ability at the Cut-Point (4) 6) Select Items to Discriminate at Latent Ability Determined in (5)

25. Steps Involved in Development of Each Screen 1) Scale Predictors Using IRT where possible and Estimate Ability for all subjects 2) Determine Criterion of Interest 3) Predict Criterion from Latent Ability Estimate 4) Manipulate Cut-Point to Achieve Desired False-Negative Rate 5) Determine Latent Ability at the Cut-Point (4) 6) Select Items to Discriminate at Latent Ability Determined in (5)

27. End of Grade 1 Basic Reading Using December Kindergarten Decision Rules

28. End of Grade 1 Basic Reading Using April Kindergarten Decision Rules

29. End of Grade 1 Reading Using October Grade 1 Word Reading and Phonological Awareness

30. End of Grade 2 using End of Grade 1 Word Reading

31. End of Grade 2 using End of Grade 1 Word Reading and Phonological Awareness

32. Beginning of Grade 2 to End of Grade 2

33. TPRI Field Study • Investigate teacher use • Examine sources of variability in test performance (Content, administrator, time) • Data Collected on 4 forms • End of Kindergarten • Beginning of First Gradea • End of First Grade • Beginning of Second Gradeb aparticipants were at the end of Kindergarten bparticipants were at the end of Grade 1

34. Design of TPRI Field Study

35. Field Study Estimates of Reliability for Screens

36. Conclusions • The TPRI is currently in use in over 80% of Texas Public Elementary schools • Feedback from teachers has been widely positive • Teachers have asked for more training • Current emphasizing development of interventtion strategies and developing more effective means for disseminating this information to teachers • The screening device has proven helpful in optimizing assessment time for teachers

37. Conclusions (cont.) • Opportunities for Research • linking to outcome assessments used in the accountability system • evaluation of instructional decision making and intervention strategies • improved decision making through computer aided administration and student profiling • development of comparable instruments for literacy instruction in Spanish (Tejas Lee)