Development, Purposes, Uses of EGRA

1. Development, Purposes, Uses of EGRA Session 1.1

2. Outline • Purpose: why Early Grade Reading (Assessment)? • More focus on quality • Using Early Grade Reading as a “marker” of quality • Development • Uses

3. Outline • Purpose: why Early Grade Reading (Assessment)? • More focus on quality • Using Early Grade Reading as a “marker” of quality • Development • Uses

4. Purpose: Why? Quality issues • What are the big international goals? • How do low income countries compare to high income countries? • LI to HI ratio • Gross primary enrollment: 95% • Net primary enrollment: 80% • Gender parity NER: 94% • Completion: 58% • Learning achievement: Approx 30%? • Learning achievement: Median LI = 3rd percentile of HI or lower LI= Low income, HI = High income

5. Just as interesting is the internal distribution within the poor countries, and the comparison with the rich countries ...

6. PIRLS 2006 Results

7. PIRLS 2006 Results

8. PIRLS 2006 Results Countries

9. In path of development of good performance, the biggest and fastest drop is in the number of worst performers… In the “standard” or better path, an educational middle class is developed by helping those with worst performance… And the drop in those with very poor performance is much bigger than the increase in those with good performance But this does not happen by accident… 9

10. PIRLS 2006 Results 90 80 South Africa Morocco 70 Kuwait Qatar 60 50 Percent of learners Indonesia 40 Iran 30 20 Slovenia 10 Slovak Rep. France 0 Denmark Italy Lowest Medium Highest Reading competency levels

11. Need to work at the worst end of the distribution, and with the countries that are really lagging… 11

12. Conclusion on quality issues • So, quality, and above all equality of quality (more equality in learning outcomes) seems to be lagging… • EFA goals do refer to quality • But most of the practical emphasis is on access, most of the judgment about “are we on track” is around access • Some moves afoot to track learning outcomes in a consistent way • And countries do participate in international and regional assessments, so we are not completely in the dark… So, why a focus on Early Grade Reading?

13. If quality matters, why focus on Early Grade Reading? Early Grade + Reading = • intervene early, • intervene on reading, • have some way to assess orally Let’s see if we can motivate those conclusions

14. Why early? • Refer to the large and reliable literature on benefits of early (even ECD) intervention • But let’s focus on a few simple things

15. Why early? “For unto every one that hath shall be given, and he shall have abundance: but from him that hath not shall be taken away even that which he hath.” Matthew Effect?

16. Why early? Matthew Effect in reading Data fromthe US Children below a certain level by the end of Grade 1, stay behind forever, and the gap widens And, if they cannot read, they fall behind in everything else Words per minute Initial SES gets amplified Grade in years and months (thus 1. is 6 months into Grade 1) 6 Good, Simmons, Smith (1998)

17. Why early? Reading Trajectories of Low and Middle Readers Words per minute Good, Simmons, Smith (1998)

18. And continues to university & graduate school … This controls for SES and prior academic achievement, so it is the NET effect of a demanding curriculum. Kerchoff and Glennie (1999)

19. Why early? • Not only does the problem, if unresolved, amplify over time, but… • According to various experts, fixing a problem: • in grade 1 takes 30 minutes • in grade 5 takes 2 hours • Teaching good habits early is key, and it becomes very difficult to remediate later

20. Why reading? • No, it is not “the only thing that matters” • But it is a good one to start with • It is a (the?) foundational skill - Hard to imagine anything else going well if childrencan’t read well and soon • It can be used as a marker - Hard to imagine a good school that can’t teach children to read, if children are not reading, the school (district, country) needs serious help

21. Why oral reading? Reason 1 • Oral reading seems to be good predictor • At least in the US (other industrial?) perhaps the best predictor, in early grades, of concurrent and even later success in a broad range of areas

22. Oral reading predictive power Examples: • Wilson (2005): 0.74 correlation between oral reading measures and broader cognitive achievement in Arizona • Fuchs, Fuchs, Hosp, and Jenkins (2001) survey and explain the rather high correlations between oral reading fluency and a large variety of other tests. • Vander Meer, Lentz, and Stevens (2005): 96% of children judged to be at risk using oral reading turned out to be “non proficient” in the Ohio’s more comprehensive test, while of those classified as “low risk” using oral reading fluency, 72% were classified as proficient using a more comprehensive test. • Shaw and Shaw (2002): find similar results for the relationship between simple measures of oral fluency and deeper state-wide measures of reading in Colorado • Fuchs, Fuchs, and Maxwell (1988): correlation of 0.91 between oral reading fluency and other comprehensive tests • Juel (1988) : “The probability of remaining a poor reader at the end of fourth grade, given a child was a poor reader at the end of first grade, was .88 .... the probability of remaining an average reader in fourth grade, given an average reading ability in first grade, was .87.” • Schilling, Carlisle, Scott, and Feng (2007): 80% of at-risk with ORF turned out to be in bottom quartile with Michigan’s own reading test at EOG • Wood (2006) finds significant correlations between oral reading fluency and later curriculum-based tests, and finds that oral reading fluency adds explanatory value even when other factors are considered. • Some of these recommend adding comprehension and vocabulary (EGRA does comprehension , not vocabulary), but ORF by itself does a pretty good job

23. Why oral reading? Reason 2: If use pencil and paper test too early: children bottom out, and test provides too little information on why children can’t read

24. Why oral reading? Reason 2 Case of country X • Pencil and paper test • Mostly attempt to directly measure comprehension • Sufficient: 15% • Basic: 24% • Below grade: 15% • Less than below grade: 46% • We know they do not comprehend, but there seems to be little sense of why: can they dominate the basics? • Authorities admit that childrenbottom out if only try to measure comprehension, so try to measure simpler things:

25. About the easiest level:

26. Why oral reading? Reason 2 • If can’t do it, why? • Can’t decode? • Don’t have the vocabulary? • Some other problem? • Hard to say, it seems…

27. Why oral reading? Reason 3 Elements of oral reading are in accord with curricular framework (at least for some countries) but frequently there are no specific (teacher-level) guidelines on how to assess…

30. Mother Tongue, Grade 1 But this almost never happens

31. Mother Tongue, Grade 2

32. Why oral reading? Reason 3 So, at least in some countries the curriculum does get down to specifics, but… • Not in all (don’t know really how many – has anyone studied this?) • Even then maybe not specific enough (compare to any US state)? • (How specific is optimal? How specific is “good enough”?) • In no country I’ve studied is it operationalized in terms of specific assessment standards or procedures that teachers and officials can actually use

33. Outline • Purpose: why Early Grade Reading (Assessment)? • More focus on quality • Using Early Grade Reading as a “marker” of quality • Development • Uses

34. Development thus far • “Organic” process: meets “market test” at each step • First: informal, small samples, see if it was useful at generating awareness, very little funding • Attention attracted • Some funding to try it a bit more formally • USAID funding: validate efforts thus far with expert opinion, try some more applications • Experts validate, suggest increased formality, seriousness of trials • World Bank adds some funding, try it in two more international languages, local languages

35. Development thus far • Tried it in some more countries • Countries found it useful: “market test” met • Assess statistical properties, test further country reaction, do a few more countries, “market test” further met • And here we are: further evaluate results, discuss, chart further direction and uses

36. Outline • Purpose: why Early Grade Reading (Assessment)? • More focus on quality • Using Early Grade Reading as a “marker” of quality • Development • Uses

37. Possible uses • Range of uses of the approach not predetermined • Seek guidance from all of you • And, different uses would require very different versions of the idea… • That being said, here are some of the uses that are possible… 37

38. Possible uses • But… these uses have been tried or at least discussed (and there is some sequentiality here): • Policy awareness and motivation • Macro • Community-based • Impact tracking and evaluation • Project monitoring • Project impact and evaluation • System monitoring over time • Teacher-based assessment • (Could link to community-based awareness, accountability?)

39. Possible uses • Country comparisons • Within language groups? Not even? • Not a good use, according to many experts’ recommendations: • High-stakes bureaucratic accountability • E.g., bureaucratic reporting for some high-stakes purpose 39