120 likes | 275 Views
A Causal Rasch Model for Understanding Comprehension in the Context of Reader-Text-Task. AERA/NCME April 13-17, 2012 Vancouver, Canada A. Jackson Stenner Donald S. Burdick Mark H. Stone.
E N D
A Causal Rasch Model for Understanding Comprehension in the Context of Reader-Text-Task • AERA/NCME April 13-17, 2012 • Vancouver, Canada • A. Jackson Stenner • Donald S. Burdick • Mark H. Stone
Reading is a process in which information from the text and the knowledge possessed by the reader act together to produce meaning as measured by a particular task. Anderson, R.C., Hiebert, E.H., Scott, J.A., & Wilkinson, I.A.G. (1985) Becoming a nation ofreaders: The report of the Commission on Reading Urbana, IL: University of Illinois
Data Structure for Testing the Lexile Theory Texts (Native Task) a b c d e f g h i j k l m … N 1 2 3 4 5 6 7 8 9 10 11 12 13 … N Reader-text-task transaction produces an outcome which can be viewed as reading comprehension and/or text comprehensibleness usefully presented as a percent. Readers Reader Ability Measures Empirical Text Complexity Measures New Task Types
A Causal Rasch Model Conceptual Reader Ability - Text Complexity Task Difficulty = Comprehension - Statistical e (RA – TC - TD) i Raw Score = 1 + e (RA – TCi - TD) i RA = Reading Ability TC = Text Complexity TD = Task Difficulty
The Measurement Trade-off Property 0 -300L 300L 200L 200L 1700L 1700L Text Complexity Dial Reader Ability Dial Task Difficulty Dial 72% Comprehension Display
Imagine a world where assessment items are generated and scored in real-time.
Theoretical versus Empirical Text Complexity for 719 Articles* Mean Theoretical = 884.4L (356.2) Mean Empirical = 884.4L (355.0) Reliability = 0.997 SEM = 12.8L r = 0.968 r” = 0.969 R2” = 0.938 RMSE” = 89.6L * Inclusion criteria: 50 encounters and 1,000 items
Artifactual Sources of Variance in Empirical Text complexity Measures • Random measurement error • Sampling error • Range restriction • Systematic error in empirical complexity measures • Wrong function form (not linear) • Variation in empirical text complexities across estimation algorithms We have estimated that the first three of these artifactual sources of variance account for no more than 4% of the total variance in the system – leaving 2% still unexplained. Sources 4-6 may account for this remaining 2%.
Student 1528 7th GradeMaleHispanicPaid Lunch May 2007 – April 2011 347 Encounters138,695 Words3,342 Items983 Minutes Text Demands forCollege and Career 1600 1400 1200 1000 May 2016(12th Grade)
Conclusions • There is only a very small proportion of variation in empirical text complexity left unexplained. • None of the hypotheses about genre (expository vs. narrative), coherence, cohesion, grade dependence, gender dependence, second language dependence have been supported. • It is possible that the small amount of unexplained variance is due to artifacts in the estimation of empirical text complexity.
Contact Info: A. Jackson Stenner Chairman & CEO, MetaMetrics University of North Carolina, Chapel Hill jstenner@Lexile.com