Other Research Directions?

1. Other Research Directions? October 13, 2006 Ross Moen National Center on Educational Outcomes University of Minnesota www.nceo.info

2. Discussed individual studies Now consider big picture for PARA Noting Martha’s next session on timeline Time for an Overview

3. A Test Validity Framework Reduce undesirable variance Parse confounded variance Increase desirable variance Where we could go

4. Sidebar- Converging Streams

5. Obstacle = High skills obscured by features of the assessment process Imbalanced Components = High skills in some areas obscured by low skills in other areas Off the Radar = High skills in some areas that are not looked at Threshold = Weaker skills that are not strong enough to show up Models of Student/Test Match

6. Most common solution Classical absolutist perspective = “construct irrelevant” variance Relativist perspective = more bias than construct Other terms = “method”, “nuisance” Corresponds to a model of “removing an obstacle” or “level the playing field” Reduce Undesirable

7. High construct relevance and high bias Cannot discard the variance Look at it separately Little emphasis in traditional validity studies Feedback to stay with irrelevant and under-representation Standard psychometric is to clean up the construct Corresponds to an imbalanced component model Parse Confounded

8. Under-representation usually receives less attention than excess undesirable variance AERA et al “Standards” allows leeway for choice of emphasis NCLB permits some standards to be tested by classroom etc Corresponds to both an off radar and below threshold model Below threshold is popular and controversial Increase Desirable

9. Student Characteristics Research Motivation Research Item Features Research Where we currently are going

10. Continue (with refinements)? Reduce? Expand? Where we should go