Verifying Learning and Advanced Skills and Knowledge

1. Verifying Learning and Advanced Skills and Knowledge Eva L. Baker National Center for Research on Evaluation, Standards, & Student Testing Graduate School of Education & Information Studies University of California, Los Angeles Korean Educational Research Association Educational Innovation for the 21st Century: Sharing Visions and Experiences Hanyang University, Seoul, Korea 29 October 2010

2. Presentation Overview • How have we designed, used, and validated assessments in the past? • Advanced knowledge and skills call for new methods • 21st century skills require deeper, more research-based analyses related to learning and learners • The elastic relationships among practice, research and policy decisions, and research needs

3. Before the Marriage of Learning and Assessment Assessment design Uses and validity Reporting of position in normal distributions Technical quality indices were sensitive to numbers of items and wide ranging response Correlations and predictions Quotas and other limits on access • Focused on stable findings rather than change • Required many items sampled across a broad construct • Use of item analyses, etc. to improve item performance • Detect the best

4. Is it Time to Flower? http://www.google.com/imgres?imgurl=http://www.rubythroat.org/images/RoseOfSharon01.jpg&imgrefurl=http://www.lacitycollege.edu/academic/departments/forlang/Kor/Kcourses.html&h=359&w=250&sz=15&tbnid=ioukMy9nXKdceM:&tbnh=121&tbnw=84&prev=/images%3Fq%3Dsouth%2Bkorean%2Bflower&zoom=1&q=south+korean+flower&hl=en&usg=__VJcjOdfCNSfjONMmSgPpY3LNbMs=&sa=X&ei=SV3ATN2yDY7CsAPku8DxCw&sqi=2&ved=0CCMQ9QEwAw

5. Overview of Assessment of Advanced Skills and Knowledge Needs Goals and strategies Building in validity Different learners, motivations, constraints Emphasis on change Integration/application interpretive, not just “correct” Shared analyses, methods, and technical approaches—convergence and individuality • Making education more accessible • High expectations for all • Emphasis on adaptation and flexibility • Preparation for global environment

6. Beyond “Alignment” Metaphor • Standards-based systems imply lining up instruction provided and tests used to produce learning (ISD) • Too limiting as rarely can standards be fully or completely sampled, almost never do they apply directly to future contexts • Good teaching in 21st century cannot be simply following of approved text and endless repetition to prepare for tests

7. Towards Coherence and Transfer • Paradox: to prepare for flexibility takes greater attention to components of learning rather than general constructs • Biggest change: transfer must be taught and measured • Harder to prepare for accountability exams • Both horizontal (subjects) and vertical (grade level) integration

8. What Are the First Steps? • Clarifying and developing a reasonable number of standards • Making sure the standards are valid—by assessing their quality, commonality of interpretation, and relationships to one another • Logical and empirical processes—building the content ontology

9. 1. Content Clarification by Ontology • Graphic representation of nodes (what is to be learned) and relationships among them • Not strictly a teaching sequence • Verify common interpretations among teachers, educators, content experts, and policymakers • Determine priorities and fundamentals • Create a top-level meta-tag system for managing learning and performance database

10. Common Core Standards Progression

11. 2. Link to Cognitive Demands • Cognitive demands are learnable/teachable processes that are embedded in subject matter • Individuals differences may apply, but these are not traits • They range between very specific to a domain, e.g., trouble shooting, somewhat general, develop a plan for persuasion; to more independent of specific domains, general writing skill

12. 2. Cognitive Demands—21st Century Skills • Adapt problem solving approaches • Transfer and apply skills to new situations • Seek and manage new learning experiences • Take risks in intellectual or entrepreneurial tasks • Communicate in a variety of ways • Expand skills for global work—teamwork, collaboration, language proficiency, cultural situation awareness

13. Problem Solving Ontology Mayer, S. (in progress). Problem solving ontology (working document). Los Angeles: University of California, National Center for Researchon Evaluation, Standards, and Student Testing.

14. 3. Integrate Content and Cognition in Model Tasks • Format may be of multiple types driven by outcome rather than convenience • Support teacher and student initiative in creating similar tasks • Assessments used for accountability and formative purposes • Create reusable objects or templates for assessments (in online environments) with immediate automated scoring • Monitor in database

15. English Language Arts

16. 4. Conduct Ongoing Validity Studies • Systems evolve and validity is a continuing process • Functional for purposes, accountability, improvement • Sensitive to instruction not only to student background • Fair to cultural/linguistic differences, disabled, gender • Minimal gaming or negative side effects • Include in database of individuals and items • Timing

17. 5. More Expansive Goals for 21st Century • Agility • Intellectual and emotional resilience • Persistence • Societal and family values: dependability, ethics, caring, and joy

18. Continuing Challenges • Performance assessments are back, but need research on making them more comparable to one another—validity and fairness • New psychometrics required if feedback to teachers and learners grows in importance—improvement guidance not rank order • Costs can only be dealt with by serious use of simulations in design, administration, scoring

19. Continuing Challenges (Cont’d) • Teacher evaluation using students’ results very imperfect—but has numbers so may drive out more interpretive models • Domains of tasks for configurations of standards need to be developed in advance, tried, and sampled. Problem is short time frame, school year, knowledge dependencies • Need theory of situations (Resnick) to improve process

20. Research Dreams • Small sample, few—tasks models of validity • Integrated databases from standards, through learning, through performance, mediated by individual background and recognition of many paths, with room to grow • New theories of design • Real investigations of variables, not only evaluation of entire interventions • Finding ways to re-motivate students for intrinsic learning in this age of distraction, celebrity, and catastrophe

21. Ready to Fly ??? http://en.wikipedia.org/wiki/File:Black-billed_Magpie.png

22. http://www.cse.ucla.edu Eva L. Baker voice fax email 310.206.1530 310.267.0152 baker@cse.ucla.edu

23. Backup

24. Notes from the U.S. • Emphasis on testing expanding with Race to the Top • Competition funded two consortia to measure Common Core Standards—each with overlapping majorities of states • Involve both formative and accountability assessment • Pressure to link teacher evaluation to student work—controversial • Rethinking role of commercial test producers • No way to provide standard-by-standard reporting

25. Speaking Truth to Power • Not easy unless you are friends • Policymakers in U.S. have odd mix of research-based ideas and preferences—now almost an overwhelming wave • Desire conflicting goals without recognizing conflict