A Kid’s Eye View of Assessment

1. A Kid’s Eye View of Assessment Daniel L. Schwartz Stanford University

2. Kids views of learning may differ from those who dictate what should be learned. • Adults make curricular standards. • Often a good vision of useful learning. • Somewhere along the line, the standards are translated into assessments. • Children see the assessments. • Let’s see what message kids are getting.

3. Question #1 • A California standard for US History, 11th grade. • US11.1.2. • Analyze the ideological origins of the American Revolution; the divinely-bestowed unalienable natural rights philosophy of the Founding Fathers and the debates surrounding the drafting and ratification of the Constitution; the addition of the Bill of Rights. • Predict the question kids see for this standard:

4. Test Item for US11.1.2 • Analyze the ideological origins of the American Revolution; the divinely-bestowed unalienable natural rights philosophy of the Founding Fathers and the debates surrounding the drafting and ratification of the Constitution; the addition of the Bill of Rights.

5. Question #2 • Predict the standard this item satisfies. • Here is what the kid sees:

6. High School Biology • BI6.f • Students know at each link in a food web some energy is stored in newly made structures but much energy is dissipated into the environment as heat. This dissipation may be represented in an energy pyramid. Which of these organisms would most likely be found at the top of an energy pyramid? A. Clams B. Sardines C. Sharks D. Kelp

7. Dual problem. • For adults, the information flowing back from the tests is misaligned with the original intent. • Policy makers may not know this (unless they have taken a test lately). • For kids, assessments provide the raw data for what counts as useful learning. • Adults can tell them lots of things, but the tests are the direct evidence of what we think matters.

9. What do assessments teach kidsabout learning in general? • Sequestered Problem Solving format (SPS). • Measures of efficient retrieval and application. • Important for decoding, fact families, etc. • SPS tests can be misleading… • For kids:. • Smart = Performance without help or resources. • For adults: • Retrospective measure when we really want a prospective measure.

10. A thought experiment on where SPS might lead to misdiagnosis. • Two candidates applying for a job: • Bob took a 5-week course in Excel spreadsheet software. • Mike does not know Excel, but he taught himself 3 other spreadsheet programs. • The company uses Excel • Bob & Mike take an SPS Excel test.

11. Vygotsky(1934/1987, p. 200) • “[A] gardener… would proceed incorrectly if he considered only the ripe fruit in the orchard and did not know how to evaluate the condition of the trees that had not yet produced mature fruit…” • Fueurstein’s dynamic assessment alternative to IQ tests.

12. An alternative to SPS format. • Preparation for Future Learning (PFL) • What can they learn during the assessment? • Include resources so they can learn during assessment. • These dynamic PFL assessments can be very diagnostic of different learning experiences. John Bransford

13. Taylor Martin (UT Austin) Learning Resource in Test (worked example) 67% 33% Correct Solutions Sequestered Problem Solving (SPS)vs.Preparation for Future Learning (PFL) Guided Discovery Tell then Practice Problem past edge of instruction.

14. PFL assessments seem more valid with respect to life.

15. Now, we can turn to technology issues.What can technology do about assessment? • 1. Keep measuring the same way only faster. • Adaptive testing, for example. • “Oh my god, it just gave me an easier problem!” • 2. Change views of assessment (and useful learning). • It is possible to assess process of learning and not just product. • Assessments can also better reflect learning outside school. • Given a relatively open environment how do people choose to learn?

16. Cathy Chase A Choice-Based Assessment • 8th-grade children could play any of five brief games on genetic inheritance. • They could play a game multiple times, or switch around. • They tried to accrue points. • If they lost a game, they lost one of their five lives. • They needed to learn new material for the games. • There were resources to help, including readings and precise feedback.

17. Lives lost: 0 Lives left: 5 Points: 0 Family Tree Fun G: Type in the genotype for each missing alien from this family tree. P: Type in the probability that the offspring would result from the genetic cross of its parents. I’m finished!

18. Lives lost: 0 Lives left: 5 Points: 20 Congratulations! You won the game with a score of 85%! What do you want to do next? Play Again Go to Resources Try a Different Game See Answers for This Game Design an Alien

19. Do Choices Diagnose Anything Interesting about Learning Behaviors? • The critical choice was abandonment after failure. • Children who switched games when they failed. • Abandonment predicted 50% of learning from game. • (Students completed a posttest on genetic inheritance.) • Abandonment also explained 33% of the variation among students in their classroom science grades (in physics). • Abandonment after failure was a better predictor than: • The rate of failure itself. • Standardized test scores. • Pretest scores. • Choice not knowledge was the important variable.

20. Current Efforts. • Many people starting to data mine choice-filled learning environments. • We are making a crowd-sourcing platform for developing choice-based assessments per se. • Dynamic assessments require refinement. • Not just pulling usual formats off the shelf. • “Items” come in the form of learning games. • Hopefully infrastructure will build a critical mass.

21. Conclusion • Will assessments that are game-like and focus on choices for learning change the kid’s eye view of what counts as useful learning? • Probably sooner than it will change adults’ views. • But it is an empirical question. Stay tuned. • Thank you! • For more on this, go to: <AAALab.Stanford.Edu> • Download: Choice-based assessments for the digital age. • Also: Constructivism in an age of non-constructivist assessments.

22. Choice-based assessments. • A clue comes from law school. • The issue is whether, what, how, and when people learn. • Whether, what, how, when… • Choice is the key construct here.