1 / 29

Dramatically Better Assessment Systems: Advice for RTTT “Common Assessment” RFP

Dramatically Better Assessment Systems: Advice for RTTT “Common Assessment” RFP. Brian Gong Center for Assessment Presentation for the Input Meetings Sponsored by the U.S. Department of Education for the “Common Assessment” RFP, “Race to the Top” funding November 17, 2009 Atlanta, GA.

adem
Download Presentation

Dramatically Better Assessment Systems: Advice for RTTT “Common Assessment” RFP

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Dramatically Better Assessment Systems: Advice for RTTT “Common Assessment” RFP Brian Gong Center for Assessment Presentation for the Input Meetings Sponsored by the U.S. Department of Education for the “Common Assessment” RFP, “Race to the Top” funding November 17, 2009 Atlanta, GA

  2. My Main Point • The future of assessment in the United States will be shaped by what gets funded in this “Common Assessment” RFP. • USED should shape the RFP and fund it with a longer-term view of having in place dramatically better assessment systems in ten years. • When USED has to compromise, choose longer-term investments over short-term gains • Say very clearly what you want in the RFP • Help foster good responses to the RFP Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  3. Personal recommendations • Hedge bets by funding multiple ways to do multi-state common assessment, especially high school • Invest in six “game changers” that could make assessment dramatically better within a decade, but should not be framed as being operationally implemented on the short time schedule (“2012”) • Help foster good responses to the RFP and after Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  4. Short-term and Longer-term Investments Common Assessment RFP should fund • For implementation by 2012, what we already know how to do in large-scale assessment but • With new set of content standards • With groups of multiple states (difficult to do) • For development through 2015, what we do not know how to do well at scale, but which has potential to lead to dramatically better assessment systems Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  5. Implementing a new multi-state summative assessment takes years 2009 2010 2011 2012 2013 2014 2015 50 state systems, NAEP, TIMMS, PISA, PERLS, many LEA systems, NRTs, ACT/SAT, college’s tests, etc. Fast Implementation of RFP: 2012 (e.g., multi-state assessments with common content standards, “Peer Review” quality of things we know how to do) Test Specifi-cations; Develop Items; Use specs, reports, equating design,administration agreements, etc. (2009-10) Fourth operational administra-tion; first graduating high school class, etc. (2014-15) Pilot Test Items, promulgate high stakes policies, etc. (2010-11) First operational administra-tion & reporting, etc. (2011-12) Second operational administra-tion; first report using growth, etc. (2012-13) Award RFP(s) (9/2009) And aligning curriculum, instruction, accountability, and supports takes longer. Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  6. RFP: Specify, Specify, Specify • USED should specify its purpose, theory of action, and how the assessment results will be used so responders know the big picture • Specify what is wanted as an deliverable and the set parameters for responders’ creative proposals (e.g., time schedule) • Specify the means an outcome should be done if USED really wants a specific means Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  7. Some Model Systems for 2012 • Cross-state comparisons • Standards-based interpretation • Inform better instruction • Rapid turn-around • Measure growth • Measure student performance for teacher/administrator evaluation Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  8. Cross-state Comparisons (2012) • Purpose, TOA, Use: Hold students, schools, LEAs, and states accountable to a common performance standard by triggering sanctions • Outcome: Statistically robust reports of performance on common metric with no “wiggle room” – stronger than current NAEP mapping studies • Means: Same content standards, same test specifications, same performance standards, single assessment across states, same administration procedures, strong equating across years Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  9. Standards-based Interpretation (2012) • Purpose, TOA, Use: Promote equity through holding students and schools to common opportunity-to-learn (content standards) and minimal performance standards • Outcome: Valid reports of performance related to the designated standards • Means: Aligned, grade-level only (?), matrix-sampled (?), high school (?), SWD (?), ELL (?) Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  10. Inform Better Instruction (2012) • Purpose, TOA, Use: Assess more complex and applied learning (monitor); model and encourage instruction (drive) • Outcome: Incrementally better, more valid and reliable measurement of higher-order, complex student performances (?); more widespread “good” instruction (?) • Means: Curriculum-embedded assessments (e.g., standardized units, portfolios, graduation projects) (?); curricula with (local) matched assessments (?) Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  11. Rapid Turn-around (2012) • Purpose, TOA, Use: Promote improvement through rapid feedback to inform actions • Outcome: Reports of performance useful to decisions and actions, in appropriate timeframe (distinguish actions that are multi-year or annual monitoring from annual rich content analysis from shorter-term uses, down to course grades and student instructional feedback) • Means: Trade-off speed for quality, cost: greater reliance on multiple-choice/machine-scored; trade-off centralized standardization for complex performances, local scoring; ignore administration variations (e.g., missing students) Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  12. Measure Growth (2012) • Purpose, TOA, Use: Accountability, program improvement, teacher accountability? • Outcome: Report of student progress over time related to what is/could be/should be: grade-level standards (?), own starting point (?), other students (?), program supports (?), “teacher’s contribution” (?); how to use in accountability (?) • Means: Out-of-level testing (?), adaptive testing (?), vertical [moderated] scales (?), use math to predict reading for greater reliability (?), pre- post-measures within year (?) Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  13. Teacher/administrator evaluation (2012) • Purpose, TOA, Use: Improve teacher quality by providing feedback (?); use in accountability or other high-stakes decisions (?) • Outcome: Changes in student performance associated with (attributable to ?) specific teachers, administrators, programs • Means: many statistical approaches (check assumptions, limitations) (?); combine with other information (?) Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  14. Personal recommendations • Hedge bets on multiple ways to do multi-state common assessment, especially high school • Invest in six “game changers” that could make assessment dramatically better within a decade, but should not be framed as being operationally implemented on the short time schedule (“2012”) • Help foster good responses to the RFP and after Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  15. Invest in “Game Changers” - 1 • Develop technology that provides evidence of more complex knowledge and skills (i.e., more valid) • E.g., interactive simulations, non-academic knowledge and skills • Only use technology with an evidence-centered design approach to maintain construct relevance, most students Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  16. Invest in “Game Changers” - 2 • Develop technology for validity • Develop complex performance assessment • Specify extended learning and content, real application contexts, student choice • Develop credible (local) administration and scoring • Include all students (and teachers) • Develop means of certifying validity and reliability, and of combining with other evidence Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  17. Invest in “Game Changers” - 3 • Develop technology for validity • Develop complex performance assessment • Develop curricula that specify “what” and “how” of learning, and associated local assessment systems • Interim and formative assessments are needed to inform learning directly • Real assessment problem is informing “What should be done next?” – cannot solve without curriculum and teacher/administrator expertise Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  18. Invest in “Game Changers” - 4 • Develop technology for validity • Develop complex performance assessment • Develop curricula, local assessment systems • Develop new measurement models and technical criteria for assessments of complex knowledge and skills • We know current models’ assumptions and limitations; do not impose on innovations! (Example: reliability vs. validity of complex performances; cognitive vs. unidimensional models) Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  19. Invest in “Game Changers” - 5 • Develop technology for validity • Develop complex performance assessment • Develop curricula, comprehensive assessment systems • Develop new measurement models and technical criteria • Develop better accountability models and support better use of assessment results for program improvement • Assessments, assessment use, and instruction are being distorted by our current accountability model Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  20. Invest in “Game Changers” - 6 • Develop technology for validity • Develop complex performance assessment • Develop curricula, local assessment systems • Develop new measurement models and technical criteria • Develop better models of accountability and program improvement • Develop model specifications for a coherent comprehensive assessment system that incorporates above five • e.g., NAEP, state, Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  21. Invest in “Game Changers” - 7 • Technology for validity • Complex performance assessment • Curricula & comprehensive assessment systems • New measurement models and technical criteria • Better accountability models and support for program improvement Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  22. Personal Recommendation - 2 • Invest in five assessment “game changers” • Hedge bets on multiple ways to do multi-state “2012” common assessment, especially high school • Good current models: all MC, mixed MC-CR, computer-based, end-of-course, survey, etc. • Interwoven with state policies (e.g., high school exit requirements) • Help foster good responses to the RFP and after Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  23. Hedge bets on 2012 assessment • End of course AND Grade 11 survey • Computer-based AND paper & pencil • All multiple choice AND modest short CR AND larger amount and more extensive CR • Fund multiple “common content standards” • To find out costs and benefits of multi-state common assessments • Because no one set of content standards is clearly superior • Because no one approach is clearly superior • Because reporting on a common score metric is less important Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  24. RFP Portfolio of Awards • Multiple (around 8) strong models that represent advances that can be implemented strongly by 2012 and that help get to the longer-term goal • Consider strategy: Do not fund strong models that will be adopted even if not funded • Multiple (perhaps 12) strong “game changer” awards Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  25. Personal Recommendation - 3 • Invest in four assessment “game changers” • Hedge bets on multiple ways to do multi-state common assessment, especially high school • Help foster good responses to the RFP and after • If USED wants certain outcomes of states working together, then promote leadership to make that happen among states, NGOs, test vendors, etc. Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  26. Fostering Strong RFP Responses • Provide clear RFP specs and different awards for “2012 implementation” and “game changers” • If USED wants states to have vendor partners in their RFP responses, need to indicate that early and facilitate it well (vs. states’ issuing an RFP) • USED should think about what states who don’t get RTTT common assessment funds will do • USED should think how what it funds will be adopted after RTTT and how that will shape what is available in the future Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  27. Envision Intended & Unintended Consequences • What if in 2012 there were five widely used assessments, all aligned to the same common content standards • Four were commercially available from current test publishers (like the Achieve/Pearson Algebra 2 end-of-course exam) • One was available by joining a consortium (like the WIDA ELP exams) • States were purchasing elementary math from one vendor and high school English from another vendor • What if there were only one assessment being used? What if there were 46? Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  28. Envision Intended & Unintended Consequences – 2 • What if in 2012 each commercially available assessment came in five versions: • An all multiple-choice, computer-administered short form that took 20 minutes and cost $3/per student • An all multiple-choice, computer or paper & pencil form that took 50 minutes and cost $7/per student • A computer or p & p version that took 120 minutes, had 40 multiple choice, 8 short constructed response, and 4 extended constructed response items and cost $15/per student • A computer of p & p version that took 150 minutes, had 40 multiple choice, 4 extended constructed response, and 2 long constructed response items and cost $60/per student • A version that included a standardized test like option 3 and had a curriculum-embedded project and other performance evidence that was centrally audited and cost $200/per student Gong – USED Common Assessment RFP Input Mtg – 11/17/09

  29. For more information: Center for Assessment www.nciea.org Brian Gong bgong@nciea.org Gong – USED Common Assessment RFP Input Mtg – 11/17/09

More Related