1 / 36

What Role Does Transcript Analysis Play in Multiple Measures Assessment?

What Role Does Transcript Analysis Play in Multiple Measures Assessment?. Regional Research Groups October 19, 2012. Clarify the current status of common assessment efforts in the state Explain how one college used an analysis of transcript data to change its placement practices

amina
Download Presentation

What Role Does Transcript Analysis Play in Multiple Measures Assessment?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. What Role Does Transcript Analysis Play in Multiple Measures Assessment? Regional Research Groups October 19, 2012

  2. Clarify the current status of common assessment efforts in the state Explain how one college used an analysis of transcript data to change its placement practices Share preliminary findings from a study replicating this analysis at 20 colleges Session Outcomes What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  3. Statewide efforts on common assessment Building common systems for sharing assessment data What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  4. Statewide Efforts on Common Assessment: the Task Force SSTF Recommendation 2.1 Community colleges will develop and implement a common centralized assessment for English reading and writing, mathematics, and ESL that can provide diagnostic information to inform curriculum development and student placement and that, over time, will be aligned with the K-12 Common Core State Standards (CCSS) and assessments. What’s Completion Got to Do with It? – April 2012

  5. Statewide Efforts on Common Assessment: Some History • AB743 • Centralized Assessment Bill signed by the Governor in 2011 but a permanent funding source has yet to be identified. • 2012 TTAC Recommendations, April 2012 • Develop a universally available assessment system that provides access to assessment results, as well as K-12 and CCC transcript data, to inform local placement. • CCC Assessment Committee, June 2012 • Bring together an ad hoc workgroup to flesh out the details of the data warehouse and bring back the results to this committee. This should include the crosswalk as well. What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  6. Statewide Efforts on Common Assessment: TTAC • The topic of centralized assessment has highlighted the importance of utilizing a variety of data points in the process of student assessment and predictive behaviors. • Since an ongoing funding source has yet to be identified, the Centralized Assessment Advisory Committee approved the plan to pursue the development of a centralized data warehouse • The idea of predictive analytics offers additional possibilities for the area of student placement and support services. What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  7. Statewide Efforts on Common Assessment: TTAC • The Data Warehouse Work Group proposes to link data in phases. Phase 1 will focus on data that is accessible to the project team, Cal-PASS, EAP, CCCApply and MIS. • A key challenge with these data sets will be to link student records. • The project team recognizes the value of transcript information among community colleges and will work to develop a plan for collecting college transcript information. • K-12 transcript information is dependent upon K-12 institutions implementing an electronic transcript system; currently no K-12 institution has a system in place. What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  8. Statewide Efforts on Common Assessment Source: Tim Calhoon What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  9. Q & A What questions do you have about the statewide work on common assessment? Use the chat box on the left side of your screen to type in questions. What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  10. How things got started Looking closer at assessment and placement What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  11. LBCC’s Research Sought local answers to 3 questions: • What predicts how students assess and place into developmental courses? • What predicts how students perform in those courses? • How well are placement and performance aligned? What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  12. Our Research Five longitudinal cohorts tracking more than 7,000 high school graduates who attended LBCC from a large local unified school district Simplified presentation of utility of 11th grade California Standards Test (CST) scores and high school grades for predicting: • How students are assessed and placed into the levels of our developmental skills sequences • How students perform in those classes What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  13. Alignment in English * p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10

  14. Alignment in Math * p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10

  15. The Take-Aways • Standardized tests best predict standardized tests • Classroom performance best predicts classroom performance • More information tells you more about a student than less information. What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  16. Evidence-Based Change:Fall 2012, Promise Pathways 966 students who completed matriculation by May 5 received • Pilots predictive placement model in English & math using multi-method, evidence-based assessment of students’ preparation and readiness • Prescriptive, full-time course load • Emphasis on early completion of foundational skills Taken together, what does this mean for our students? • Elephant in the subway slides What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  17. English: Full Implementation of Placement and Prescriptive EnrollmentAlternative vs. Traditional Placement What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  18. Math: Partial Implementation of Placement (but only partial prescriptive enrollment) What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  19. Placement is Not Enough: Full-Time Students What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  20. What LBCC Gained By Using Multiple Measures • Gives students earlier but appropriate access to transfer level courses • Increases the number of students that attain meaningful educational outcomes & decreases time to do so • Challenges conventional wisdom about students’ readiness for college • Opens new avenues of discussion about research and kick-starts opportunities for experimentation and innovation • Provides concrete steps that any college—like yours—could take to dramatically improve all students’ futures What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012 20

  21. Q & A What questions do you have about Long Beach City College’s research and placement changes? Use the chat box on the left side of your screen to type in questions. What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  22. A brief overview The Student Transcript-Enhanced Placement Project (STEPS) What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  23. What Are We Trying to Do? Examine the value of using transcripts as part of the assessment process Create predictive models using Cal-PASS data to study students who had already taken community college courses The models analyze how well transcript data predicts the first English & math courses students take and how well they do in them Recruit colleges to do local analyses to generate campus-specific insights and trigger CCC/K-12 conversations What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  24. Related Work Brown & Conely. (2007). Comparing state high school assessments to standards for success in entry-level university courses. Brown & Niemi. (2007). Investigating the alignment of high school and community college assessments in California. Belfield & Crosta. (2012). Predicting Success in College: The Importance of Placement Tests and High School Transcripts. Willett, Hayward, & Dahlstrom. (2007). Leveraging the CST’s. Wurtz. (2007). Relations among high school math performance and college placements and achievement. Fuenmayor, Hetts, & Rothstein. (2011). Assessing Assessment: Evaluating Models of Assessment and Placement. English Curriculum Alignment Project (ECAP) between Grossmont College and Grossmont UHSD. What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  25. What Is Happening Now? Missing data weakened the predictive models (this highlights the value of strengthening data sharing among segments) Most colleges are still working on their analyses Bottom line: So far, high school performance is partially predictive of college performance What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  26. In English tests predict tests, grades and courses matter but vary by college

  27. Cox & Snell pseudo R-square ~ 0.35 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  28. In math, the specific classes taken in high school predict placement most strongly

  29. Cox & Snell pseudo R-square ~ 0.50 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  30. In English, grades predict grades

  31. Cox & Snell pseudo R-square ~ 0.20 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  32. In math, success predictors vary by college

  33. Cox & Snell pseudo R-square ~ 0.20 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  34. What Happens Next? Due to challenges with missing data, additional techniques will be applied with participating colleges We are launching a larger study using Cal-PASS data that will help answer additional research questions about using transcripts for assessment What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  35. What Can You Do? Replicate this research at your campus • http://www.rpgroup.org/projects/STEPS or http://bit.ly/RPSTEPS Learn more about research on placement exams • “Where to Begin? The Evolving Role of Placement Exams for Students Starting College” – Jobs for the Future/Achieving the Dream http://bit.ly/AlternativeAssessment Find out more about Long Beach’s efforts • http://www.rpgroup.org/resources/promising-pathways Find out more about the previous Cal-PASS study • http://www.rpgroup.org/content/2009-rp-statewide-research-award-winner-using-california-standards-test-identify-remediation What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

  36. Find Out More Terrence Willett, RP Group twillett@rpgroup.org John Hetts, Long Beach City College jhetts@lbcc.edu

More Related