1 / 60

From a Culture of Evidence to a Culture of Action & Inquiry APPENDIX: Examples of Outcomes Data That Matter

From a Culture of Evidence to a Culture of Action & Inquiry APPENDIX: Examples of Outcomes Data That Matter. Dr. Rob Johnstone Aspen / ATD Leadership Symposium Wye River, MD April 1, 2014. Acknowledgements.

aida
Download Presentation

From a Culture of Evidence to a Culture of Action & Inquiry APPENDIX: Examples of Outcomes Data That Matter

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. From a Culture of Evidence to a Culture of Action & InquiryAPPENDIX: Examples of Outcomes Data That Matter Dr. Rob Johnstone Aspen / ATD Leadership Symposium Wye River, MD April 1, 2014

  2. Acknowledgements • The content in this presentation has been collected from a variety of sources, all of which are publicly available on websites or have been presented in public settings. • These sources include local college IR efforts, state system efforts, and national projects such as: • Completion by Design • The Aspen Prize for Community College Excellence • The Carnegie Foundation’s Statway / Quantway • Collegemeasures.org • Content has also been developed by and with a host of national partners, including CCRC, the Aspen Institute, the RP Group (CA), and AIR (American Institutes for Research)

  3. April 2014 Examples of Data that Matter

  4. Aspen’s Take on Data that Matters: Progress / Completion Outcomes Data Labor Market Outcomes Data Learning Outcomes Data Equitable Outcomes Data

  5. April 2014 Progress / Completion Data: A “Quick” Look

  6. Completion Example #1 CUNY ASAP Program Graduation Rates

  7. ASAP Program Results http://www.cuny.edu/academics/programs/notable/asap/about.html

  8. Completion Example #2 Completion Rates in Completion by Design

  9. Example from CBD: Outcomes for LAS Concentrators

  10. Outcomes by CTE Concentrators

  11. Completion Example #3 Valencia College

  12. Outcomes of Prize finalist colleges: Exceptional improvements over time in completion

  13. Progress Example 1: Grades in Sequenced Courses

  14. Grades in Sequenced Courses Backdrop • Course sequences analyzed in Mathematics, English, Biology, Chemistry, Physics & Economics • Relationship between grade in 1st course in sequence and success in 2nd course in sequence was examined. • Enrollment data tracked over five-year period • Selected highlight slides follow; full PPT provided on request

  15. Data Too Often Presented Like This…

  16. And Not Often Enough Like This… Pre-Algebra Introductory Algebra Intermediate Algebra Intermediate Algebra Transfer-Level Math 306 Math 308 Math 310 Math 310 Math 1300s The grade received by students in Math 308 Success Rate of those same students in Math 310 100% 90% A A 81% 80% 70% Those placing directly into Math 310 60% 60% 57% B B 50% 40% 37% 30% C 20% 10% 0%

  17. Calculus Course Sequence Calculus I Calculus II Calculus II MATH 2413 MATH 2414 MATH 2414 The grade received by students in Math 2413 Success Rate of those same students in Math 2414 100% 90% 90% A A 78% 80% Those placing directly into Math 2414 72% 70% 60% 50% B B 50% 40% 30% C 20% 10% 0%

  18. Developmental Education Writing Course Sequence Two Levels below Transfer One Level below Transfer Transfer Level English One Level below Transfer ENGL 306 ENGL 307 ENGL 1301 The grade received by students in English 307 Success Rate of those same students in English 1301 100% 90% A A 80% 74% Those placing directly into English 1301 71% 70% 61% 60% B B 46% 50% 40% 30% C 20% 10% 0%

  19. College Biology Course Sequence Anatomy Physiology Physiology BIO 2401 BIO 2402 BIO 2402 The grade received by students in Biology 2401 Success Rate of those same students in Biology 2402 100% 88% 90% A A 80% 72% Those placing directly into Biology 2402 70% 67% 60% 50% B B 50% 40% 30% C 20% 10% 0%

  20. College Chemistry Course Sequence General Chemistry I General Chemistry II General Chemistry II CHEM 1411 CHEM 1412 CHEM 1412 The grade received by students in Chemistry 1411 Success Rate of those same students in Chemistry 1412 100% 86% 90% A A 80% 70% Those placing directly into Chemistry 1412 70% 66% 60% B B 46% 50% 40% 30% C 20% 10% 0%

  21. Economics Course Sequence Macro Economics Micro Economics Micro Economics ECON 2301 ECON 2302 ECON 2302 The grade received by students in Economics 2301 Success Rate of those same students in Economics 2302 100% 89% 90% A A 81% 80% 74% Those placing directly into Economics 2302 71% 70% 60% B B 50% 40% 30% C 20% 10% 0%

  22. Progress Example 2: Developmental Education Cohort Tracking

  23. Math cohort tracking starting in Pre-Algebra INTRODUCTORY ALGEBRA INTERMEDIATE ALGEBRA TRANSFER LEVEL PRE-ALGEBRA 100% 100% Attempt the Course 90% 80% 70% 72% Pass the Course 60% Of the original cohort 50% 43% Attempt the Course 40% 30% Of the original cohort 27% Pass the Course 20% 18% Attempt Of the original cohort 10% 11% Pass 5% Att 4% Pass 0% Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.

  24. Math cohort tracking starting in Introductory Algebra INTRODUCTORY ALGEBRA INTERMEDIATE ALGEBRA TRANSFER LEVEL PRE-ALGEBRA 100% 100% Attempt the Course 90% 80% 70% 64% Pass the Course 60% Of the original cohort 50% 50% Attempt the Course 40% 39% Pass the Course 30% Of the original cohort 25% Attempt 20% 19% Pass 10% 0% Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.

  25. Math cohort tracking starting in Intermediate Algebra INTRODUCTORY ALGEBRA INTERMEDIATE ALGEBRA TRANSFER LEVEL PRE-ALGEBRA 100% 100% Attempt the Course 90% 80% 70% 67% Pass the Course 60% Of the original cohort 50% 50% Attempt the Course 40% 37% Pass the Course 30% 20% 10% 0% Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.

  26. English Writing cohort tracking starting at Developmental Writing I DEV WRITING II COMPOSITION I COMPOSITION II DEV WRITING I 100% 100% Attempt the Course 90% 80% 70% 71% Pass the Course 60% 53% Attempt the Course Of the original cohort 50% 45% Pass the Course 40% 33% Attempt 30% Of the original cohort 26% Pass the Course 20% 10% 0% Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.

  27. English Writing cohort tracking starting at Developmental Writing II DEV WRITING II COMPOSITION I COMPOSITION II DEV WRITING I 100% 100% Attempt the Course 90% 80% 78% Pass the Course 70% 65% Attempt the Course 60% Of the original cohort 54% Pass the Course 50% 40% 30% 20% 10% 0% Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.

  28. Progress Example 3: Look for Examples from Four-Year Schools: Retention & Progress at Georgia State http://www.edtrust.org/higher_ed_practice_guide for more examples

  29. First Year to Second Year Retention, Georgia State University

  30. First Year Retention & Progression Rates - Georgia State University

  31. April 2014 Labor Market Outcomes Data: A Quick Look

  32. Labor Market Outcomes Example 1: CollegeMeasures.org ESM for Program-Level LMI Data

  33. CollegeMeasures.org is one possibility to get LMI data • Collaboration between the Lumina Foundation and AIR • Six states currently signed on - AR, CO, FL, TN, TX, VA • Any state can theoretically participate • Check it out: http://www.collegemeasures.org/esm/

  34. Wages of Individual Degree Programs at an Arkansas CC

  35. Wage Earning Rate of Individual <1 Year Certificate Programs at a Colorado CC

  36. Wages for AA Degrees in Business / Mgmt / Marketing in Tennessee, By College

  37. Wage Earning Rate of Licensed Vocational Nurses in Virginia, By College

  38. Labor Market Outcomes Example 2: Aspen Prize’s Approach for Comparing Labor Market Outcomes

  39. Aspen Prize Sample Model Comparing Labor Market Outcomes

  40. Lessons from the Aspen Prize for Community College Excellence Outcomes of Prize finalist colleges: Exceptional short-term labor market outcomes for graduates Average salaries of recent graduates compared to the average for all new-hires in the region (TOP THREE PERFORMERS) 82% above avg. 79% above avg. Regional average 40% above avg. Regional average Regional average 40 COLLEGE EXCELLENCE PROGRAM

  41. Lessons from the Aspen Prize for Community College Excellence Outcomes of Prize finalist colleges: Exceptional long-term labor market outcomes Average salaries of graduates 5 years after graduation compared to the average for all workers in the region (TOP THREE PERFORMERS) 65% above avg. 38% above avg. 55% above avg. Regional average Regional average Regional average 41 COLLEGE EXCELLENCE PROGRAM

  42. April 2014 Learning Outcomes Data: A Quick Look

  43. Goals of Learning Outcomes Assessment • An ideal Learning Outcomes Assessment process: • Focuses on authenticity vs. compliance • Is driven & supported by the practitioners tasked with improvement • Leads to the generation of insight that is woven into improvement efforts • Creates campus conversations about student success and improving outcomes

  44. Learning Outcomes Assessment Thoughts • Lots of outcomes to assess: • Institutional Learning Outcomes • General Education Learning Outcomes • Program Learning Outcomes • Course-level Learning Outcomes • Challenging & complex – but critical to work on generating insight and improvement at all levels

  45. Learning Outcomes Assessment Thoughts • Not many nationally normed LOA models aside from standardized tests (e.g. CLA & CAP) – many local models that are good • Accreditation focus on LOA has resulted in a massive collection / reporting effort – taking away from the actual use of data for improvement…

  46. Learning Outcomes Assessment Thoughts • Check out how CCBC’s Executive Summaries at link tell the assessment & improvement story at the course level • My favorite all-time story is Chem 108 – tells how an attempt to improve outcomes actually made it worse, but then further redesign achieved very high levels – tells that LOA is a journey to improvement story • CCBC also strong on LOA at GE levels through a rubric assessment model – see link for more info

  47. Learning Outcomes Examples • Alverno College • The gold standard example of LOA – check out the student, community, and faculty responses in addition to the mechanics - link • Valencia College • detailed program level assessments, excellent sharing of LOA outcomes with peers – link • Santa Barbara City College • closing the loop by focusing on improvement plans – link • West Kentucky Community & Technical College • overall approach to student learning; significant gains in reading based on LOA assessment – link

  48. April 2014 Equitable Outcomes: A Quick Look

  49. Equitable Outcomes Example 1: Northern Virginia CC & Transfers to George Mason University

  50. Northern Virginia Community College Students Transferring into George Mason University 2005–2012 Source: SCHEV 2014

More Related