Probability Based Advising for Basic Skills Courses By Ted Younglove and Aaron Voelcker

1. Probability Based Advising for Basic Skills Courses By Ted Younglove and Aaron Voelcker Office of Institutional Research and Planning Antelope Valley College tyounglove@avc.edu

2. Note From Ted: If you would like to try the ‘cheat sheet’ part of this project I can send you an example file for the data, and the data can be run for an hourly fee ($105/hr) by: Scott M Lesch, Ph.D. Principal Consulting Statistician C&C / Statistical Consulting Collaboratory University of California Riverside scott.lesch@ucr.edu

3. Modeling Evaluation Prediction Intervention

4. Modeling and Prediction Update • Predictive models for success and persistence developed in 2006 using • SAS Stepwise Discriminant Analysis • 11 parameter models and • 4 parameter models • Persistence defined as continued enrollment from Fall to Spring • Success defined as Success in all courses taken (A,B,C,P,CR)

5. Modeling and Prediction Update • 4 parameter model selected for ease of use • Success • Age at start of term • Ethnicity Black (Yes/No) • Enrolled in at least one basic skills course (Yes/No) • Units completed beyond 30

6. Modeling and Prediction Update • Validated in Fall 2007 on independent data

7. Modeling and Prediction Update • Validated in 2008 on independent data.

8. Prediction Works! • So what do we do about it? • More of the same? • New efforts?

9. Intervention • Many (90% Fall 2009) students enter AVC with reading, writing and/or math skills that are below college level, • Students are placed into courses by their performance on an entrance exam, • 3 levels of English (ENGL 095, 097, and 099) • 2 levels of Reading (READ 097 and 099) • 3 levels of Math (MATH 050, 060, 070) • Students may be placed into one or more Basic Skills courses.

10. Intervention • Lack of success in Basic Skills courses is a significant impediment to persistence and success at AVC, • Given that a new student tests into a specific Basic Skills course and can successfully pass this course, what other college level courses can this student concurrently enroll in and pass? • Can we help these students by providing guidance to counselors and students based on past students success (or lack of success)?

11. Intervention • Our solution to the problem posed previously is a Logistic Regression model (specifically a Logistic ANOCOVA), • Logistic regressions are used to predict probabilities of occurrence of binary variables, • Frequently used in medical research and marketing, • Predicting the effect smoking has on the probability of a heart attack, • Predicting the probability a customer will purchase a product.

12. Intervention • One possible alternative solution: simply calculate the percent success in concurrent classes for students in the different basic skills courses, • Logistic regression chosen to provide a statistical framework.

13. Intervention • In our case, we are interested in the covariates, the courses taken with the Basic Skills course. • For ENGL095, ENGL097, ENGL099 the model would be: Represents the global mean Represents the specific non ENGL course effect Represents the adjustment effects of placement into ENGL095 and ENGL097 Represents the effect of passing the ENGL course

14. Intervention • Past data was used to estimate the parameters of the logistic equation,

15. Intervention • The model was estimated using SAS proc logistic, • Minimum sample size for a course to be included was 30, • Two important assumptions: • Individual student effect assumed to be random and negligible, • Individual instructor effect assumed to be random and negligible, • Because of the large number of students and instructors these effects can not be easily estimated.

16. Intervention • Once the model has been estimated, the parameter estimates are then be used to calculate the specific probabilities for passing each analyzed secondary course for each Basic Skills English level, • All probabilities are estimated for the case where the student has passed the ENGL course.

17. Intervention Counseling ‘Cheat Sheets’ • ‘Cheat Sheets’ were produced to help counselors and students in selecting courses to improve success in the other courses, • It is hoped that by improving selection of concurrent courses success will improve in ENGL as well, • The project has been implemented in the Intersession 2009 and Spring 2009 registration period.

18. Intervention Counseling ‘Cheat Sheets’ • 122 concurrent courses had sufficient data for estimates for ENGL 095, 097, and 099 • 108 concurrent courses had sufficient data for estimates for MATH 050, 060, and 070 • 20 concurrent courses had sufficient data for estimates for READ 097, and 099

19. Intervention Example: ENGL095

20. Intervention Discussion: What guidance do you give to counselors?

21. Modeling Evaluation Prediction Intervention

22. Evaluation Intersession/Spring 2009 Plan • Test for changes in registration pattern, • Test for increase in percent success • Overall • Basic Skills • Test for differential effect on students predicted not likely to succeed.

23. Evaluation Intersession/Spring 2009 -Complications • New variable created for tracking which students were advised using new method was not used consistently, • Consistency in identification of students provided counseling lacking, • Counseling = 1, student received counseling during this term, probably advised using ‘cheat sheets’ • Counseling = 0, student probably did not receive counseling during this term, probably not advised using ‘cheat sheet’.

24. Evaluation (Registration Behavior) Spring 2009 - ENGL 095 097 099

25. Evaluation (Registration Behavior) Spring 2009 - MATH 050 060 070

26. Evaluation (Registration Behavior) Spring 2009 – ENGL 099

27. Evaluation Intersession/Spring 2009 -Complications • Additional evaluation suggestions? • Two focus areas: • Change in registration behavior, • Change in Success percentage.

28. Evaluation Spring 2009 • After the end of the term: • Are persistence rates higher in the students who were advised using the ‘cheat sheets’? • Are success rates higher in the students who were advised using the ‘cheat sheets’?

29. Conclusions Spring 2009 • LANOCOVA analysis provides a workable way to estimate pass probabilities for concurrent courses, • Adoption by counselors is under way and leading to changes in registration behavior, • Effects on success may be difficult to estimate on Spring data.

30. Discussion Spring 2009 • Suggestions on improving use of the ‘cheat sheets’?, • Suggestions on analysis of success? • Other courses?