1 / 58

Predicting Success in Job Corps Students What Variables Create a Completer?

Chicago Job Corps Longitudinal Study Alexander Adam Eschbach, Ph.D. Center Mental Health Consultant Paul Simon Chicago Job Corps Center Operated by Management and Training Corporation. Predicting Success in Job Corps Students What Variables Create a Completer?.

dulcea
Download Presentation

Predicting Success in Job Corps Students What Variables Create a Completer?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Chicago Job Corps Longitudinal Study Alexander Adam Eschbach, Ph.D. Center Mental Health Consultant Paul Simon Chicago Job Corps Center Operated by Management and Training Corporation Predicting Success in Job Corps StudentsWhat Variables Create a Completer? 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  2. Predicting SuccessPresentation Structure • Past • Inspecting the data first… • Sample size • Gender, race • Entry drug testing • Educational characteristics • Approach to psychological symptom checklist 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  3. Future • Present Predicting SuccessPresentation Structure • Past • How we are using the data… • Past • Present • Future 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  4. Predicting SuccessPresentation Structure How we are using the data… • Past • Examining trends in student characteristics • Identifying factors that appear to predict success 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  5. Present Predicting SuccessPresentation Structure How we are using the data… • Present • Using data collected today to help students today • Interventionally relevant data (e.g., suicide risk assessment for students endorsing suicidal thoughts) 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  6. Future Predicting SuccessPresentation Structure How we are using the data… • Future • Regularly using mathematical formula to identify “at-risk” students • Developing “tailor-made” interventions to increase student retention • Employing “evidence-based” methods to increase success 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  7. Past Predicting SuccessPresentation Structure Warnings/caveats about using/interpreting the data… • Reflects data from Chicago Job Corps only • Every center has its own unique geographical location (urban vs. rural) • Unique racial, ethnic background • Differences in academic and vocational programs • Cannot assume data presented here will apply to another Job Corps center • Necessary to test this prediction model at other centers 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  8. Predicting SuccessAcknowledgements Many thanks to… • MTC Research Institute • Carl Nink, Director of the MTC Research Institute • Rob Olding, Ph.D., Statistician and Dean of University of Phoenix • Management Training Corporation (MTC) • Christina Hunter , MTC Director of Operations • Anita Sharp, MTC Vice President, Central Region • Chicago Job Corps Administration and Colleagues • Bryan Mason, Center Director, Paul Simon Chicago Job Corps • Gemma Ross, Health and Wellness Manager, Chicago Job Corps • National and Regional Mental Health Consultants • Valerie Cherry, Ph.D., Lead National Mental Health Consultant • Helena Mackenzie, Ph.D., Region V Mental Health Consultant 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  9. Past Longitudinal Study • Data from nearly 3,000 CJC students • Separated from March 2004 – December 2009 • Only students who completed SCL-90-R • Only students with complete data 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  10. Past Longitudinal StudySCL-90-R Information • 90-item checklist for physical, behavioral, and psychological complaints • Every new student completes SCL-90-R on second day • People with 6th grade reading level can understand questions • Literature that shows this checklist has been used with various ethnic and racial groups • “Face-valid,” “What you see is what you get” 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  11. Past Longitudinal StudyWhy look at this data? • Problem with “revolving door” noticed in the early 2000s • 350 – 375 positions, requiring 1 – 2 years to “complete” • Averaging 600 new students a year • Huge attrition (loss of students) due to • Drug use • Violence • Resignation • AWOL • Medical problems 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  12. Past Longitudinal StudyWhy look at this data? • Huge cost to Department of Labor secondary to “revolving door” • Considerable stress on CJC staff • Much effort to orient new students • Much disappointment when student does not finish program • Minimizing negative experiences for students due to failure • Ultimately identifying “at-risk” students early to improve retention and chance for success 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  13. Past Longitudinal StudyBeginning Assumptions(A Priori Hypotheses) • Probably a combination of factors would predict success • Math, reading ability • Prior drug use (“positive on entry”) • Psychological issues (depression, etc.) • Behavioral issues (self-discipline, persistence) • Age • Gender 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  14. Past Longitudinal StudySample Information 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  15. Past Longitudinal StudySample Information 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  16. Past Longitudinal StudySample Information 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  17. Past Longitudinal StudySample Information 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  18. Past Longitudinal StudySample Information 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  19. Past Longitudinal StudySample Information Z = 5.78, p < 0.01 (tw0-tailed t-test). Females significantly less likely to enter the center with a “positive” on drug testing. 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  20. Past Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  21. Past Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) TABE Scores by Separation Type 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  22. Past Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) • Weird, consistent trends from SCL-90-R data • Reflected Chicago Job Corps students’ unique response pattern • A pattern noticed before during dissertation research • A subset of male --- but sometimes female --- students who selected zero or just a few SCL-90-R items were often found to be “positive on entry” for drug use. These were “Deniers” (students who selected 0 – 3 SCL-90 items) • “There’s nothin’ wrong with me, but, oh, I use weed a lot.” • These students were often separated for drug use or violence. 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  23. Past Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) • Generally speaking, it appeared that Ordinary separation students (“Completers”) endorsed a moderate number of items as a group. They were called “Reporters” (Students who selected 4 - 37 SCL-90-R items) 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  24. Past Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) • Students who endorsed many SCL-90-R items were often separated for medical or psychiatric reasons. These were called “Exaggerators” (Students who selected 38 – 90 SCL-90-R items.) 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  25. Past Longitudinal StudyUnivariate Look at the Data(“One Thing at a Time”) • Using the SCL-90-R to help students in the “Here and Now.” • SCL-90-R “Top 10” Items Endorsed by Chicago Job Corps students… 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  26. Past Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) • 10. Trouble concentrating. • 9. Awakening in the early morning. • 8. Pains in lower back. • 7. Trouble remembering things. • 6. Having to check and double-check what you do. 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  27. Past Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) • 5. Trouble falling asleep. • 4. Feeling easily annoyed or irritated. • 3. Worrying too much about things. • 2. Headaches. 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  28. Past Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) • 1. Feeling that most people cannot be trusted. 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  29. Past Top 10, Most Frequently Endorsed SCL-90-R Items Chicago Job Corps Students 3/2004 – 12/2009 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  30. Past Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) • Curious about the least selected items? • SCL-90-R “Least Popular” Items Endorsed by Chicago Job Corps students… 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  31. Past Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) • 81. Shouting or throwing things. • 82. Trembling. • 83. Heavy feelings in your arms or legs. • 84. Having thoughts that are not your own. • 85. Feeling afraid to go out of your house alone. 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  32. Past Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) • 86. Thoughts of ending your life. • 87. The idea that someone else can control your thoughts. • 88. Feeling afraid you will faint in public. • 89. A lump in your throat. 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  33. Past Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) • 90. Hearing voices that others do not hear. Only 128/2883 (4%) of students selected this item. 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  34. Past Bottom 10, Least Frequently Endorsed SCL-90-R ItemsChicago Job Corps Students 3/2004 – 12/2009 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  35. Present Longitudinal Study Suicide Info (Center for Disease Control, Atlanta) • In 2009, 13.8% of U.S. high school students reported that they had seriously considered attempting suicide during the 12 months preceding the survey; 6.3% of students reported that they had actually attempted suicide one or more times during the same period. 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  36. Present Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) SCL-90-R Data in the “Here and Now” • Students endorsing Item #15 on the SCL-90-R (Thoughts of ending your life) are seen within 24 hours of arriving on center • Suicide Risk Assessment conducted by CMHC and/or Psychology Extern (Practicum Student) • Base rate of suicidal ideation reported on SCL-90-R lower at CJC than reported base rate by CDC for same age group (13.8% U.S. high school population; 216/2887 [7.0 %] CJC students endorsed Item #15) 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  37. Present Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) SCL-90-R Data in the “Here and Now” • In our sample, Item #15 (Thoughts of ending your life) was ranked 86th out of 90 items. • One of the least frequently endorsed items. • When it was endorsed, more than 2/3 of the students reported they misread the item, made a mistake, or they were reporting a former problem and not current suicidal thinking. 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  38. Past • Future Longitudinal StudyUnivariate Look at the Data (“One Thing at a Time”) Summary of the Univariate Findings • “Positive on Entry” is important • TABE reading and math scores are Important • Comparing “Deniers” to “Exaggerators” to “Reporters” is important • Is gender important? • Other variables to predict success? 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  39. Past • Future Preliminary Conclusions • Mathematical model confirms our original scientific guess (hypothesis) that there are identifiable factors predict success 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  40. Future • Past Next StepsWhere did we go from here? • Further data analyses • Use logistic regression to introduce different variables that could predict “group membership” • “Plug in” TABE scores, Positive on Entry, SCL-90-R data, and other possible variables into a formula to calculate a “number” • The greater that “number” is more than 0 (zero), the greater the odds that the student will be in the Successful Completer group • If the calculated “number” is less than 0 (zero), the lower the number, the more likely that the student will be in the Unsuccessful/Noncompleter group 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  41. Future Next StepsWhere did we go from here? • Various separation types divided into Unsuccessful (Noncompleters), Successful (Completers), and Neutral outcomes. • Completion, Ordinary Transfers = Successful • Resignation, Disciplinary, AWOL = Unsuccessful • Medical separation = Neutral • Use univariate observations to “plug into” multivariate (“Logistic Regression”) analysis • Concept is that outcomes are explained by the complex interaction of several variables 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  42. Future Next StepsWhere did we go from here? Building the Logistic Regression (Mathematical) Model 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  43. Future Next StepsWhere did we go from here? Building the Logistic Regression (Mathematical) Model • Positive on entry • Highest grade achieved • Initial TABE math score • Initial TABE reading score • SCL-90-R category • Denier (0 – 3 items) • Reporter (4 – 37 items) • Exaggerator (38 – 90 items) • Gender 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  44. Future Next StepsWhere did we go from here? Building the Logistic Regression (Mathematical) Model Correctly predicted percent = 65.73% ≈ 66%. False positives = 36.46%. False negatives = 32.59%. Area under the ROC curve, c = 0.7158. Sensitivity = 59.9%. Specificity = 70.7%. (Random samples of larger data yielded same results.) 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  45. Future Next StepsWhere did we go from here? Building the Logistic Regression (Mathematical) Model • The Logistic Regression Model (the equation [formula] produces a number. If we want to predict better than “chance,” we started by using 0.5 as the “cut-off” score. The greater than 0.5 that a student scores, the greater the odds that he or she will be a Successful Completer. If the student scores much less than 0.5, the greater the odds that he or she will be an Unsuccessful Noncompleter. 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  46. The logistic function is useful because it can take as an input any value from negative infinity to positive infinity, whereas the output is confined to values between 0 and 1. The variable z represents the exposure to some set of independent (predictor) variables, while ƒ(z) represents the probability of a particular outcome, given that set of explanatory variables. The variable z is a measure of the total contribution of all the independent variables used in the model and is known as the logit. The variable z is usually defined as:

  47. Probability for Success Plotting Using Logistic Regression Z

  48. Future So a student who was positive on entry (= 1), was male (= 1), had reached the 9th grade, received an Initial TABE Reading score of 490, an Initial TABE Math score of 507, and was in SCL-90-R Category 3 would have the following regression prediction equation beginning with the Intercept (mathematical constant): -7.212 + 1(-1.1830) +1(.21641) + 9(.24777) + (490/100[.30038]) + (507/100[(.44276]) + 3(.29142) = -1.3577448. If -1.3577448 is plotted on the graph, it becomes apparent that the student would definitely have less than a 50-50 chance of completing the Job Corps program if we do not intervene and help him. 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

  49. Plotting Chance for Success Male Student Example ( -1.3577448 ) Less Than 50-50 Chance for Success 50-50 Chance Z = -1.3577448 Z

  50. Future In contrast, a student who was negative on entry (= 0), was female (= 0), had reached the 12th grade, received an Initial TABE Reading score of 650, an Initial TABE Math score of 700, and was in SCL-90-R Category 2 would have the following regression prediction equation beginning with the Intercept (mathematical constant): -7.212 + 0(-1.1830) +0(.21641) + 12(.24777) + (650/100[.30038]) + (700/100[(.44276]) + 2(.29142) = +1.39587. If +1.39587 is plotted on the graph, it becomes immediately apparent that the student has considerably more than a 50-50 chance of being a successful completer from Job Corps. 2011 Dept of Labor Job Corps Health and Wellness Conference Alexander Adam Eschbach, Ph.D., Paul Simon MTC Chicago Job Corps

More Related