Implementing and Scaling an Online Early Intervention System in a Multi-Campus Environment

1. Implementing and Scaling an Online Early Intervention System in a Multi-Campus Environment Norm Downey Perkins Grant Director Mary Kate Quinlan Director of Learning Outcomes & Achieving the Dream Dream 2013 Conference, Anaheim, California Thursday, February 7, 2013

2. Agenda • CCAC and Early Intervention • The Challenges of Scaling • Statistically Driven Incremental Yearly Improvement • Lessons Learned • Rewards of an Early Intervention System • Q & A Early Intervention - Dream Conference 2013

3. Community College of Allegheny County • Serves Pittsburgh, PA metro • Four Campuses/Five Centers • Annual unduplicated headcount • 32,000 credit • 36,000 non‐credit • 2,793 grads 2011‐12 • Among two year colleges: • Largest producer of nursing grads • Second largest producer of allied health professionals Early Intervention - Dream Conference 2013

4. What is Early Intervention? • Early Intervention (EI) is a web based system that allows faculty to refer struggling students for assistance via a secure website • The staff members who receive the referrals attempt to contact referred students in order to link them to college resources • The staff members then provide feedback to faculty and provide on-going support to the referred students Early Intervention - Dream Conference 2013

5. Early Intervention’s Goals • Reach struggling students as early in the term as possible • Connect struggling students to college resources to resolve their academic issues • Help students to maintain or improve their GPA • Improve student retention rates Early Intervention - Dream Conference 2013

6. Which Students are Academically at Risk? • Students who: • Have poor attendance • Are continually late for class or leave early • Don’t take notes • Are inattentive • Don’t participate • Any student who in the instructor’s opinion is unlikely to be successful in a given course Early Intervention - Dream Conference 2013

7. Early Intervention is NOT… • A substitute for good classroom management • A place to refer students with behavioral problems or students who are potentially dangerous - those students should be referred to the Behavioral Intervention Team (BIT) Early Intervention - Dream Conference 2013

8. What’s Early? • Early Intervention is best done as soon as a potential problem is recognized • Earlier in the term there are more and better options to address student problems • Early Intervention concludes about ⅓ the way into the term Early Intervention - Dream Conference 2013

9. The Challenges of Scaling Scenario EI is a successful practice that is benefitting a select population You think it could benefit all students Question What are some of the challenges to taking EI to scale?

10. Challenges of Scaling - Background • EI had originally been a Perkins Grant initiative • Designed to serve only Career & Technical Education students or roughly 50% of CCAC students • Scaling would expand EI to developmental students • Scaling meant “new” staff would suddenly be handling EI, roughly tripling the staff involved • Timing of scaling • 4 months from scaling decision to implementation Early Intervention - Dream Conference 2013

11. Challenges of Scaling - Background • One year after scaling to all students, three broad problem areas with EI were identified • Staff buy-in • Faculty buy-in • IT Development of data collection and reporting systems • Priority was given to • Staff buy-in by developing systems to streamline Early Intervention workloads • IT development to gather the necessary statistical evidence to determine EI efficacy • Faculty buy-in and increasing referrals would come later Early Intervention - Dream Conference 2013

12. CCAC’s Hierarchy Of Student Services Early Intervention - Dream Conference 2013

13. Challenges of Scaling – IT Issues • Needed systems to: • Sort and assign referrals to correct student services group • Allow student services to provide feedback to faculty • Track referrals, staff responses and student outcomes • First year lacked automated referral assignment, unified feedback systems and tracking systems • Difficult to assess EI’s effectiveness because of a lack of statistical data Early Intervention - Dream Conference 2013

14. Challenges of Scaling – Staff Buy-in • Confusion over the purpose of Early Intervention • Scaling of Perkins Grant initiative not well received by all student services staff members • Concern over workloads and unequal work distribution • Concerns over perceived inappropriate referrals • Faculty feedback issues • No easy mechanism to provide feedback • Lack of understanding why feedback is important Early Intervention - Dream Conference 2013

15. Challenges of Scaling – Faculty Perceptions • Faculty confusion over Early Intervention’s intended purpose… • What is a good referral vs. a bad referral? • What’s meant by Early? • Lack of timely and robust staff-to-faculty feedback created quality perception issues • Confusion about EI process and student services structure Early Intervention - Dream Conference 2013

16. Incremental Improvement Scenario You expanded EI and identified several BIG challenges Question How do you identify where to start?

17. Where to start? Early Intervention - Dream Conference 2013

18. Where to start? • Staff buy-in, Faculty perception and the IT issues were strongly related around the area of streamlining workflows, capturing EI data and providing timely robust feedback to faculty • Developing IT systems to better handle EI while providing additional staff training were identified as starting points • We agreed to not attempt to grow the number of referrals until the systems these were in place to handle such an increase Early Intervention - Dream Conference 2013

19. Fixing the IT Issues Problem Solution Develop systems within Datatel Colleague to sort and assign student referrals to services groups in the heirarchy Create Datatel screens to capture staff intervention data and report that back to faculty • Sort and assign all referrals to correct student services group • Give student services simple system to provide feedback to faculty • Track referrals, staff responses and student outcomes Early Intervention - Dream Conference 2013

20. Fixing the IT Issues • EI scaled in Fall of 2008 • Unified Datatel systems to automatically sort and assign referral introduced in Fall 2010 • Outcomes: • No student referral “falls through the cracks” – all are assigned to staff • Allows staff to easily send feedback to faculty • Improves ability to gather and report EI statistics Early Intervention - Dream Conference 2013

21. Fixing the IT issues • Same system used to sort referrals is used to: • Capture staff/student interactions • Issue feedback to faculty • Send e-mail to students referred for attendance problems • Outcomes: • Provide systems for staff accountability • Gathers statistical evidence to determine EI’s efficacy • Improved faculty feedback and staff buy-in by reducing workload through automation Early Intervention - Dream Conference 2013

22. Fixing Staff Buy-in Problems Solutions On going training Datatel screens to handle staff/student interactions and issue feedback E-mails to students with poor attendance Workshops for faculty, ability to reassign referrals • Confusion over EI’s purpose • Faculty feedback issues • Concern over workloads and unequal work distribution • Concerns over perceived inappropriate referrals Early Intervention - Dream Conference 2013

23. Fixing Staff Buy-in – Outcomes • Asked staff to develop scripts for talking to students and to create suggested work flows • Reduced workloads through automation • Greater accountability through staff report cards • Outcomes: • Increase in student referrals with documented staff interactions • Increase in EI students with passing grades for referring course Early Intervention - Dream Conference 2013

24. Fixing Staff Buy-in – Report Cards Early Intervention - Dream Conference 2013

25. Fixing Faculty Perceptions Problem Solution Workshops on classroom management Reminders to faculty differentiate EI from BIT Newsletter articles Improved feedback Yearly survey of faculty • Confusion about EI’s purpose… • Quality perception from lack of staff-to-faculty feedback • Confusion about process and student services structure Early Intervention - Dream Conference 2013

26. Sample text from automated feedback e-mail The Instructor Dear Walt, You referred the following student for early intervention: 9999999 Duck, Donald Class Name: BUS-101-XX71 Location: XX Term: 13SP Informed of referral: Y Late, poor attendance: N Does not submit work on time: N Does not take notes: N Poor participation/inattentive: N Low grades: Y Low grades despite effort: Y Comments: Instructor comments here. Early Intervention action taken by Mouse, Mickey. Contact methods: Telephone: Attempted 01/20/10 Email: Completed 02/15/10 Postcard: Office Visit: Attempted 05/01/10 Action Taken/Type of Referral: Personal? N Career/Transfer? Y Academic? Y Tutoring? N Supportive Services? N Counseling? Y Job Placement? Y Comments: Staff Comments here line one Staff Comments here line two Staff Comments here line three PLEASE DO NOT REPLY TO THIS MESSAGE. The Student Student Services Staff member Shown as two columns for display purposes Early Intervention - Dream Conference 2013

27. Fixing Faculty Perceptions – Outcomes • Fewer inappropriate referrals for classroom management issues • Increases in total student referrals • Increase in number of faculty making referrals • Based on faculty survey results • Improved opinion by faculty of EI’s effectiveness • EI referral time increase by one week to accommodate faculty wishes Early Intervention - Dream Conference 2013

28. Lessons Learned What have you learned that you could apply to your own intervention?

29. Lessons Learned • Timing of scaling • Preparing staff and faculty for scaling , thus gaining buy-in • Building minimum IT systems before scaling • Importance of accountability • Incremental improvement based on survey data and student outcomes Early Intervention - Dream Conference 2013

30. Rewards of an Early Intervention System

31. Assessing EI’s Efficacy • CCAC’s homegrown EI Datatel system allows for the capture of • Referral data that links the referral to a specific student, course and instructor • Intervention data that links the referral data to the staff member’s intervention efforts and student’s final grade • Data is combined in an Access database to produce reports and publish statistics. Early Intervention - Dream Conference 2013

32. Early Intervention Statistical Outcomes † A student can be referred from more than one class – this is an unduplicated count * For Academic Challenges Referrals with staff interaction Early Intervention - Dream Conference 2013

33. Early Intervention Student Retention • This table is a comparison of the retention rates for students referred for EI versus the overall college retention rates. • The data indicates that students referred for EI have retention rates that are on average 10% greater than the rest of the student population. Early Intervention - Dream Conference 2013

34. What’s next? • Continue with staff training • Work to improve feedback quality and timeliness • Add new tools to the staff tool box – StudentLingo, Math Café and Learning Commons • Conduct assessments of these new tools • Work on increasing faculty referrals • Outreach to students to better explain EI and it’s goals Early Intervention - Dream Conference 2013

35. Questions, Answers and Discussion

36. Presenter Contact Information Norm Downey Mary Kate Quinlan Director of Learning Outcomes & Achieving the Dream Community College of Allegheny County 800 Allegheny Ave., Pittsburgh, PA 15233 mquinlan@ccac.edu (412) 237-3196 Perkins Grant Director Community College of Allegheny County 808 Ridge Ave., Pittsburgh, PA 15212 ndowney@ccac.edu (412) 237-4670 Early Intervention - Dream Conference 2013