Qualitative & Quantitative Perspectives

1. Research and Planning Group 40th Annual Conference Pacific Grove, CA May 1-3, 2002 Faculty Perceptions on Early Alert Qualitative & Quantitative Perspectives Esau Tovar, M.S. Retention Counselor Santa Monica College tovar_esau@smc.edu (310) 434-4012

2. Need for Study • Increased faculty concerns with current program • High rates of unsuccessful grades awarded to “EA” students • No clear evidence that EA works • Need to systematically assess EA • High probationary rates & low persistence rates at SMC

3. Why Early Alert?

4. Matriculation Components • Admissions • Orientation • Assessment • Counseling & Advisement • Student Follow-Up • Coordination & Training • Research & Evaluation • Prerequisites, Corequisites, Advisories

5. Student Follow-Up • Post enrollment evaluation of non-exempt students • Early intervention • Appropriate referrals • Positive and non-positive feedback

6. “Early Intervention” Model • Early Alert Program • Coordinated by Matriculation Office • Instructors identify “at-risk” students through EA rosters • Typically between 4th & 6th week • All instructor responses merged for each student identified • Notices sent to students informing them of status and possible referrals • No additional follow-up

7. Goals of Matriculation • Two primary goals • Student success • Institutional effectiveness • Student services and instruction partnership • Systemwide accountability • Efficient use of resources • Institutional research • Increased participation of underrepresented students

8. Studying Early Alert • Driven by limited research

9. Research Questions • How effective is the EA program in the identification of “at-risk” students? • To what extent has EA impacted student success? • What do instructional and counseling faculty think of EA? • What components should EA incorporate to improve its effectiveness?

10. Institutional research Academic performance outcomes Student interviews Tutoring use Counseling/Instructional faculty interviews & survey Major findings EA Evaluation Approaches

11. EA Academic Performance Outcomes • EA Grade Records • 7,913 • All Grade Records • 72,912 • EA students succeed to a lesser extent than All students (28% vs. 67%) • Withdrawal rates are significantly higher for EA students 47% vs. 19% • Despite differences in grades awarded, conclusions are hard to draw Source: Geltner, P. (2001). The Characteristics of Early Alert Students: Fall 2000. Santa Monica College Office of Institutional Research.

12. Fall 2000 EA Messages • Already contacted 11.9% • Absent or tardy 24.5% • Turn in assignments 16.3% • Tutoring needed 19.1% • Study skills needed 13.0% • Meet with instructor 10.4% • Meet with counselor 4.8% Source: Geltner, P. (2001). The Characteristics of Early Alert Students: Fall 2000. Santa Monica College Office of Institutional Research.

13. EA Faculty Interviews • Faculty not satisfied with EA • Use and Understanding of EA • No clear link between EA & student success • How is it helping students? • Problems with Existing EA Program • Timeliness is crucial if students are to succeed • Exclusion of short-term & online classes • The current EA program does not provide any feedback to instructors • Once notices are sent out students do not respond • Negative image of EA & increased load factor • Suggestions for Improvement • Use EA to identify at-risk students at any point during semester • Letters should be customized by instructor; hand-delivered in class, or emailed • Feedback to instructors of services provided to students and outcomes • Phone calls to students

14. “The Survey”

15. Online & Printed Form Background (8 items) EA Effectiveness (15 items X 2 scales) Student Academic Information (7 items X 2 scales) Revamped EA Scenario (9 items) Reporting Features (5 items) Open-ended questions (5 items) SMC EA Survey

16. Participants N = 80 Position Held 61% Full-time teaching (n = 48) 32.5% Part-time teaching (n = 26) 4% Full-time counseling (n = 3) 1% Part-time counseling (n = 2) 1% Other (n = 1) FT Faculty Status 43% Tenured (n = 30) 57% Non-tenured (n = 21) Years Employed FT: M = 11.6 PT: M = 10.3 Department Participation Cross-sectional representation (20 departments) Frequency of EA Use 1% Never 1% Rarely 11% Sometimes 13% Often 74% Always Average Time Per Class 96% 1 hour or less Preference for EA Use 14% At any point, Ongoing 5% 1st to 2nd week 23% 3rd to 4th week 46% 5th to 6th week 11% 7th to 8th week 1% 9th week and beyond Summary of Findings

17. Assessing Importance & Satisfaction withEarly Alert Components

18. Referring students easily to college survival workshops (e.g., study skills, time management) Supporting increased student contact with instructional resources Making students aware of academic support resources: tutoring Supporting increased student contact with counselors Providing identified students with timely feedback and suggested interventions to improve performance Providing identified students with accurate feedback on identified problems Supporting increased student contact with instructors Supporting increased student contact with special programs (e.g., Latino Center, EOPS) Providing counselors with appropriate training on use of EA and follow-up 2 Monitoring individual and/or group student academic performance Providing or improving communication among instructors, counselors, and students Tracking students after initial identification Providing instructors with appropriate training on use of EA and follow-up 1 Aiding instructors to identify students at risk of performing poorly academically Assisting instructors by providing a means by which to evaluate student classroom performance 72 4.26 0.96 69 4.26 1.22 76 4.25 1.12 67 4.22 1.18 77 4.22 1.13 75 4.05 1.27 75 3.97 1.33 67 3.9 1.28 54 3.7 1.27 69 3.65 1.39 73 3.64 1.34 69 3.45 1.41 69 3.26 1.45 76 3.16 1.55 75 2.91 1.45 Importance of EA Components n M SD Part-Time vs. Full-Time Faculty Member Mean Importance 1 PT = 3.77 (SD = 1.31), FT = 3.02 (SD = 1.47); t(67) = 2.05, p < .05 2 PT = 4.38 (SD = 0.89), FT = 3.42 (SD = 1.31); t(52) = 2.67, p < .01 Scale: Importance of Early Alert Components 1 = Not at all important, 5 = Very important

19. Making students aware of academic support resources: tutoring 1 Referring students easily to college survival workshops (e.g., study skills, time management) Supporting increased student contact with special programs (e.g., Latino Center, EOPS) 2 Supporting increased student contact with instructional resources Supporting increased student contact with instructors Providing identified students with timely feedback and suggested interventions to improve performance Aiding instructors to identify students at risk of performing poorly academically Providing counselors with appropriate training on use of EA and follow-up Supporting increased student contact with counselors Monitoring individual and/or group student academic performance Providing identified students with accurate feedback on identified problems Assisting instructors by providing a means by which to evaluate student classroom performance Providing or improving communication among instructors, counselors, and students Tracking students after initial identification Providing instructors with appropriate training on use of EA and follow-up 68 3.50 1.19 63 3.27 1.17 50 3.16 1.00 58 3.14 1.15 70 3.11 1.27 71 3.03 1.44 71 2.89 1.36 39 2.82 1.19 51 2.80 1.37 61 2.80 1.26 72 2.76 1.45 67 2.67 1.17 67 2.61 1.28 60 2.60 1.15 64 2.55 1.18 Satisfaction with EA Components n M SD Part-Time vs. Full-Time Faculty Member Mean Satisfaction 1 PT = 4.00 (SD = 1.02), FT = 3.23 (SD = 1.20); t(66) = 2.67, p < .01 2 PT = 3.81 (SD = 0.91), FT = 2.85 (SD = 0.89); t(48) = 3.53, p < .001 Scale: Satisfaction of Early Alert Components 1 = Not at all satisfied, 5 = Very satisfied

20. EA Priorities Matrix* Importance – Satisfaction = Performance Gap Very Important High Status ItemsImprovement Needed(large performance gaps) Greatest Strengths(smallest performance gaps) Not At All Satisfied Very Satisfied Low Status Items Opportunity to Study/Redirect(medium performance gaps) Low Status ItemsOpportunity to Redirect Efforts (medium performance gaps) Not At All Important * Priorities matrix based on Noel-Levitz’s Matrix for Prioritizing Action

21. Supporting increased student contact with counselors Providing identified students with accurate feedback on identified problems Providing identified students with timely feedback and suggested interventions to improve performance Providing or improving communication among instructors, counselors, and students Supporting increased student contact with instructional resources Referring students easily to college survival workshops (e.g., study skills, time management) Supporting increased student contact with instructors Tracking students after initial identification Monitoring individual and/or group student academic performance Providing counselors with appropriate training on use of EA and follow-up Making students aware of academic support resources: tutoring Supporting increased student contact with special programs (e.g., Latino Center, EOPS) Providing instructors with appropriate training on use of EA and follow-up Assisting instructors by providing a means by which to evaluate student classroom performance Aiding instructors to identify students at risk of performing poorly academically 49 1.41 1.47 70 1.29 1.44 70 1.19 1.70 65 1.06 1.56 57 1.04 1.59 62 0.94 1.24 67 0.91 1.62 56 0.89 1.42 60 0.87 1.42 36 0.78 1.24 67 0.72 1.40 47 0.70 1.35 59 0.69 1.32 66 0.29 1.41 70 0.29 1.45 Performance Gaps inEA Components* n M SD Importance – Satisfaction = Performance Gap

22. Additional Sources of Information Beneficial to Assist Students

23. High school academic history (e.g., GPA, courses completed)* t(67) = 2.49, p < .01 English—writing—placement information English—reading—placement information Directory of campus referral sources College academic history (e.g., SMC or other college GPA, courses completed) t(69) = -2.71, p < .01 Mathematics placement informationt(65) = -2.60, p < .01 Training on referral skills 68 4.78 2.64 3.70 1.37 69 4.30 1.09 4.38 1.02 69 4.29 1.11 4.41 0.99 67 4.13 1.17 4.19 1.17 70 3.93 1.30 4.16 1.22 66 3.39 1.48 3.71 1.32 55 3.35 1.35 3.42 1.33 Information Beneficial to Tailor Courses for All Students & to Identify At-Risk Students All At-RiskStudents Students n M SD M SD Scale: Beneficial to assist ALL STUDENTS & Beneficial to IDENTIFY AT-RISK STUDENTS 1 = Definitely not beneficial, 5 = Definitely beneficial

24. Revamping Early Alert Program at SMC

25. Revamped EA: Scenario • Revamped Early Alert Program to address shortcomings expressed by instructors and students. Among the potential solutions being explored are the following: • Moving from a paper-and-pencil student evaluation roster to an online-based student evaluation allowing individual customization (for instructor and student); • Allowing instructors to identify and assess only at-risk students when necessary; • Allowing instructors to provide timely feedback directly to students by means of email or printed letters that can be delivered to the student in class; • Allowing instructors, counselors, and other campus programs to follow up with instructor recommendations and maintain contact with identified students; • Tracking all sources of student contact (e.g., meetings with counselors, workshops attended, tutoring attendance) after initial identification to prepare and easily present reports on such things as the impact of early alert on student success indicators individually, by group, etc.

26. Obtaining handouts at the beginning of the semester addressing interventions that are likely to help at-risk students (before they experience difficulty and are identified for EA purposes) I would be willing to hand out the EA letter directly to the student at a subsequent class meeting Reports pertaining to interventions suggested and those provided Reports pertaining to student success indicators (retention, persistence, grades) The college should design and implement an online-based Early Alert system Early alert evaluations should be completed only for students at risk of performing poorly or experiencing problem in classes I would be willing to have my name appear on the EA letter or email sent to students Reports pertaining to how students are contacted and by whom I would be willing to use an online-based early alert system The college should continue to use a paper-and-pencil-based early alert system 1 Reports pertaining to the success of specific groups (gender, ethnicity, age, etc.) Early alert evaluations should be completed for every student in every class 2 77 4.71 1.32 79 4.00 1.24 72 3.97 1.05 70 3.96 1.08 76 3.88 1.24 78 3.77 1.49 78 3.76 1.50 70 3.64 1.27 78 3.03 1.53 78 2.76 1.36 70 2.64 1.32 77 2.42 1.57 Level of Agreement to Revamped/Integrated EA Components n M SD Part-Time vs. Full-Time Faculty Member Mean Agreement 1 PT = 3.57 (SD = 1.26), FT = 2.30 (SD = 1.20); t(76) = 4.41, p < .01 2 PT = 2.97 (SD = 1.61), FT = 2.08 (SD = 1.46); t(75) = 2.47, p < .01 Scale: Level of Agreement Given Scenario 1 = Strongly Disagree, 5 = Strongly Agree

27. Open-Ended Questions

28. Additional Components/Feedback Desired: For Students • Increase Student/Instructor/Counselor Contact • Formalized Referrals/Resources • Assessment/Prerequisites • Completion & Accuracy • Student Unpreparedness • Lack Study Skills • College Survival Workshops

29. Additional Components/Feedback Desired: For Faculty • Implement Follow-Up Component • Provide feedback on what happens with the identified student, & the services provided • Timeliness of EA Notices • Assessment/Prerequisites • Completion & Accuracy • Student Background Information • Assessment Information • Progress in Other Courses • Increased Instructor/Counselor Collaboration • Facilitate Student Learning • Office Visits • Classroom Management • Faculty Training • Formalized Referrals/Resources

30. Next Steps

31. Discussions • Expanding EA Task Force • Consulting with MIS • Online EA System • Reviewing Products: • RetentionTRAX by Noel-Levitz • Student Correspondence System by Precision Dynamics

32. Timeline • Finalize discussion Late May • Develop/Adopt Product Summer ‘02 • Pilot Test Fall ’02 • Assess Effectiveness Fall ‘02 • Institutionalize Spring ‘03