We’ll Change Your Life.

1. STEM Learning Communities End of the Year Report 2013-2014 Sponsored by Title V Presented by: Oralia De los Reyes, Ph.D. Director of Title V Michele Hansen, Ph.D.External Evaluator from IUPUI We’ll Change Your Life.

2. PRESENTATION OVERVIEW • Program Results by Grant Objectives • 1) Professional Development • 2) Learning Communities • Assessment Instruments and Results • Intervention for Math Readiness • Conclusions and Implications

3. Comprehensive Evaluation Plan Designed to Address The Following: • Has the centralized, comprehensive faculty development program encouraged faculty to integrate instructional course innovations for improved learning outcomes? (Objective 1) • In what ways have the structured, integrated learning communities (LCs) for STEM students and intrusive support interventions made distinguishable impacts on student learning and academic success outcomes? (Objective 2)

4. Summative and Formative Evaluation: Critical Thinking Assessment Test (CAT) Student Academic Achievement Retention Rates Course Embedded Assessment - Actual Student Work Student Focus Groups Tutor Impact Analysis

5. PROFESSIONAL DEVELOPMENT THROUGH CAT

6. CRITICAL THINKING ASSESSMENT TEST (CAT) • The CAT instrument was designed to measure the following skills: • Evaluating Information • Creative Thinking • Learning and Problem Solving • Communication • One hour short answer essay test • Faculty scored • Detailed scoring guide

7. FACULTY DRIVEN ASSESSMENT • The CAT Instrument is scored by the institution's own faculty using the detailed scoring guide. • Training is provided to prepare institutions for this activity. • During the scoring process, faculty are able to see their students' weaknesses and understand areas that need improvement. • At the end of each scoring session, faculty across disciplines share strategies to improve student’s critical thinking skills. • Faculty are encouraged to use the CAT instrument as a model for developing authentic assessments and learning activities in their own discipline that improve students' critical thinking and real-world problem solving skills. These features help close the loop in assessment and quality improvement. (CAT, para. 3). Source: CAT Overview retrieved from https://www.tntech.edu/cat on August 21, 2014

8. CAT PROFESSIONAL DEVELOPMENT Source: Professional Development Database, compiled by Title V, Spring 2014.

9. CRITICAL THINKING ASSESSMENT TEST (CAT) • Test was administered to all Learning Communities students every fall since 2010 and expanded to general education courses and senior level courses in Fall 2013 • The CAT test has a maximum scale of 38 points. • The national norm for lower level division students is 13.66 • Average score for UTB lower division courses is 13.33 and 14.19 if we include Pre-Health. • There is not enough data to establish a baseline for UTB upper division courses. Summary of Results by Year:

10. LEARNING COMMUNITIES

12. STUDENTS ENROLLED IN STEM LEARNING COMMUNITIES Data Source: Title V STEM Learning Communities Student Tracking Database, compiled by Title V, Spring 2014

13. LEARNING COMMUNITY STUDENTSMAJOR DISTRIBUTION, N=1868 Data Source : Title V STEM Learning Communities Data Team and Office of Data Management and Reporting Database, Spring 2014

14. Learning Communities’ Fall to Fall Freshmen Retention Rate NOTE: UTB Retention is for Fall to Fall FTFT College Ready Freshmen students. Fall 2013 retention is an estimation, official retention data will be available after the 12th day of class.

15. CONTROL GROUP The CONTROL GROUP was drawn from institutional data and was defined by random selection of non LC students... with same classification taking same courses in the same semester

16. FALL 2013 - SPRING 2014 LC STUDENTS’ DEMOGRAPHICSLC N=648, Control N=863 NOTE: Control Group was defined by Student Classification and Non LC students taking same courses during the same semester, control group includes post-bachelor students’ classification (1%)

17. GRADE DISTRIBUTION: SPRING 2014, BIOLOGY 1307, LC N=37; Control N =37; Data Source : Title V STEM Learning Communities Data Team and Office of Data Management and Reporting Database, Spring 2014

18. GRADE DISTRIBUTION: SPRING 2014, CHEMISTRY 2325, LC N=60; Control N=55 Data Source : Title V STEM Learning Communities Data Team and Office of Data Management and Reporting Database, Spring 2014

19. COMPARISON GROUP The COMPARISON GROUP was drawn from Grade Distribution Reports for All Non-dual and Non-LC sections in Fall 2013 and Spring 2014, extracted from Office of Data Management and Reporting Informer Report, Spring 2014.

20. OVERALL GRADE DISTRIBUTION: FALL 2013 - SPRING 2014, LC N=903; Comparison N =4532 Data Source : Title V STEM Learning Communities Data Team and Office of Data Management and Reporting Database, Spring 2014

21. GRADE DISTRIBUTION: FALL 2013, COSC 1336, LC N=14; Comparison N=47; Data Source : Title V STEM Learning Communities Data Team and Office of Data Management and Reporting Database, Spring 2014

22. GRADE DISTRIBUTION: FALL 2013, MATH 2412, LC N=157; Comparison N=150 Data Source : Title V STEM Learning Communities Data Team and Office of Data Management and Reporting Database, Spring 2014

23. L2S AND FIVE YEAR AVERAGE GROUP FIVE YEAR AVERAGE is the average passing rate from Fall 2007 to 2012 and Spring 2007 to 2012- before full scale intervention. L2S is the grade distribution data for the institutionalization of the Title V structured learning assistance model.

24. GRADE DISTRIBUTION: FALL 2013 MATH 1314, LC N=39; L2S N=436 Data Source : Title V STEM Learning Communities Data Team and Office of Data Management and Reporting Database, Spring 2014

25. GRADE DISTRIBUTION: FALL 2013, HISTORY 1301, LC N=46; L2S N=335 Data Source : Title V STEM Learning Communities Data Team and Office of Data Management and Reporting Database, Spring 2014

26. GRADE DISTRIBUTION: FALL 2013, ENGLISH 1301, LC N=69; L2S N=618 Data Source : Title V STEM Learning Communities Data Team and Office of Data Management and Reporting Database, Spring 2014

27. GRADE DISTRIBUTION: SPRING 2014, History 1302 , LC N=46; L2S N=283 Data Source : Title V STEM Learning Communities Data Team and Office of Data Management and Reporting Database, Spring 2014

28. GRADE DISTRIBUTION: SPRING 2014, ENGLISH 1302, LC N=40; L2S N=410 Data Source : Title V STEM Learning Communities Data Team and Office of Data Management and Reporting Database, Spring 2014

29. GRADE DISTRIBUTION: Fall 2013, MATH 2412, LC N=157; Comparison N=150 Data Source : Title V STEM Learning Communities Data Team and Office of Data Management and Reporting Database, Spring 2014

30. Pass Rates of Pre-Calculus MATH 2412 from Fall 2010 to Spring 2014

31. COMP I WRITING ASSESSMENT FALL 2013, N=60 Data Source:Blackboard Outcomes System, English Department, Compiled by Title V STEM Learning Communities, Jan 2013.

32. COMP II WRITING ASSESSMENT SPRING 2014, N=60 Data Source:Blackboard Outcomes System, English Department, Compiled by Title V STEM Learning Communities, June 2014.

33. SURVEY RESULTS

34. Learning Communities Student Survey • Designed to assess students’ satisfaction levels with their experiences in learning community courses. • Administered Fall 2010, Spring 2011, and Fall 2011, Spring 2012, Fall 2012, Spring 2013 (Washington College Version), Fall 2013, Spring 2014. • Responses on Likert-Type Scale where 1=Very Little to 5=Very Much • Current Results Spring 2014 • N= 139

35. LC SURVEY: Spring 2014 Overall Satisfaction Rates Responses on Likert-Type Scale where 1=Very Little, to 5=Very Much, the percentage represents responding “Much” and “Very Much” Data Source: Spring 2014 Title V STEM Learning Communities Student Survey

36. Describe what you liked most of your tutoring experience: “The fact that I enjoyed my tutors and their company because their attitude was great.” “That they explained what we didn't understand in class” “The tutors were very friendly and proficient, well-versed in what they explained, and always ready to help at a moment's notice.” “The tutors were really helpful, and I actually learned thanks to them.” “The tutors were easy to understand and got the content across in a non-confusing manner.” “Hands on help.” “They tried different teaching methods with us to help us better understand the material.” “Our tutors are the greatest, every day with a positive attitude, always keeping us on task, I felt like they did it not only because it was their job, but because they really wanted to help us. I got more knowledge from my tutors than from my teacher. “ “I like the fact that I have a comfortable environment to study and learn new things in. I got to see the way other students study, think, and organize themselves for a certain subject. “ Spring 2014 Student Open-Ended Questionnaire Responses

37. Please describe what you liked least about your tutoring experience: “I enjoyed everything about my tutoring experience.” “There really wasn't anything that I DIDN'T like about the tutorial. Everything was really beneficial. “ “Sometimes we wouldn't cover certain assignments because we didn't have time during class.” “Student to tutor ratio” “Not enough help to go around” “The time sometimes was a bit short like a good 15 minutes more would be better. “ “Some people do not get along and they get too competitive.” “Sometimes competition games get too aggressive and things don't go as planned.” “I dislike that we were in a big group.” “Many of the time, students don't want to participate in the plans the tutors have set up for us. Other times students also blame the tutors for "being bad" but really they just don't want to do anything - which takes away from my experience.” Spring 2014 Student Open-Ended Questionnaire Responses

38. What specific suggestions do you have for improving tutoring? “Excellent. No improvement needed. “ “Nothing. It's perfect the way it is.” “No suggestions.” “I have nothing to change about it.” “Everything is fine the way it is.” “More tutors.” “More tutors, so more of us can get help.” “I would suggest to do the tutoring session right after class.” “Avoid focusing on a single student and try to cover any problems the class in general has with their work.” “Remove tutoring for English.” “To have a variety of activities so everyone can be able to learn.” “More interaction between tutors and professors.” Spring 2014 Student Open-Ended Questionnaire Responses

39. Classroom Community Scale (CCS) (Rovai, 2002) • The 20-item scale measures sense of community in a learning environment. • Research results have suggested that the Classroom Community Scale (CCS) is a valid and reliable measure of classroom community and learning. • Responses on Likert-Type Scale 1 to 5: Strongly Agree to Strongly Disagree • 126 students responded in Spring 2014.

40. Students’ Perceptions of LC Experiences % Agree and Strongly Agree

41. Students’ Perceptions of Tutoring % Agree and Strongly Agree

42. How Did Students Hear About Learning Communities?

43. Peer Mentoring Program Helped Students Improve the Following Skills

44. STUDENT AND TUTOR FOCUS GROUP RESULTS

45. External Evaluator Student Focus Groups • During the Title V funding period, a total of 44 undergraduate students participated in focus group interviews (9 in 2010, 11 in 2011, 12 in 2012, and 12 in 2014). • A group of 5 peer tutors participated in a focus group in January 9, 2014.

46. Student Focus Group Methodology The student focus groups were designed to gain a greater understanding of their learning experiences. Students were recruited to voluntarily participate in hour-long semi-structured interviews regarding their in and out of the classroom experiences. The interviews were tape-recorded and subsequently content analyzed. The qualitative results were analyzed for content and coded into individual theme categories. After completion of each focus group, a detailed report was provided for campus faculty, advisors, staff, and administrators, so that students’ feedback could be used in decision making.

47. Most Valuable Aspects of Tutoring Learning Experiences • “The tutors know how to teach. They have good personalities. They are smart, but it is not just about being smart. They want to teach and know how to teach so that we learn.” • “There are different types of learners and the tutors try to understand our learning styles that we need and adapt.” • “Tutors always make time to meet your needs and are there for us if we need help or do not understand something from class.” • “Tutors made things concrete, provided good examples with details and helped me learn.” • “Feel comfortable with the tutors because they are also students and been in our shoes. They ask what we need help with and teach to that…they ask how they can help us and teach in ways that help us understand material better based on OUR learning needs. They are very understanding.”

48. Tutoring Support (Continued) • “The tutors are more hands-on and responsive to our learning needs…what [we] need help with rather than just lecture.” • “Tutors help us make better connections with what is taught by professors…they make it hands on and help us learn math better. “ • “They are students; they really care and go out of their way to help us. They even come in on weekends! You get to talk to them.” • “Tutors have taken the class before. They know what to expect and tell us what to expect from instructors.” • “Sometimes the tutors teach better than professors. They are more hands on and really teach us what we need to learn better.”

49. LCs Foster Integrative Learning and Critical Thinking “Read articles on the same subjects and made connections between different classes” “We wrote about controversial topics in essays” “We evaluated the information.” “We heard and wrote about different points of view on discussion boards” “Read articles…had us question what is…what is fictional, what is not, inspired us to want to continue and learn more.”

50. Importance of Forming a Sense of Community Through the Learning Community Program “Nice to be part of a small group, there was more group work. We could work in teams and form study groups” “We had classes together and became friends. We could help each other.” “For me I think for both classes we had the same classmates… and I know that’s a lot of fun because you form like a little community amongst each other. So once you get that confidence of like knowing everybody for both classes you feel more free to like ask more questions or ask each other questions that we don’t understand.” “This is my first semester so like I didn’t know anybody and just being in the learning community and meeting new people and having them there with me - helping me… Instead of those classes that I didn’t have with other students that I know. I think it just helped me more.” “Having the same students and actually going to the other classes with those students was extremely helpful. And I’m a pretty open person but in other classes you could see people shut down because they’re afraid…because they don’t know their peers. And so they’re afraid to ask questions.”