Multiple Methods for Assessing Learning Community Outcomes

Multiple Methods for Assessing Learning Community Outcomes Maureen Pettitt, Ph.D., Skagit Valley College Shanda Diehl, Spokane Falls Community College AIR San Diego May 2005

Session Overview • Learning Communities at SVC & SFCC • Rationale • Organization • LC Assessment at SVC • LC Assessment at SFCC • Conclusions

Collaborative Courses at SVC • Collaborative courses are a means of delivering instruction and fostering student learning; they do not have associated credit requirements • Learning Communities (and English links) are required for the transfer degree • Collaborative courses are options for the technical arts degree. • 60+ Learning Communities are offered each year at the college.

Rationale for Interdisciplinary Learning at SVC • Advantages (greater retention, student involvement, etc.) were known, but not part of the initial rationale for requirements. • A response to curricular issues: Faculty felt that students • did not see connections between and among disciplines, and • needed to engage subjects more fully, to see education as a dynamic and interconnected process of exploration and discovery

Multiple Assessment Methods 3. CCSSE 1. Student Satisfaction Survey 2. Student Writing

1. Locally-Developed Student Satisfaction Survey • Faculty-developed two-item survey • All responses on a scale from “strongly disagree” to “strongly agree” • Current N for LCs= 1364 • Faculty are provided their course results and comments, plus the cumulative for the quarter for all courses

“This learning community has been a valuable learning experience”

“It was more valuable to take these classes together than separately”

Student Comments • LC: “This class has been fun and not sucky at all. I think I have learned a lot.” • DE LC: “Doing both [topics] in depth was overwhelming at times.” • DE LC: “Art History rocks!”

Student Comments • “If they had been separate, I would have known the what and where, but not the why, and the why is always the most important question.”

2. Student Writing • Review of Learning Outcomes Assessment Plans (LOAPS) indicated that the overarching Gen Ed outcomes were not addressed in any obvious way • Faculty/IR team developed two-year research project to assess student attainment of overarching General Education learning outcomes

Study Goals Assess students' ability to: • Apply a variety of concepts/texts/contexts and perspectives to solving problems and thinking about issues. • Connect one’s own life experience, ideas, and abilities with those that others bring. • Understand and value the learning process for oneself and for others. • Write, speak, read, and listen effectively. • Demonstrate critical thinking skills.

Study Goals  Assess learning outcomes by modality • determine "if" and "how" this learning occurs in the context of "where" (i.e., interdisciplinary courses, stand-alone courses, distance education courses)

Approach • Identified courses being taught in Learning Communities, stand-alone, and DE that could be “matched” over a two-year period • Faculty agreed to participate and give students course credit • Reviewed work done at SVC with Bill Moore (MID), Alverno College interviews, etc. • Developed a set of questions for students beginning-, mid-, and end-course

Faculty Participation • Winter 2002: Developed 20-item list -- “Learning That We Value” – collapsed to 5 items • Spring 2002: Analyzed student writing from Winter quarter to “pilot” the framework • Summer 2002 & 2003: Stipends to analyze student writing; discuss and report results

Beginning of Course Questions • What are your learning expectations for this course? • How will you know that your learning expectations are being met? • What value, if any, do you expect this learning will have for you?

Mid-Course Questions • Are your learning expectations being met? • Have you learned things that you hadn't anticipated? If so, please describe. • Do you find your learning experience in this class is any different from high school or other community colleges that you know about?

End of Course Questions • Have your learning expectations been met? • Have you learned things that you hadn't anticipated? If so, please describe. • What do you think are the most important aspects of your experiences in this course that account for your learning? • What have you learned in this course that will matter to you five years from now?

Findings • Student writing elicited adequate depth and breadth of comments that demonstrated • application of a variety of concepts, texts, contexts to solving problems/thinking about issues • understanding/valuing learning process • Fewer, but adequate, comments demonstrating self-other connection and critical thinking • Least useful for demonstrating writing, speaking, reading and listening skills

Students in Learning Communities • “After taking this course I feel that I can make connections to various things, such as history, influences, people, and culture. This course taught me the value of making connections and things from my own perspective.” • “I think I will be more likely to make connections between subjects, both similarities and differences.”

Students in Learning Communities • “I do believe that I may think in a way of applying what I am learning to something else.” • “The link between history and music and the connection to the present day versus the past was important writing. The weekly seminar papers really forced me to look at the world differently.”

Students in Learning Communities • “By combining course topics you get the ‘bigger picture’ and are able to sort of apply what we are learning better. By applying a subject or topic to another subject or topic you have to comprehend what you are learning and apply it to other things.”

Comparisons • Comparing student responses in the stand-alone courses with students in collaborative courses—same courses with the same instructor(s)--students in stand-alone courses: • were less likely to write about these connections or about learning, and • tended to focus on personal growth, liking faculty, etc.

Lots more to do…. • Qualitative assessments of this kind are very ambitious projects--review, coding and analysis very time consuming

3. Community College Survey of Student Engagement

CCSSE Factors Analyzed Based on Learning Community Attendance • Active & Collaborative Learning • Student Effort • Student-Faculty Interaction • College Contribution to Knowledge, Skills & Personal Development • Mental Activities

Mental Activities • Similar in structure to Bloom’s Taxonomy • Prompt: “During the current school year, to what extent has your coursework emphasized the following mental activities…” • Response Options: Range from 1 “Very Little” to 4 “Very Much”

Groupings • SVC 1 = SVC students who have already taken a learning community or linked course • SVC 2 = SVC students who have not taken, but are planning on taking a learning community or linked course • SVC 3 = SVC students who have not taken and do not plan on taking a learning community or linked course • Consortium = all students in the Northwest consortium of colleges, excluding those from Skagit Valley College and Douglas College • All = all students who participated in the CCSSE Survey, based on 93 colleges

Memorizing ”memorizing facts, ideas, or methods from your courses and readings so you can repeat them in pretty much the same form.”

Analyzing “mental activities: analyzing the basic elements of an idea, experience, or theory.”

Synthesizing “synthesizing and organizing ideas, information, or experiences in new ways.”

Making Judgments “making judgments about the value or soundness of information, arguments, or methods.”

Applying Theories or Concepts “applying theories or concepts to practical problems or in new situations.”

Using Info for a New Skill “using information you have read or heard to perform a new skill.”

Students who had taken Learning Communities were: • Significantly more likely to have: • Discussed grades with an instructor • Worked with classmates outside of class to prepare class assignments • Had discussions, conversations, and contacts that encourage multiculturalism and global awareness. • Used advising and computer labs

SFCC Learning Communities • Learning community is a package of interdisciplinary courses where students and faculty are consistently involved • Learning community provides strong network of relationships, both intellectual and emotional • Purpose: Improve Student Success in Community College System

Benefits and Costs • Benefits • Improve Student Outcomes – congruent with college mission • See interconnectedness of disciplines • Establish relationships with students and faculty • Student-centered • Costs • Higher expense • Efficiency measures

Methodology • Conducted Study in Fall 2003 • Cohorts: 2000-01, 2001-02, and 2002-03 full-time, day-time, academic degree-seeking freshmen taking state-supported classes • Separated cohorts into two groups: Learning Community participation and NO Learning Community participation

Methodology (continued) • Compared between the two groups • Cumulative Grade Point Average (T-Test) • Percent of Credits Earned versus Attempted (T-Test) • Retention – came back the subsequent academic year (Chi-Square) • Analyzed each cohort separately • Supplemented Analysis with Student Satisfaction Surveys of Learning Community Participation

Findings

Results Ambiguous • Participation in learning communities make a difference in 2000-01, make a slight difference in 2001-02 but no difference at all in 2002-03 • Study was replicated for late 90s cohorts and results were consistent with the 2000-01 results • What does this mean?

Thought Process • Fall 2003: Hypothesis- not enough time had elapsed to get full cumulative GPA, credits earned versus attempted and only Fall 2003 was considered as the time to come back for the 2002-03 cohort. • Study put on hold.

Spring 2005 • Replicated the study, reran all cohort years, and added 2003-04 cohort using the same criteria. • Found that 2003-04 showed same results as 2002-03. No difference was detected in student outcomes by learning community participation in the 2003-04 freshman cohort.

Revised Study • Is there a change in the learning community offerings that have led to these findings? • Yes! The change in learning community offerings was offering more developmental education (remedial classes) as a part of the learning community. The theory was that students who form strong intellectual and emotional connections would do better. • Writing • Reading • Math

Changes in Learning Communities

So . . . 5 Groups Now • Group 1: Took developmental education in learning community as a freshman • Group 2: Took developmental education AND had no participation in a learning community as a freshman • Group 3: College Level students (NO developmental education) AND participated in a learning community as a freshman • Group 4: College Level students (NO developmental education) AND did NOT participate in a learning community as a freshman • Group 5: Participated in a Learning Community AND took developmental education NOT in a learning community as a freshman

Methodology • Analysis of individual cohort years • ANOVA tests were used to determine differences between the 5 groups in cumulative GPA and percent credits earned versus credits attempted • Chi-Square Analysis was used to detect differences in retention

Results • Developmental Education seems to be the key factor: • Of all developmental education freshmen included in the study, those who take their developmental education courses outside of the learning community AND participate in a learning community have the best outcomes • Students in learning communities with the best outcomes are those students who participate in learning communities but take developmental education courses outside learning community

What Does This Mean? • SFCC purpose of learning communities is to improve student success in community college system • Yet, we find that this is not the case, due in part to changes in learning communities to incorporate developmental education offerings • These findings will change the content and themes of learning communities at SFCC • Re-examine delivery of developmental education courses

Multiple Methods for Assessing Learning Community Outcomes