Assessment of Learning Communities

1. Assessment of Learning Communities Carolee Larsen, Ph.D. Director of Assessment, University College Kennesaw State University clarsen1@kennesaw.edu

2. What is Assessment? There are two general types. A comprehensive assessment program includes elements of both. • Formative Assessment: On-going collection of data for program improvement. • Summative assessment: Used to evaluate programs and student competencies in certain areas

3. What Assessment is Not • It is not a tool to evaluate individual courses or faculty members.

4. Benefits of Assessment? • Provides evidence of program strengths and weaknesses • Provides data to demonstrate program effectiveness for • The college • State and federal governing bodies • Parents and students • Accrediting agencies • The press and public

5. Benefits of Assessment • Ability to compare student learning against defined learning outcomes • Defining clear and known objectives • Enable ongoing improvement through data-driven decision making • Combs, Gibson, Hays, Saly, and Wendt (2008, p. 89) Larsen, Thompson, and Tillman | Learning Outcomes for Learning Communities as a Whole | NLCC 2012 | Nov. 8-10, 2012

6. Levels of Assessment • Individual • Class • Program* Assessment in general is moving toward Assurance of Learning (AOL). (This means movement from RPG analysis to direct assessment of student learning). *For Learning Communities we are mainly concerned with Program Level assessment. Often, this can be carried out alongside classroom–level assessment

7. Types of Evidence Used in Assessment A comprehensive LC assessment program incorporates direct and indirect measures.

8. Short Course: How to Assess LCs • Decide what needs to be assessed. Your program may want to answer the following questions. • How well are the learning outcomes being met in the LCs? • Is GPA improved as a result of being in a LC? • Do LCs help with retention? • How well are the LC courses integrated around the points of intersection (Integrative learning) • Etc.

9. How to Assess LCs, cont’d. • Examine existing standards and learning outcomes for clues about what needs to be assessed. • Rewrite standards and outcomes if needed. Most programs will need to revise or write learning outcomes so they are assessable.

10. How to Assess LCs, cont’d. • Determine what data are available already and what additional data need to be collected in order to answer assessment questions. • Design data collection tools and/or use already made ones (such as the Washington Center LC student survey and the AAC&U value rubrics). • Set an assessment schedule Washington Center AAC&U Value Rubrics

11. An Often Overlooked Key to Successful Assessment: Defining and Refining Learning Outcomes (LO) • LOs are used to assess student learning and related program outcomes. • Are the LOs clearly defined and assessable? • With Learning Outcomes, less is often more. “Learning outcomes? What learning outcomes?”

12. What percentage of learning community (LC) programs sampled in this study appear to have a set of assessable learning outcomes (LOs)? Larsen, Thompson, and Tillman | Learning Outcomes for Learning Communities as a Whole | NLCC 2012 | Nov. 8-10, 2012

13. KEEP CALM AND WRITE ASSESSABLE LEARNING OUTCOMES

14. What is a Learning Outcome? • Is “Improve RPG” a learning outcome? • Is “GPA” a Learning Outcome? • Is “Critical Thinking” a Learning Outcome?

15. Learning Outcomes • ABCD Method of writing LOs that are specific and measurable • Audience – individuals who perform action to be assessed • Example: “The students will …” • Behavior – action performed by the audience; the verb • Examples: Explain, demonstrate, differentiate, propose • Condition – circumstance under which behavior takes place or further define the action • Example: “By the end of the spring semester” • Degree – level of competence • Example: “at least 85% correct” or “with fewer than three errors” • Jordan, DeGraaf, and DeGraaf (2005) Larsen, Thompson, and Tillman | Learning Outcomes for Learning Communities as a Whole | NLCC 2012 | Nov. 8-10, 2012

16. Learning Outcomes, cont’d • Guiding questions • Are students learning what they should be learning? • Which teaching, curricular, and co-curricular approaches are working well, and which approaches need to be modified? • What additional educational experiences should be furnished to students, and how should our existing experiences be reorganized? • Is our process for assessing learning communities working effectively? • Adapted from Penn’s questions for general education assessment (2011, p. 10) who modified Tyler’s guiding questions for curriculum development (1949) Larsen, Thompson, and Tillman | Learning Outcomes for Learning Communities as a Whole | NLCC 2012 | Nov. 8-10, 2012

17. Learning Outcomes, cont’d. • Major challenge to advancing assessment: Engaging faculty • Some faculty think • Assessment is a distraction • Grades are sufficient measure of student success • Measures such has retention and persistence are beyond the job description • However, faculty show interest in evidence from student work that is directly linked to teaching and learning • Kinzie (2010, p. 3) Larsen, Thompson, and Tillman | Learning Outcomes for Learning Communities as a Whole | NLCC 2012 | Nov. 8-10, 2012

18. Making LOs assessable Example: Consider “Critical Thinking” as a Learning Outcome • Is this directly assessable? Why or why not? • How can you make it an assessable outcome?

19. Learning Outcomes: Operationalization Examples Critical Thinking: An example of breaking down an LO to making it specific and assessable. Critical thinking is a habit of mind characterized by the comprehensive exploration of issues, ideas, artifacts, and events before accepting or formulating an opinion or conclusion. By completing an assignment that requires producing evidence of critical thinking, students will: • Explain an issue/problem to be considered critically • Select and use information to investigate a point of view or to question a conclusion • Identify assumptions and relevant contexts when presenting a position • Express a conclusion that is logically tied to a range of information, including opposing viewpoints

20. Learning Outcomes Integrative Learning: A key outcome for assessing “points of intersection between and among classes. In a learning community, integrative learning is achieved when a student becomes predisposed to make “simple connections among ideas and experiences [and to synthesize and transfer] learning to new, complex situations.” • Though completion of an integrative learning assignment or project, students will: • Demonstrate connections between relevant experience and academic knowledge • Identify connections across between and among courses in the learning community • Adapt and apply skills, abilities, theories, or methodologies gained in one situation to new situations • Demonstrate a developing sense of self as a learner, building on prior experiences to respond to new and challenging contexts (may be evident in self-assessment, reflective, or creative work)

21. Learning Outcomes Global Learning for Engaged Citizenship Global Learning for Engaged Citizenship, or Intercultural Knowledge and Competence, is "a set of cognitive, affective, and behavioral skills and characteristics that support effective and appropriate interaction in a variety of cultural contexts.” By completing assignment(s) designed to integrate intercultural knowledge and competence into the educational experience, students will: • Identify their own cultural rules and biases • Demonstrate adequate understanding of the complexity of elements important to members of another culture in relation to its history, values, politics, communication styles, economy or beliefs and practices • Recognize intellectual and emotional dimensions of more than one worldview

22. Learning Outcomes Inquiry and Analysis Inquiry and analysis involves the ability to know when there is a need for information, to be able to identify, locate, evaluate, and effectively and responsibly use and share that information for the problem at hand. Through completion of a paper or project that requires students to engage in inquiry and analysis, students will • Identify a topic related to the theme of the learning community that is focused and that appropriately addresses relevant aspects of the topic • Organize evidence to reveal important patterns, differences, or similarities related to the focus • State a conclusion based on inquiry findings • Discuss relevant limitations and implications of the inquiry

23. A Solution for Assessing Student Work (Direct Assessment/AOL) • Sample student work and use rubrics based on carefully written LOs to assess student learning. • Good News: AAC&U VALUE rubrics are there to help. These are free of charge and can be altered to fit program needs. • Download them from the AAC&U website.

24. Assessing Learning Outcomes at the Class Level (a process example) • Use the Rubric for Assessment of Inquiry & Analysis, Global Learning for Engaged Citizenship, and Critical Thinking • This is used for one paper in each LC class. • Assignments can be adapted or designed to correspond with the rubric.

25. Assessment at the LC level: Tips • Use the Integrative Learning Rubric in the anchor course • Develop or adapt an assignment that can be assessed using this rubric.

26. Faculty reporting of assessment rubric data: An example Reporting is simple. When you grade the paper or project that is linked to the learning outcome for the class, use the rubric to determine level of mastery for each category. Collect the data for each paper on an excel spreadsheet. For your convenience these will be made available to you in the fall. Send the spreadsheet data to the assessment coordinator. (This information will in no way be used to assess teaching effectiveness.)

27. Another Possibility: Outside Rubric Raters • Incentivize outside readers to use rubrics to assess collected samples of student work. These “assessors” could be faculty and staff members. • This greatly reduces the perceived assessment burden on faculty.

28. Other forms of assessment for Learning Communities • Washington Center Student Survey. Look for information at the end of the term and encourage students to participate. • Faculty Survey at the end of the term. This is a chance to report on your experiences and innovations with your learning community. This can be designed and administered through Survey Monkey

29. Questions?