CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation

1. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation 2013 CRLA Conference The Boston Park Plaza Hotel November 9, 2013 Karen S. Agee, Ph.D. Reading and Learning Coordinator Emerita University of Northern Iowa karen.agee@uni.edu

2. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation On a scale of 1 (clueless) to 10 (guru), how knowledgeable are you of student learning outcomes (SLO) assessment strategies? 1 2 3 4 5 6 7 8 9 10

3. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Developing SLO Expertise: Do our programs and services help students to succeed?

4. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Self-assessment of student learning outcomes is now required by accrediting agencies for all programs.

5. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Higher Learning Commission Requirement 4.B. The institution demonstrates a commitment to educational achievement and improvement through ongoing assessment of student learning. (continued. . . )

6. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Higher Learning Commission Requirement, continued: 1. The institution has clearly stated goals for student learning and effective processes for assessment of student learning and achievement of learning goals.

7. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Higher Learning Commission Requirement, continued: 2. The institution assesses achievement of the learning outcomes that it claims for its curricular and co-curricular programs.

8. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Higher Learning Commission Requirement, continued: 3. The institution uses the information gained from assessment to improve student learning.

9. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Higher Learning Commission Requirement, continued: 4. The institution’s processes and methodologies to assess student learning reflect good practice, including the substantial participation of faculty and other instructional staff members.

10. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Postsecondary institutions seek reaccreditation every 10 years but are expected to show long-term data and evidence.

11. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation The Higher Learning Commission wants every institution to develop a “culture of assessment.”

12. Are we concerned about/with student learning outcomes assessment?

13. Stages of Concern Related to Change Stage 0, Awareness: “I am not concerned about SLO assessment.” Stage 1, Informational “I would like to know more.” Stage 2, Personal “How will SLO assessment affect me?” Stage 3, Management “I seem to be spending all my time assessing student learning.”

14. Stages of Concern Related to Change(cont.) • Stage 4, Consequence • “How will my assessment strategies • improve student learning?” • Stage 5, Collaboration • “How can I relate what I am doing • to what others are doing?” • Stage 6, Refocusing • “I have ideas about some even better • assessment strategies!” • Adapted from Hord, Rutherford, Huling-Austin, and Hall, 1987, Taking Charge of Change

15. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Assessing student learning outcomes is a necessary and fascinating aspect of evaluation,

16. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation . . . and the CAS standards for learning assistance programs provide an outline of learning outcomes to consider.

17. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Is it even possible to assess student learning in a learning center ? How can tutoring center staff document gains in student learning ?

18. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Our thesis: Nobody on campus can better assess student learning outcomes (SLO) than we can.

19. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Do we share these premises? 1. Program evaluation is necessary. 2. Student learning assessment is an important part of program evaluation.

20. Program Evaluation • New to higher education in the 1960’s • Difficult for learning centers in the 1970’s • Rare for developmental education programs before 1980’s • After publication of A Nation at Risk, considered essential for the improvement of teaching and learning “[T]he educational foundations of our society are presently being eroded by a rising tide of mediocrity that threatens our very future as a Nation and a people.” “If an unfriendly foreign power had attempted to impose on America the mediocre educational performance that exists today, we might well have viewed it as an act of war.”

21. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Assessments in program evaluation? • Assessment of the need for the program • Assessment of program design • Assessment of how the program is being implemented • Assessment of the program's outcome or impact • Assessment of the program's cost and efficiency

22. Assessment Exercise #1 ~ volunteers needed ~

23. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation What is assessment? Assessment is measurement against a standard.

24. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation “A learning assistance center is any place where learners, learner data, and learning facilitators are interwoven into a sequential, cybernetic, individualized, people-oriented system to service all students (learners) and faculty (learning facilitators) of any institution for whom learning by its students is important” (Christ, 1971, p. 39).

25. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation A learning center must then by definition be a goal-focused program that “steers” by feedback.

26. Program Evaluation Measuresper Hunter Boylan, 1981 • Student grade point averages • Gain scores from pretest to posttest • Ratings of student satisfaction with program services • Student retention

27. Limitations of Typical Evaluation Measures: GPAs • Many variables may affect course grades and grade point averages • Difficult to measure impact of any specific activity • Stepwise regression incorporating all factors of GPA requires sophisticated statistical skills

28. Limitations of Typical Evaluation Measures: Gain Scores • Interactive effect of pretesting • Invalidity of commercial, norm-referenced test for assessing student gain • Potential difference between gain score and actual behavior

29. Limitations of Typical Evaluation Measures: Student Satisfaction • Questions constructed to elicit favorable responses • Bias from survey setting or conditions • Bias from student attitude and unrelated experiences • Not always differentiated by student usage • Satisfaction not synonymous with skill

30. Limitations of Typical Evaluation Measures: Student Retention • Even more generalized than GPAs • Confounded by transient issues causing dropping out temporarily • Lower retention sometimes associated with program success

31. Program Evaluation Meanwhile, back in the 1970’s, student affairs professionals decided to write standards for student services programs.

32. CAS Standards Council for the Advancement of Standards in Higher Education (CAS) www.cas.edu • Founded in 1979 • Promotes standards in student affairs • Creates book of professional standards and guidelines • Develops self-assessment guides

33. CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation Overview of CAS Standards • General Standards (2011), www.cas.edu • 43 functional area standards available for purchase as digital documents or printed books • Learning Assistance Program Standards and Guidelines and LAP Self-Assessment Guide available for $35 at www.cas.edu Standards store

34. CAS Standards Which standards appropriate for us? Learning Assistance Programs (LAP) TRiO and Other Educational Opportunity Programs (TOEOP)

35. CAS Standards Learning Assistance Programs (LAP) Approved by CAS in 1986 with “expert” input from CRLA and NADE and revised in 1996 and 2007

36. CAS Standards Ultimate purpose of CAS? To foster and enhance student learning, development, achievement, and citizenship.

37. CAS Standards: 12 Parts • Mission • Program • Organization and Leadership • Human Resources • Ethics • Law, Policy, and Governance

38. CAS Standards: 12 Sections • Diversity, Equity, and Access • Institutional and External Relations • Financial Resources • Technology • Facilities and Equipment • Assessment and Evaluation

39. CAS Standards: Program CAS Learning and Developmental Outcomes (2008) Six domains, 28 dimensions • Programs and services must assess relevant and desirable student learning and development outcomes and provide evidence of their impact on student learning and development. • Programs must articulate how they contribute to or support students’ learning and development in the domains not specifically assessed.

40. CAS LAP Standards Let’s look now at the LAP standards (pp. 5-10 of handout) and the Learning and Developmental Outcomes (separate handout)

41. Orthodox Methods of Assessing SLOs Direct methods • Students demonstrate learning via some form of standardized test focusing on aspects of student learning • Examples: CAAP, CLA, ETS Proficiency Profile (formerly MAPP), GRE subject tests, PRAXIS exams Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus.

42. Orthodox Methods of Assessing SLOs Direct methods • Students demonstrate learning via some form of standardized test focusing on aspects of student learning • Examples: CAAP, CLA, ETS Proficiency Profile (formerly MAPP), GRE subject tests, PRAXIS exams . . . but couldn’t we assess even more directly?

43. Orthodox Methods of Assessing SLOs Indirect methods • Students report perceptions of their learning and the educational environment that supports that learning • Examples: CSEQ, NSSE, NSLLP Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus.

44. Orthodox Methods of Assessing SLOs Indirect methods • Students report perceptions of their learning and the educational environment that supports that learning • Examples: CSEQ, NSSE, NSLLP Couldn’t students report their perceptions even more powerfully?

45. Orthodox Methods of Assessing SLOs Performance-based methods • Students represent learning in response to assignments or projects that are embedded in their educational experiences Examples: • Successful solution of a problem not previously understood • Demonstrated writing skill in a senior honors paper Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus.

46. Orthodox Methods of Assessing SLOs Performance-based methods • Students represent learning in response to assignments or projects that are embedded in their educational experiences Examples: • Successful solution of a problem not previously understood • Demonstrated writing skill in a senior honors paper Couldn’t we assess students’ performance?

47. CAS Self-Assessment Outline • Review/revise mission • Identify program goals and learning goals • Align major programs and services with goals • Specify desired outcomes • Program and operation outcomes • Learning and development outcomes • Map learning and development goals to CAS learning domains and dimensions • Determine assessment approach

48. CAS Self-Assessment Outline • Review/revise mission • Identify program goals and learning goals • Align major programs and services with goals • Specify desired outcomes • Program and operation outcomes • Learning and development outcomes • Map learning and development goals to CAS learning domains and dimensions • Determine assessment approach

49. Specify Desired Outcomes Program and Operation Learning and Development

50. Write SLO Statements: • Specify who, under what conditions, what, to what extent, how, why • Consider immediate learning and longer-term outcomes