Getting Started in Evaluating Student Learning in Student Affairs/Services

1. Getting Started in Evaluating Student Learning in Student Affairs/Services Rebecca A. Sanderson, PhD Director, Student Affairs Research and Evaluation Oregon State University San Diego State University Evaluating Institutional Learning Centeredness Conference July 12-14, 2007 Sanderson, 2007

2. Assessment • Definition “a rich conversation about students and student learning informed by data.” (adapted from Ted Marchese – AAHE) . . .the systematic collection, review, and use of information about programs/services undertaken for the purpose of improving student learning and development. (adapted from Palomba & Banta, 1999) Sanderson, 2007

3. What is Assessment? • A systematic process of gathering information upon which to make decisions • Using information to determine what is working and what is not • An evaluation of effectiveness • Systematic feedback • An integrated and circular process Sanderson, 2007

4. Assessment • Why? • To improve student learning in essential areas (e.g., diversity/multiculturalism) • To improve programs and services • To ensure that students are learning that which we intend • To focus efforts • To inform decision-making and clarify intentions Sanderson, 2007

5. Assessment This is important—do it. . . ! And then there is this. . . . • Accreditation • Administrative mandates • Accountability • Spelling Commission report • Expectations from professional organizations • Another thing to add to an already busy schedule Sanderson, 2007

6. Benefits • Keeps the focus on students • Provides systematic information to guide program development • Can transform collaborations between Student Affairs and Academic Affairs • Fosters improvement Sanderson, 2007

7. Limitations • Outcomes being measured may not reflect the true impact of a program/service • Probably will not prove that your program alone caused the outcome • Will not answer every question you may have • May take longer and need more resources than you expect Sanderson, 2007

8. Basic Assessment Questions • What are we doing and why are we doing it? • What do we expect a student to know, to think, or to do as a result of our program? • How well are we doing it? • How do we know? • How do we use the information to improve? • Does that work? (Bresciani, 2002) Sanderson, 2007

9. Getting Started • Questions to ask yourself • Why do assessment? • How will it be used? Politics? • How do you ensure broad input and active participation? • How do you engage in reflective conversation about data that informs decision making? • With whom, how, and when do you share? • How do you sustain the effort? Sanderson, 2007

10. Assessment Planning • Successful assessment finds that a written assessment plan is essential • To think through assessment needs and capture agreement about what matters • To give direction for action--To know who, when, where, what • To provide a means to determine if progress is being made Assessment Plan ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ Sanderson, 2007

11. Structural Support Issues • Assessment is best done inclusively • An Assessment Council? • A Coordinator or director position? • A common language • Training and education over time • Feedback on assessment efforts • Tie to improvement rather than to accountability to start • Visible and vocal support from the top Sanderson, 2007

12. Assessment Accountability Structure Vice Provost for Student Affairs OSU Division of Student Affairs Director, Research and Evaluation Student Affairs Assessment Council Student Affairs Departments/Units Programs/Services Sanderson, 2007

13. OSU Assessment Council • Anyone can join • Everyone agrees • to learn, • to help others learn, and • the work goes on even if a person misses a meeting • to share the work • We discuss and come to consensus around important issues • We strive for excellence and also for joy Sanderson, 2007

14. Creating Buy-In • Begin small, but meaningful • Data doesn’t need to be collected annually—create a cycle • Show your work to department • Get input • Include students • Show how data is being used • A successful experience can go a long way Sanderson, 2007

15. Elements of Assessment Planning • Department or Program Mission • Goals • Intended Student Outcomes • Methodology • Implementation • Results • Decisions/Recommendations Sanderson, 2007

16. Assessment Planning • Mission • Describes the purpose of the organization and the constituents served • Should be able to state the mission in less than 25 words Sanderson, 2007

17. Assessment Planning • Goals • Broad general statements of what a program wants its constituents to know or to do. Goals generally describe what the program is trying to accomplish. • It is not a “to do” list. • Is aligned with university goals and the departmental mission • Provides departmental focus Sanderson, 2007

18. Logic Model MISSION Mission:Describes the purpose of the organization and the constituents served. It clearly relates to the Oregon State University and the Division of Student Affairs Missions. Goals:Broad general statements of what a department wants its constituents to know or do. Goals generally describe what the program is trying to accomplish. Typically only 4-6 goals for a department. GOALS Programs:Sets of related activities and outcomes that consume a meaningful portion of the departmental resources (persons, dollars, time, etc.) and that are designed to support the department’s goals PROGRAMS INPUTS ACTIVITIES SERVICE OUTCOMES LEARNING OUTCOMES • Resources dedicated to the program: e.g., Money, Staff, Time, Equipment • Constraints on the program: e.g., Laws, Regulations, Policies Activities done to deliver the program: e.g.,Provide workshops Advise students Distribute brochures Develop handbook Teach classes Provide training Give tests Products from the activities: e.g., Number of workshops Number of people advised Types of brochures produced % served % satisfied Amount of money collected Benefits for participants: e.g.,Gained new knowledge Increased skill Modified behavior Improved their condition Positively altered their status Sanderson, 2007

19. LOGIC MODEL Influences other than your program Longer-term Outcomes Intermediate Outcomes Initial Outcomes Time Outputs What level of outcome do you have the resources to measure? What level of influence to do believe your program will have? Activities Inputs How is this linked to what we know about student learning? Sanderson, 2007

20. Assessment Planning • Learning Outcomes • Detailed and specific statements derived from the goals. They are specifically about the intended end results of your program efforts and typically use active verbs such as: arrange, define, explain, demonstrate, etc. • Levels of learning or mastery—Bloom Sanderson, 2007

21. Writing Learning Outcomes • Target group • Targeted Learning • Level of Mastery (Bloom’s Taxonomy) • Learning Outcome: • (Target group) will be able to (Bloom’s Taxonomy word)(Targeted Learning). Sanderson, 2007

22. Assessment Planning • Methods • The criteria, process, and tools used to collect evidence and to determine the degree to which the intended outcomes were reached. • Assessment methods include: • the target audience, • the methods and tools for data collection, • criteria or targets that tell you when the outcome has been met, and • how the data will be analyzed. Sanderson, 2007

23. Considerations • What method(s) will get the data to answer the question you are asking? • What level of reliability and validity are needed? • Reliability--consistency of measurement • Validity—measures what it purports to measure • Does it make sense and look like it measures what we want it to measure? • What is the Timeliness, Cost, Motivation? Sanderson, 2007

24. Assessment Planning • Methods • Types • Survey • Tests • Performance-based Measures • Checklists • Interviews & Focus Groups • Rubrics • Institutional Data Sanderson, 2007

25. Methods • Survey • Self-reported information: Demographic/descriptive data, attitudes, opinions, values, experiences, behaviors, expectations, goals, needs • Dependent on accurate and honest recall • Can be commercial and/or standardized • Internally developed Sanderson, 2007

26. Methods What no multiple choice!! H-m-m-m?? • Tests • Cognitive or thinking information • Can include written and oral presentations of material • Can be commercial and/or standardized • Internally developed Sanderson, 2007

27. Methods • Performance-based Measures • Direct evidence of learning through performance • e.g., projects, work samples, capstone experiences, direct observation • Must develop criteria for evaluating the performance Sanderson, 2007

28. Methods • Checklists • Direct evidence of • presence, • absence, • frequency of a behavior. • Often used with direct observation, can be used also for content knowledge Sanderson, 2007

29. Methods • Interviews • Perceptions of experiences, stories, opinions, can be used to assess individual knowledge • Focus Groups • Perceptions of experiences, opinions, feedback on new product/service, etc. • Considerations: content, data and analysis, external credibility, time for analysis, transcription, selection of group members and facilitator(s) Sanderson, 2007

30. Methods • Rubrics • Used to score subjective-type measures of performance • Involves prior determination of how performance will be rated • Answers the question: What does a satisfactory rating look like? Sanderson, 2007

31. Methods • Institutional Data • Demographic information • Enrollment • Retention • Migration • Ethnicity/race • Graduation • Majors • Post graduation success • Success in subsequent courses, etc. Sanderson, 2007

32. Assessment Planning • Implementation • Who does what? When? Where? How? • Often a matrix is used for this. Sanderson, 2007

33. Assessment Planning • Results • Data Analysis • Dictated mostly by the type of data you are collecting • Frequency distributions, some measure of central tendency, and may want to compare means or look for significant differences where applicable • Depiction of information • Graphs, tables—pictures can be very helpful in explaining data • Reporting • Do report and may need to produce more than one report depending on number of different audiences Sanderson, 2007

34. Assessment Planning • Decisions/Recommendations(Closing the Loop) • Now that we know--What are we doing about it? • Celebrate • Initiate changes • Study further • Enlist others to help with making further meaning of the data • Add to or take away from X • Revise assessment methods • When will we look at this again? Sanderson, 2007

35. Using Assessment Information in Decision-Making and Planning • Documentation is Important • As a record • To use as a guide for future decision-making • To talk with constituencies we serve • To use with staff and others • To show progress to ourselves and our constituencies Sanderson, 2007

36. Using Assessment Information in Decision-Making and Planning • Discussions with Staff • Share results with all staff and in multiple formats • Make results as transparent and public within the department as possible • Openness can build trust and fosters the integrity of the process • Department meetings, planning retreats, unit meetings Sanderson, 2007

37. Using Assessment Information in Decision-Making and Planning • Discussions with Students and others • For students to invest time in our assessment efforts they must see the value • Sharing results and including them in conversations about how to make improvements based on data builds investment Sanderson, 2007

38. Questions?? Comments?? OSU Student Affairs Research and Evaluation Web Site http://oregonstate.edu/studentaffairs/assessment/index.html Sanderson, 2007