380 likes | 497 Views
Getting Started in Evaluating Student Learning in Student Affairs/Services. Rebecca A. Sanderson, PhD Director, Student Affairs Research and Evaluation Oregon State University. San Diego State University Evaluating Institutional Learning Centeredness Conference July 12-14, 2007 .
E N D
Getting Started in Evaluating Student Learning in Student Affairs/Services Rebecca A. Sanderson, PhD Director, Student Affairs Research and Evaluation Oregon State University San Diego State University Evaluating Institutional Learning Centeredness Conference July 12-14, 2007 Sanderson, 2007
Assessment • Definition “a rich conversation about students and student learning informed by data.” (adapted from Ted Marchese – AAHE) . . .the systematic collection, review, and use of information about programs/services undertaken for the purpose of improving student learning and development. (adapted from Palomba & Banta, 1999) Sanderson, 2007
What is Assessment? • A systematic process of gathering information upon which to make decisions • Using information to determine what is working and what is not • An evaluation of effectiveness • Systematic feedback • An integrated and circular process Sanderson, 2007
Assessment • Why? • To improve student learning in essential areas (e.g., diversity/multiculturalism) • To improve programs and services • To ensure that students are learning that which we intend • To focus efforts • To inform decision-making and clarify intentions Sanderson, 2007
Assessment This is important—do it. . . ! And then there is this. . . . • Accreditation • Administrative mandates • Accountability • Spelling Commission report • Expectations from professional organizations • Another thing to add to an already busy schedule Sanderson, 2007
Benefits • Keeps the focus on students • Provides systematic information to guide program development • Can transform collaborations between Student Affairs and Academic Affairs • Fosters improvement Sanderson, 2007
Limitations • Outcomes being measured may not reflect the true impact of a program/service • Probably will not prove that your program alone caused the outcome • Will not answer every question you may have • May take longer and need more resources than you expect Sanderson, 2007
Basic Assessment Questions • What are we doing and why are we doing it? • What do we expect a student to know, to think, or to do as a result of our program? • How well are we doing it? • How do we know? • How do we use the information to improve? • Does that work? (Bresciani, 2002) Sanderson, 2007
Getting Started • Questions to ask yourself • Why do assessment? • How will it be used? Politics? • How do you ensure broad input and active participation? • How do you engage in reflective conversation about data that informs decision making? • With whom, how, and when do you share? • How do you sustain the effort? Sanderson, 2007
Assessment Planning • Successful assessment finds that a written assessment plan is essential • To think through assessment needs and capture agreement about what matters • To give direction for action--To know who, when, where, what • To provide a means to determine if progress is being made Assessment Plan ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ Sanderson, 2007
Structural Support Issues • Assessment is best done inclusively • An Assessment Council? • A Coordinator or director position? • A common language • Training and education over time • Feedback on assessment efforts • Tie to improvement rather than to accountability to start • Visible and vocal support from the top Sanderson, 2007
Assessment Accountability Structure Vice Provost for Student Affairs OSU Division of Student Affairs Director, Research and Evaluation Student Affairs Assessment Council Student Affairs Departments/Units Programs/Services Sanderson, 2007
OSU Assessment Council • Anyone can join • Everyone agrees • to learn, • to help others learn, and • the work goes on even if a person misses a meeting • to share the work • We discuss and come to consensus around important issues • We strive for excellence and also for joy Sanderson, 2007
Creating Buy-In • Begin small, but meaningful • Data doesn’t need to be collected annually—create a cycle • Show your work to department • Get input • Include students • Show how data is being used • A successful experience can go a long way Sanderson, 2007
Elements of Assessment Planning • Department or Program Mission • Goals • Intended Student Outcomes • Methodology • Implementation • Results • Decisions/Recommendations Sanderson, 2007
Assessment Planning • Mission • Describes the purpose of the organization and the constituents served • Should be able to state the mission in less than 25 words Sanderson, 2007
Assessment Planning • Goals • Broad general statements of what a program wants its constituents to know or to do. Goals generally describe what the program is trying to accomplish. • It is not a “to do” list. • Is aligned with university goals and the departmental mission • Provides departmental focus Sanderson, 2007
Logic Model MISSION Mission:Describes the purpose of the organization and the constituents served. It clearly relates to the Oregon State University and the Division of Student Affairs Missions. Goals:Broad general statements of what a department wants its constituents to know or do. Goals generally describe what the program is trying to accomplish. Typically only 4-6 goals for a department. GOALS Programs:Sets of related activities and outcomes that consume a meaningful portion of the departmental resources (persons, dollars, time, etc.) and that are designed to support the department’s goals PROGRAMS INPUTS ACTIVITIES SERVICE OUTCOMES LEARNING OUTCOMES • Resources dedicated to the program: e.g., Money, Staff, Time, Equipment • Constraints on the program: e.g., Laws, Regulations, Policies Activities done to deliver the program: e.g.,Provide workshops Advise students Distribute brochures Develop handbook Teach classes Provide training Give tests Products from the activities: e.g., Number of workshops Number of people advised Types of brochures produced % served % satisfied Amount of money collected Benefits for participants: e.g.,Gained new knowledge Increased skill Modified behavior Improved their condition Positively altered their status Sanderson, 2007
LOGIC MODEL Influences other than your program Longer-term Outcomes Intermediate Outcomes Initial Outcomes Time Outputs What level of outcome do you have the resources to measure? What level of influence to do believe your program will have? Activities Inputs How is this linked to what we know about student learning? Sanderson, 2007
Assessment Planning • Learning Outcomes • Detailed and specific statements derived from the goals. They are specifically about the intended end results of your program efforts and typically use active verbs such as: arrange, define, explain, demonstrate, etc. • Levels of learning or mastery—Bloom Sanderson, 2007
Writing Learning Outcomes • Target group • Targeted Learning • Level of Mastery (Bloom’s Taxonomy) • Learning Outcome: • (Target group) will be able to (Bloom’s Taxonomy word)(Targeted Learning). Sanderson, 2007
Assessment Planning • Methods • The criteria, process, and tools used to collect evidence and to determine the degree to which the intended outcomes were reached. • Assessment methods include: • the target audience, • the methods and tools for data collection, • criteria or targets that tell you when the outcome has been met, and • how the data will be analyzed. Sanderson, 2007
Considerations • What method(s) will get the data to answer the question you are asking? • What level of reliability and validity are needed? • Reliability--consistency of measurement • Validity—measures what it purports to measure • Does it make sense and look like it measures what we want it to measure? • What is the Timeliness, Cost, Motivation? Sanderson, 2007
Assessment Planning • Methods • Types • Survey • Tests • Performance-based Measures • Checklists • Interviews & Focus Groups • Rubrics • Institutional Data Sanderson, 2007
Methods • Survey • Self-reported information: Demographic/descriptive data, attitudes, opinions, values, experiences, behaviors, expectations, goals, needs • Dependent on accurate and honest recall • Can be commercial and/or standardized • Internally developed Sanderson, 2007
Methods What no multiple choice!! H-m-m-m?? • Tests • Cognitive or thinking information • Can include written and oral presentations of material • Can be commercial and/or standardized • Internally developed Sanderson, 2007
Methods • Performance-based Measures • Direct evidence of learning through performance • e.g., projects, work samples, capstone experiences, direct observation • Must develop criteria for evaluating the performance Sanderson, 2007
Methods • Checklists • Direct evidence of • presence, • absence, • frequency of a behavior. • Often used with direct observation, can be used also for content knowledge Sanderson, 2007
Methods • Interviews • Perceptions of experiences, stories, opinions, can be used to assess individual knowledge • Focus Groups • Perceptions of experiences, opinions, feedback on new product/service, etc. • Considerations: content, data and analysis, external credibility, time for analysis, transcription, selection of group members and facilitator(s) Sanderson, 2007
Methods • Rubrics • Used to score subjective-type measures of performance • Involves prior determination of how performance will be rated • Answers the question: What does a satisfactory rating look like? Sanderson, 2007
Methods • Institutional Data • Demographic information • Enrollment • Retention • Migration • Ethnicity/race • Graduation • Majors • Post graduation success • Success in subsequent courses, etc. Sanderson, 2007
Assessment Planning • Implementation • Who does what? When? Where? How? • Often a matrix is used for this. Sanderson, 2007
Assessment Planning • Results • Data Analysis • Dictated mostly by the type of data you are collecting • Frequency distributions, some measure of central tendency, and may want to compare means or look for significant differences where applicable • Depiction of information • Graphs, tables—pictures can be very helpful in explaining data • Reporting • Do report and may need to produce more than one report depending on number of different audiences Sanderson, 2007
Assessment Planning • Decisions/Recommendations(Closing the Loop) • Now that we know--What are we doing about it? • Celebrate • Initiate changes • Study further • Enlist others to help with making further meaning of the data • Add to or take away from X • Revise assessment methods • When will we look at this again? Sanderson, 2007
Using Assessment Information in Decision-Making and Planning • Documentation is Important • As a record • To use as a guide for future decision-making • To talk with constituencies we serve • To use with staff and others • To show progress to ourselves and our constituencies Sanderson, 2007
Using Assessment Information in Decision-Making and Planning • Discussions with Staff • Share results with all staff and in multiple formats • Make results as transparent and public within the department as possible • Openness can build trust and fosters the integrity of the process • Department meetings, planning retreats, unit meetings Sanderson, 2007
Using Assessment Information in Decision-Making and Planning • Discussions with Students and others • For students to invest time in our assessment efforts they must see the value • Sharing results and including them in conversations about how to make improvements based on data builds investment Sanderson, 2007
Questions?? Comments?? OSU Student Affairs Research and Evaluation Web Site http://oregonstate.edu/studentaffairs/assessment/index.html Sanderson, 2007