Selecting and Developing Assessment Approaches and Methods

1. Selecting and Developing Assessment Approaches and Methods Presented by Jennifer Fager Xavier University for University of Wisconsin-Superior Enhancement Day 1/19/2011

2. Guiding Principle • The assessment of student learning should become a system whereby planning, data collection, analyses, and improvement are included • Reactions?

3. Burning Questions • What sorts of things should a professor try to assess? • Having decided on what to assess, how should a professor go about assessing it? • What sorts of things should faculty try to assess in their programs? • Having decided on what to assess, how should faculty go about assessing these things?

4. What is your Assessment Pattern? • Why do you assess what you assess? • What are the reasons professors construct and use assessment instruments • Identify areas of deficiency • Understand end-of-instruction targets

5. Data-Driven Assessment • Professors make inferences and then decisions based upon the inferences • What does an A mean? B? C? F? • THUS It is important to clarify BEFORE the test is created the decisions that will be influenced by students’ performances.

6. How do you do that? • It’s time to give your mid-term. Before you do so, there are several questions that need to be addressed. • What should be tested? • What topics were discussed and for how long? • What type of items will you use? • How long will students have to take the exam? • How many items/points should be used given the amount of time available?

7. Two Fundamental Questions • What evidence do you have that students achieve your stated learning outcomes? • In what ways do you analyze and use evidence of student learning?

8. Another Question or two • What changes have you made to your programs, your institution, or your courses based upon evidence collected? • What evidence do you currently possess that might inform essential decisions that need to be made?

9. Defining Evidence • Information that tells you something directly or indirectly about the topic of interest • Evidence is neutral -- neither “good” nor “bad” • Requires context to be meaningful • Two types of assessment evidence • Direct and Indirect

10. Direct Evidence • Students show achievement of learning goals through performance of knowledge, skills: • Scores and pass rates of licensure/certificate exams • Capstone experiences • Individual research projects, presentations, performances • Collaborative (group) projects/papers which tackle complex problems • Score gains between entry and exit • Ratings of skills provided by internship/clinical supervisors • Substantial course assignments that require performance of learning • Portfolios • Course assignments • Others?

11. Indirect Evidence • Attitudes, perceptions, satisfaction, and experiences of learning and the learning environment • Students self-assessments of learning • Local student, alumni, employer surveys and questionnaires • Course Evaluations. • National Engagement Surveys and Satisfaction Surveys • NSSE, CCSSE, FSSE, BCSSE, SSI (Noel Levitz) • Focus Groups (student, faculty, employer) • Interviews (student, faculty, employer) • Others?

12. Finding Evidence: An Evidence Inventory • Lets you discover the evidence you already have, such as: • Institutional Research data • Student Life data • Exit Surveys (seniors) • Alumni Surveys • Start with the obvious … but don’t stop there

13. Finding Evidence: Perils and Pitfalls • Institutional history • “We’ve already done that, and it didn’t tell us anything!” • Territory; Politics • Fighting for scant resources • Institutional policy/culture about sharing information • “I don’t want somebody ‘policing’ my classrooms!” • Who owns the evidence?

14. Finding Evidence: Appropriateness • Does the evidence address student learning issues appropriate to the institution or the program? • Does the evidence tell you something about how well the institution or program is accomplishing its mission and goals? • The questions you have about student learning should guide your choice of appropriate existing evidence and identify gaps where a new type of evidence might be needed

15. Evidence Example • Attached to this packet are data and analysis examples. • Writing Results Rubric • IR Survey • CAAP results • Students Will Think Critically Form • Student Affairs data • Library Data

16. Assisting Academic Departments: Some Assumptions • Faculty are intensely interested in what students are learning • Assessment occurs in classrooms and academic departments every day • Evidence of student learning already exists in academic departments • The challenge is not to convince academic departments to gather evidence, but rather to help them recognize and use evidence they already have

17. Assisting Academic Departments: Addressing Common Barriers • “This is a lot of work!” • Use some sort of evidence inventory to help faculty understand how existing academic practices yield evidence • Keep expectations reasonable, given limited time and resources • Offer assistance and rewards • Remember: it is not necessary to gather all the evidence all of the time

18. Assessment Inventory: One Example Inventory of Written Statements and Plans • Do you have a written mission statement or statement of purpose?   yes  no If yes, please attach a copy or reference where this can be found: ________________________________________________________ • Do you have a written statement of intended educational outcomes describing what a student should know or be able to do when they have completed this program?  yes  no • Do you have a written method of assessment for measuring student outcomes?  yes  no • Does your program have a separate accreditation process?  yes  no

19. Assessment Inventory: One Example Direct Methods of Assessment 1. ________ Comprehensive Examinations 2. ________ Writing proficiency Examinations 3. ________ National Examinations assessing subject matter knowledge 4. ________ Graduate Record Exam General Test 5. ________ Graduate Record Exam Subject Test 6. ________ Certification Examinations 7. ________ Licensure Examinations 8. ________ Locally developed pre-test or post-test for subject matter knowledge 9. ________ Major paper/project 10. ________ Program/course portfolios 11. ________ Capstone coursework 12. ________ Audio/video tape of presentations/performances

20. Assisting Academic Departments: Addressing Common Barriers • “How do I know you won’t use this against me?” • Be consistent and firm in the message that assessment is not faculty evaluation, that results will only be reported in the aggregate • Partner with faculty willing to engage in the process and make her/his evidence public • Link assessment results to allocation of resources, ideally through a strategic planning process • If appropriate, develop policies regarding assessment

21. Assisting Academic Departments: Addressing Common Barriers • “My students pass the tests. Why isn’t that good enough?” • Tests often measure only content knowledge • Learning = what student know (content knowledge) + what they can do with what they know (performance) • Grades are generally not linked to specific learning outcomes and don’t aggregate well • Modify course tests to measure learning outcomes by adding performance assessments

22. Modifying Tests to Gather Direct Evidence of Learning • Identify questions on the test that provide evidence of a learning outcome: • Five questions that require the use of deductive reasoning to arrive at the right answer • Open-ended questions that require students to solve a unique problem given knowledge/skills learned • Isolate those questions and look for patterns of performance: • the average grade in the class was a “B” but 85% of the students missed four of the questions requiring deductive reasoning • 70% of students were able to use a particular theory/approach to resolve the problem

23. Meaningful Evidence • Situated within the institutional and departmental mission and context • Addresses relevant questions • Analyzed and interpreted in relation to other evidence • Examples?

24. Meaningful Evidence: Facts + Context • Fact: • National survey data indicates seniors do not feel a sense of engagement and belonging on our campus.

25. Meaningful Evidence: Facts + Context • Fact: • Seniors feel disengaged from our campus (national survey data) • Fact: • Seniors would recommend this institution to other people (senior exit surveys)

26. Meaningful Evidence: Facts + Context • Fact: • Seniors feel disengaged from our campus (national survey data) • Fact: • Seniors would recommend this institution to other people (senior exit surveys) • Context: • Over the past five years, an average of 82% of first-year alums donated to the institution

27. Recognizing Meaningful Evidence • How compelling is your evidence? • Does it make you want to do something? Will it make others want to do something? • How relevant is your evidence? • To what is it linked: departmental mission, institutional initiatives? • How trustworthy is your evidence? • How was it gathered? Who does it represent? Is it one piece? Several pieces?

28. What evidence do you have that students achieve your stated learning outcomes? * Who actually measures the achievement of student learning outcomes? * At what points in the curriculum or co-curricular activities are essential institutional (including general education), major, or program outcomes assessed? * How is evidence of student learning collected? * How extensive is the collection of evidence? In what ways do you analyze and use evidence of student learning? * Who analyzes the evidence? * What is your evidence telling you about student learning? * What systems are in place to ensure that conclusions are drawn and actions taken on the basis of the analysis of evidence? * How is evidence of the achievement of student learning outcomes incorporated into institutional planning and budgeting HLC Expanded Fundamental Questions

29. Meaningful Evidence: Example • Senior exit surveys: • Indicate a dissatisfaction with the amount of time spent on clinical skills • Departmental assessment of skill ability and development finds that, of the critical skills required: • students are outstanding on three of them, satisfactory on two, and not acceptable on two • Internship evaluations from supervisors consistently cite lack of ability in clinical skills

30. Meaningful Evidence: Qualitative Data • Appropriate uses: • Exploring an issue in more depth • Answering specific questions about individual experience: • Ex: How are you different now than you were before? • Ex: how did living with a host family inform your understanding of the culture? • Including student voices

31. Qualitative Data Analysis: Open-Ended Questions • Read the data • Strip and code the data, while looking for themes and patterns • Present the data thematically---it will “lead” you somewhere • Academic Advising • General Education • Student perceptions of particular courses

32. Qualitative Data Example • “420 was a senior level course but I felt like a freshman! There was no way I knew all of that stuff.” • “I thought I was going to fail 420 and I’m a good student.” • “I didn’t know how to do anything in 420 and the instructor didn’t care. We kept saying we didn’t know but he just kept going. It was ridiculous.”

33. Qualitative Data Example • Drill down into the data by asking pertinent questions: • What are the learning goals of 420? • How did students perform in 420? • What are the assumptions about students entering 420? • Skill level? • Knowledge base? • Analyze the program curriculum map • Where do students learn prerequisite skills and/or knowledge? • How and where are program and course learning outcomes (expectations) assessed? Are they assessed?

34. Using Assessment Results • Inform policy decisions • Strategic allocation/reallocation of resources • Make changes in curriculum • Support new initiatives • Accountability • Inform stakeholders about expectations and results • Improve teaching and learning on campus

35. Presenting Assessment Results • Consider audience • Who are they? What’s important to them? How will they use assessment information in their lives? • Appropriate presentation • Present data thematically/topically • Link data and interpretations to institutional initiatives or departmental strategic planning (provide a context)

36. Assessing and Improving Assessment • Were the assessments reasonable and manageable? • Did they answer your questions? • Did they tell you something about student learning? • Were you able to use the evidence you gathered? • What else do you need to know?

37. Questions and Comments • Where do you need to go from here? • What is your assessment system for the program in which you teach? • Is the system understood by all stakeholders? • Does the system reflect the discipline?

38. Helpful Sources • Diamond, Robert M. “Designing and Assessing Courses & Curricula” (1998) • Allen, Mary J. Assessing Academic Programs in Higher Education (2004) • Huba, Mary E. and Jann E. Freed. “Learner-Centered Assessment on College Campuses” (2000) • Suskie, Linda. “Assessing Student Learning: A Common Sense Guide” (2004) • Walvoord, Barbara E. “Assessment Clear and Simple” (2004)