Using Rubrics to Assess Student Learning

1. Using Rubrics to Assess Student Learning Emily Langdon, PhD Coordinator for Assessment, Research, and Evaluation Division of Student Affairs Laura Martin, PhD Coordinator for Institutional Assessment Office of Institutional Assessment Anne Zanzucchi, PhD Faculty Development in Assessment Coordinator Center for Research on Teaching Excellence

2. Learning Outcomes for Today: • Participant will be… • Able to describe what a rubric is and list benefits of using rubrics • Familiar with different kinds of rubrics • Able to describe the role rubrics play in assessment that provides information for program planning • Able to use a rubric to assess student work to gain actionable information on student learning

3. What is a Learning Outcome? • A change in attitude, aptitude or behavior that a student can describe or demonstrate after participating in a program(s) or using a service(s) (CSU Sacramento) • Ex. As a result of the health awareness workshop, students can explain how exercise affects stress. • Ex. As a result of the time management workshop, students have identified two tools to better manage their schedule.

4. What is a Learning Outcome? • Term refers both to intended and actual, observed outcomes • In doing so, it clarifies the role of LOs in program planning

5. Role of LOs in program planning Intended LO Actual LO

6. This is the “assessment cycle” Step 2 in Assessment Cycle: Conclusions Step 1 in SA Assessment Cycle: Identify LOs, POs & measures Step 2 in SA Assessment Cycle: Results

7. At what levels are LOs Developed & assessed? Institution Division of Student Affairs Academic SA Department SA Department Program Program Program Program Program Program Program Program

8. What is a rubric? what roles do they play in the assessment of student learning and program planning in student affairs?

9. What is a rubric? • A scoring guide: • a list or chart that articulates the criteria and standards of achievement to be used to evaluate work (Suskie, 2009) • a set of criteria specifying the characteristics of an outcome and the levels of achievement for each characteristic (J. Levy, 2012)

10. What are rubrics used to score? • Assessments that require the observation of a performance or behavior(s), ex. • Presentation • Role plays • Teamwork • Performances • Assessments of written or visual artifacts, ex. • Reflection papers • Journals • Resumes • Portfolios • Art pieces

11. What is the relationship between learning outcome and a rubric? • Learning Outcome describes what students will do to demonstrate their learning, ex. • Give a presentation • Role play a…. • Work in team to… • Write a reflection that… • Rubric describes • The expected properties of that demonstration(criteria) • The possible levels of achievement/performance (standards)

12. Example learning outcome and Rubric pairing Learning Outcome: Reflecting teamwork instruction in the leadership workshop, students will employ their teamwork skills in the execution of their final group project. • Scoring Rubric: • Teamwork rubric defines four criteria and related standards for effectiveteamwork that will be examined through the interactions needed to complete the final group project: • Contributing to team meetings • Facilitating the contributions of others • Fostering constructive team climate • Responding to conflict

13. Types of rubrics • Rubrics differ in structure - in the way criteria and performance standards are described. And thus, • In the kind of information they can provide about the quality of a piece of work • Choice of rubric depends upon what you want to use the information for. • All rubrics enable collection of qualitative and quantitative data

14. Major Types of Rubrics • Checklist • Rating Scale • Descriptive (also called analytic) • Holistic

15. Example of a Checklist rubric for a website (suskie, 2009)

16. Example of a Rating Scale Rubric for Evaluating Fellow Group Members (suskie, 2009) Standards

17. Example of a Descriptive (Analytic) rubric Standards Criteria

18. Example of a Holistic Rubric For Assessing Student Essay (Allen, 2004) Standards Criteria

19. Identify learning evidence to be collected and that will use a rubric to score Where do rubrics fit into cycle? Might refine rubric as part of this step Use rubric to score student work (measure learning) … Develop or identify and pilot rubric here

20. Activity: Scoring student work with a rubric. Steps: • Score student work using rubric. • Review data for inter-rater reliability. What if differences in scoring? • Summarize results – how best to do that? • Identify possible actions suggested by results.

21. Activity: Scoring student work with a rubric. Do set up with learning outcome….. And pull in assessment cycle. Working in pairs, use the rubric to score a set of student work. Provide scores to Laura to enter in spreadsheet.

22. Review Scores: Check for Inter-rater reliability. • Examine degree to which scores for same paper agree. • If diverge, why? • What do? • Ask another rater to score • Revise rubric to clarify • Best practice: “Calibrate” reviewers to apply rubric before scoring using example work so that reviewers share understanding of how to apply rubric

23. Summarize Results • Goal is information on which we can act, so frequencies with which particular levels of performance are observed are preferred relative to average performance for the group. • Do our example – calculate a mean and represent the results as a frequency, which is more intuitive to folks?

24. Summarize Results: Other problems with averages • Averages may not appropriately describe distribution of student performance. Ex. • ADD an example here using our rubric….showing that a mean may show that student performance is just fine but really it is dichotomous, students either did poorly or very well. • Averages are not appropriate because they assume that the difference between each standard is equal.

25. Interpreting results and taking action: What do our results suggest to you? 1) Consider student learning results: • Are you satisfied? If not, what might you do? 2) Consider usefulness of rubric: • Does it work well? Could it be improved? How?

26. Resources for rubrics • Google – tons out there • Professional societies, colleagues, networks, listservs • VALUE Rubrics • Rubistar - http://rubistar.4teachers.org/ • Develop it yourself

27. What makes a useful rubric? • The assignment asks students to produce work that addresses the rubric criteria (assignment, instruction, and rubric criteria are aligned). • Design rubric before giving the assignment, not after. • As possible, give rubric to students with assignment. Consider having them self-score before turning assignment in. • Align rubric criteria with learning outcome and the assignment that asks students to produce the outcome.

28. What makes a useful rubric? • Criteria and standards are sufficiently distinct and clear that raters consistently apply them in the same way (inter-rater reliability) • Use an even number of standards (ex. 4) to avoid tendency to score in middle. • The rubric is designed to yield the kind of information about student abilities needed to make planning decisions.

29. How can refine our rubric before we use it to conduct assessment? • Develop or identify rubric shaping it to needs • Pilot rubric by applying it to example work • Share rubric with students to understand how they interpret it • Revise/refine rubric to increase inter-rater reliability and usefulness to students

30. Activity: What might you use a rubric for in your own work? Reflect on your own program work, • Identify an opportunity to apply a rubric. • What would you use the rubric to evaluate? Why? • What kinds of rubric (analytic, holistic, etc.) would you choose and why?

31. What are some advantages of using rubrics to students?(suskie, 2009) • Provide to students to help them understand your expectations. Best practice – provide with the assignment • Students can apply rubrics themselves to guide learning and improvement (ex. self-score before turn in) • Improves feedback to students by identifying specific areas for improvement • Gather information that can be used to improve instruction by highlighting challenges shared by a significant proportion of students

32. What are some advantages of using rubrics for teachers and programs?(suskie, 2009) • Development of criteria and standards help to clarify vague or fuzzy goals for a program and among colleagues • Makes expectations public and shared; all colleagues can understand precisely what intend students to be able to do • Make scoring easier and faster • Increase consistency in scoring across students, among raters, and through time; allow to measure improvement • Can gather complementary indirect evidence through student self-ratings • Makes it easier to identify common strengths and weaknesses in student work, behavior, etc. to inform program planning.

34. What makes a Good Learning Outcome? • Written from perspective “Students will be able to….” • Includes an action verb describing specifically what students will be able to do if they have learned • Describes how students will demonstrate their learning (i.e. is measurable) • Yield information that can be used to refine programming (i.e. actionable information) • Is understandable to students!

35. Example Learning Outcomes Original: Participants will be able to appreciate the benefits of exercise Revised: Participants will be able to explain how exercise affects stress

36. Example Learning Outcomes Original: Participants will have more confidence in their abilities to address racial discrimination Revised: Through role play, participants will demonstrate their ability to analyze and respond to arguments about racial discrimination

37. Activity: Refining LOs Original: Participants will understand the nine reasons for conducting a needs assessment. • With a partner, answer the following questions and, as necessary, develop a revised statement: • Is the verb active, specifying what students can do if they have learned? • Can it be measured? Does it describe how students will demonstrate their learning? • Will it yield actionable information? • Is it understandable to students?

38. Activity: Refining LOs Original: As a result of the time management workshop, students will arrive to class on time daily. • With a partner, answer the following questions and, as necessary, develop a revised statement: • Is the verb active, specifying what students can do if they have learned? • Can it be measured? Does it describe how students will demonstrate their learning? • Will it yield actionable information? • Is it understandable to students?

39. What kind of learning evidence is often most easily collected and “actionable”? Direct Evidence: • Evidence of learning that is tangible, visible; actual student work or demonstrations of student ability. Exs. • Written or oral explanations, descriptions, identifications, etc. • Role plays • Lists • Project products • Etc.

40. What is indirect evidence? Indirect Evidence: • Proxy signs that students are learning, for example, student self-perceptions of abilities. Exs. • Questions that ask students to rate their own abilities • Questions that ask students to rate gains in abilities or knowledge (ex. Relative to the start of this workshop, how well can you…) • Satisfaction type questions on surveys

41. What is the relationship of LOs to measures in the SA template? • A refined LO will clarify the methodology for assessment because it indicates how students will demonstrate their learning. Examples…. • “…explain how exercise affects stress.” • “…identify the most appropriate university resource for addressing their academic or personal concern or need.” • “…list three reasons for…”

42. Worksheet activity: Developing and revising your own outcomes

43. Benefits of Learning Outcomes… for the Teacher and Department • Framework for program design/development = content and activities • Reference point for “researching” student learning, yielding insights relevant to future offerings • Articulates intentions of programming/service, which provides opportunities for educators to • Discuss and improve educational offerings • Align educational offerings within and among departments (co-curricular and academic) and in relation to division and institutional goals for student learning

44. Benefits of Learning Outcomes… for the Teacher and Department • Advertise value and impact of programming to build interest and buy-in of students, faculty, student affairs colleagues at UC Merced and beyond, alumni and potentially even donors! • Builds integrated picture of programming within department to support planning… • What are we doing collectively? Is it what we intend? • Are we investing our most precious resource (time) in ways that maximize our intended educational impact?

45. Benefits of Learning Outcomes… for Students • Explains to students why they should engage in programming or service • Supports student stewardship of their learning • Choosing to participate in light of personal goals and interests • Monitoring learning during program • Supports holistic integrated view of education, ex. connect classroom and co-curricular learning • Helps students to see that all experiences have educational value, enabling development as life long learners (an SA Divisional Outcome)

46. Activity: With a partner, compare and contract the example types* of rubrics & address the following questions. Imagine using the rubric to score student work: • What do you like about each type of rubric? What don’t you like? Why? • What if any rubric type(s) would you prefer to use or not to use? Why? • As you compare and contrast each type of rubric record your observations. * Please focus on the rubric’s structure and your thoughts about its usefulness, not the actual criteria and standards.

47. Where do rubrics fit into cycle? • A rubric is a scoring guide • A tool for determining to what degree students have met the learning outcome (measuring learning)

48. Example of hierarchical LO alignment Institution: Students will understand and value diverse perspectives in both the global and community contexts of modern society in order to work knowledgeably and effectively in an ethnically and culturally rich setting (Self and Society) Program – Cultural Expression: 100% of participants in the cultural expression workshop will identify an example of cultural expression they have observed and describe a contribution it makes to the UC Merced community. Division: Develop an understanding and appreciation of human differences Department – Housing and Residential Life: Through informal and formal residential life experiences and programming, resident interest in and respect for new ideas and cultural and life style differences increases.

49. Activity: At what level are you developing & assessing outcomes? • Review previous assessment work, at what level was student learning assessed?

50. Where do LOs come from? • Derived from/support departments goals • Address worthwhile and shared learning priorities • LOs are specific examples of how a department goal is met.