Creating Effective Assessment Tools by Anne Marie Karlberg

1. Creating Effective Assessment Tools Anne Marie Karlbergamkarlberg@gmail.comJanuary 30th, 2009

2. Overview • What has been done at NWIC in terms of course outcomes? • Why use an assessment tool? • 2 Basic Assessment tools • (i) Checklists • (ii) Rubrics: • a. The analytic rubric (scoring guide) • b. The holistic rubric • Rubric websites • Break into groups: Work on creating rubrics for course outcomes

3. Overview • What has been done at NWIC in terms of course outcomes? • Why use an assessment tool? • 2 Basic Assessment tools • (i) Checklists • (ii) Rubrics: • a. The analytic rubric (scoring guide) • b. The holistic rubric • Rubric websites • Break into groups: Work on creating rubrics for course outcomes

4. Assessment Data • Direct indicators (outcomes): e.g., essays, capstone projects, demonstrations, presentations

5. Assessment Data • Direct indicators (outcomes): e.g., essays, capstone projects, demonstrations, presentations • Indirect indicators (perceptions): e.g., surveys, focus groups, interviews

6. Assessment Data • Direct indicators (outcomes): e.g., essays, capstone projects, demonstrations, presentations • Indirect indicators (perceptions): e.g., surveys, focus groups, interviews • Institutional data: e.g., retention, graduation, enrollment, transfer trends

7. Assessment Levels • Course level • Program level • College level

8. Examples of Activities at Each Level of NWIC’s Assessment Program

9. Examples of Activities at Each Level of a Tribal College Assessment Program

10. History of course outcomes at NWIC Beginning in 2003, faculty started consulting with one another and developed consistent outcomes for many courses – regardless of where or how the courses were being taught (i.e., course outcomes forms).

11. History of course outcomes at NWIC • Each course outcomes form and syllabus identifies two types of learning outcomes that will be assessed during the course… • NWIC outcomes • Course specific outcomes (unique set of outcomes)

12. History of course outcomes at NWIC The outcomes are assessed in two ways: • Directly: instructors assess students’ work • Indirectly: students identify how well they think they have mastered each of the outcomes on course evaluation forms at the end of the quarter

13. History of course outcomes at NWIC The outcomes are assessed in two ways: • Directly: instructors assess students’ work • Indirectly: students identify how well they think they have mastered each of the outcomes on course evaluation forms at the end of the quarter

14. Characteristics of effective course outcomes

15. Course Outcomes Plan

16. Overview • What has been done at NWIC in terms of course outcomes? • Why use an assessment tool? • 2 Basic Assessment tools • (i) Checklists • (ii) Rubrics: • a. The analytic rubric (scoring guide) • b. The holistic rubric • Rubric websites • Break into groups: Work on creating rubrics for course outcomes

17. 2. Why use an assessment tool? Makes the criteria for assessment explicit. Guides the evaluation of students’ work and makes it less subjective. Improves the assessment of learning. Enhances learning. Assists in targeting important content and in sustaining consistent attention to evaluative criteria. Can be used for self and peer evaluation. Increases consistency among instructors.

18. Overview • What has been done at NWIC in terms of course outcomes? • Why use an assessment tool? • 2 Basic Assessment tools • (i) Checklists • (ii)Rubrics: • a. The analytic rubric (scoring guide) • b. The holistic rubric • Rubric websites • Break into groups: Work on creating rubrics for course outcomes

19. The checklist Isn’t about quality—it’s just about the parts that are necessary. Is a good indicator of “can do–can't do” and “done–not done”. Defines the parts that make up a complete task (either product or process). Is useful for formative evaluation of a product or process.

20. Checklists: Use checklists to… Record observed performance. For example,

21. Use checklists to… Keep track of progress over time (e.g., as an inventory of skills at the beginning and end of a course). For example,

22. Use checklists to… Indicate the minimum requirements of a product or performance, which are not included in your qualitative criteria (i.e., rubrics). For example,

23. Use checklists to… Help students fulfill task requirements. For example,

24. An example of a “process” checklist For working in a team, giving a presentation, working to solve a problem, etc.

25. An example of a “product” checklist For example, papers, portfolios, projects, solutions, etc.

26. Advantages of checklists • Easy to construct and use. • Align closely with tasks. • Effective for self and peer assessment. • Make learners aware of task requirements, allowing them to self-monitor progress. • Useful for sharing information with parents and others. • Provides basic formative feedback.

27. Disadvantages of checklists • Provide limited information about how to improve performance or product. • Do not indicate relative quality of performance or product. • Do not provide information for assessing grades.

28. Overview • What has been done at NWIC in terms of course outcomes? • Why use an assessment tool? • 2 Basic Assessment tools • (i) Checklists • (ii) Rubrics: • a. The analytic rubric (scoring guide) • b. The holistic rubric • Rubric websites • Break into groups: Work on creating rubrics for course outcomes

29. Rubrics • A scoring scale used to assess student performance along a task-specific set of criteria • Comprised of two components: criteria and levels of performance • Some rubrics include descriptors: describe more precisely what performance looks like at each level

30. Criteria Level of performance

31. Criteria Descriptors Level of performance

32. Possible terms for level of performance

33. Adjectives and Adverbs for Descriptors

34. 2. Written Communication Skills: Outcome a. The student writes standard English

35. 2. Written Communication Skills: Outcome a. The student writes standard English

36. Chocolate chip cookie rubric What cookie elements could we judge? • Number of chocolate chips • Texture • Color • Taste • Richness (flavor)

37. Chocolate chip cookie rubric

38. Chocolate chip cookie rubric

39. Chocolate chip cookie rubric

40. Chocolate chip cookie rubric

41. Chocolate chip cookie rubric

42. Rubrics • Advantages: provides • Clearer expectations • More consistent and objective assessment • Better feedback

43. Overview • What has been done at NWIC in terms of course outcomes? • Why use an assessment tool? • 2 Basic Assessment tools • (i) Checklists • (ii) Rubrics: • a. The analytic rubric (scoring guide) • b. The holistic rubric • Rubric websites • Break into groups: Work on creating rubrics for course outcomes

44. a. Analytic rubrics (scoring guide) • describes specific details we are looking for in the work (provides a lot of information) • performance is judged separately for each criterion. • useful for formative evaluation of a product or process • many formats

45. See hand-out

46. Overview • What has been done at NWIC in terms of course outcomes? • Why use an assessment tool? • 2 Basic Assessment tools • (i) Checklists • (ii) Rubrics: • a. The analytic rubric (scoring guide) • b. The holistic rubric • Rubric websites • Break into groups: Work on creating rubrics for course outcomes

47. b. Holistic rubrics • assigns a level of performance by assessing performance across multiple criteria as a whole • describes criteria more holistically (provides fewer details, does not list separate levels of performance for each criterion) • is time-saving for the instructor • is useful for summative evaluation of a product or process (tool to advance)

48. See hand-out

49. Compare analytic and holistic rubrics

50. Examples of rubrics