Checklists & Rating Scales

1. Checklists & Rating Scales Scoring Performances

2. Goals • understand the importance of observing performances and grade them appropriately • understand, spot, compensate for usual errors • understand and use appropriately the following observational tools: • checklists • rating scales • exemplars • keys Observation Checklists

3. Observation Checklists

4. Principles of Effective Rating • Characteristics should be educationally significant: in harmony with department’s vision, goals, and objectives and desired learning outcomes. • Characteristics should be directly observable: occur in the setting and clearly visible to an observer – skills – less so are interests, feelings, and attitudes. Observation Checklists

5. Principles of Effective Rating • Characteristics and points on the scale should be clearly defined. • Between three and seven rating positions should be provided, and raters should be permitted to mark at intermediate points. • Raters should be instructed to omit ratings when they feel unqualified to judge. Observation Checklists

6. Principles of Effective Rating • Ratings from several observers should be combined whenever possible -> greater reliability. Observation Checklists

7. Checklists and Rating Scales • are MONITORING and • RECORD-KEEPING devices • With increased emphasis on self-reflection and self-assessment, individual students should use self-checklists and rating scales to ASSESS THEIR OWN PROGRESS and DEVELOP AN IMPROVEMENT PLAN. Observation Checklists

8. from adult framework recorded objective & nonjudgmental consider whole learner regular, systematic, frequent observing include spontaneous & ongoing observations efficient recording purposeful monitor product & process before, during, after task from many points of view use spectator and participant modes Effective Observations Observation Checklists

9. Affective Measurement Problems • Fakability • Self-deception • Semantic problems • Criterion inadequacy Observation Checklists

10. Checklists & Rating Scales • Checklist: Attitude – Yes or No • Checklist: Attitude – Yes or No? • shows originality Yes No • respects own work Yes No • is easily motivated Yes No • Rating Scale: Attitude -- Degree • Rating: Attitude SA A U D SD • shows originality SA A U D SD • respects own work SA A U D SD • is easily motivated SA A U D SD Observation Checklists

11. Constructing Checklists • Base on course objectives. • State in clear and precise terms. • Avoid generalities & focus on specifics. • Design them so they can be used as a basis for discussion by instructor and student. Thus, write them so they are easily understood by the student. • May also be used for self- and peer-evaluation. Observation Checklists

12. Constructing Checklists • Keep them short so that they can focus on a few specifics. • May be designed for either a single evaluation or a cumulative record. • Can be used for diagnostic, formative, and/or summative evaluation as well as a basis for anecdotal comment, as well as an instructional tool for learner to follow in practicing the skill. Observation Checklists

13. CHECKLISTS psychomotor & social skills indicate presence or absence/mastery or non-mastery of skill may indicate sequence of actions in a performance RATING SCALES for psychomotor and affective skills indicate quality, degree, or frequency of skill or trait useful in summarizing general impressions Differentiating Assessment Tools Observation Checklists

14. Checklist: Preparation for Varnishing • __ sands and prepares surface properly • __ wipes dust from surface with appropriate cloth • __ selects appropriate brush • __ selects varnish and checks varnish flow • __ pours needed amount of varnish into clean container, etc. Observation Checklists

15. Rating Scale: Job Attitude • is enthusiastic G Av N Imp • is willing to work & cooperate G Av N Imp • desires to improve G Av N Imp • is a good team worker G Av N Imp • is concerned with safety G Av N Imp • works without supervision G Av N Imp • aware of job requirements G Av N Imp Observation Checklists

16. T-Form: Encouragement • Looks Like__________Sounds Like_____ Observation Checklists

17. Why Use Observation Checklists • A quick and easy way to observe and record many of the skills and behaviors rarely assessed prior to final test or summative exam. • Show areas that need work early enough to provide help and correct before student fails area/unit. Observation Checklists

18. Why Use Observation Checklists • Provide opportunities to change gears” re feedback. • The characteristics of intelligent behavior can be identified, taught, and checked. • Observation checklists let us check off whether student can demonstrate skill or attribute being measured. Develop base; practice; check progress. Observation Checklists

19. Why Use Observation Checklists • They focus on observable performances or criteria that are often more especially useful for monitoring growth or need for improvement. They are more meaningful or authentic. • Useful re what learners can do and what their learning needs are. Observation Checklists

20. Why Use Observation Checklists • Useful to discover patterns, assess progress, and make plans to help students continue their learning. Observation Checklists

21. How to Use Observation Checklists • First, conduct a TASK ANALYSIS to determine specific areas to include in the observation checklist. • Then, ensure students are aware of the areas that will be observed. • Do this by training participants in what the skill “LOOKS LIKE” and “SOUNDS LIKE” – even “FEELS LIKE.” Observation Checklists

22. How to Use Observation Checklists • Students may be assessed by peers or perform a self-assessment, before an instructor assessment. • Model, Teach, and Practice the skills prior to observation. • Students should work with the instructor to develop the indicators. Observation Checklists

23. Indicators for Persistence • knows how to access information • tries several approaches • does not give up quickly • has patience • brainstorms alternative solutions • checks own work • Do so in a T-Form. Observation Checklists

24. Making Valid Observations • Plan in advance what is to be observed and prepare an observation list, guide, or form to make the observations more objective and systematic. • Concentrate on a few significant behaviors at a time. • Use clear, unambiguous terminology and accompanying descriptions in the observation tools. Observation Checklists

25. Making Valid Observations • Each trait [item] should be mutually exclusive for coding. • The observer must be cognizant of the time-sampling errors: use frequent, short observations distributed over a period of several weeks and at different times of the day and week. Observation Checklists

26. Making Valid Observations • Coordinate the observations with your teaching: instructional strategies and observations must be coordinated. • Extensive observations should be selective – the most needy cases. • Carefully record and summarize the observation immediately after it has occurred. Observation Checklists

27. Making Valid Observations • Make no interpretations concerning the behavior until later on. Doing so may interfere with objectivity. • Have categories and coding schemes that are simple to use, that call for behaviors easily observed, and that deal with behaviors that can be conveniently recorded. Inferences should occur only after data have been gathered and recorded. Observation Checklists

28. Making Valid Observations • Wherever possible, observers and observations should be unobtrusive. When people know they are being observed, they often tend to avoid or overemphasize certain behaviors to gain attention. Observation Checklists

29. Checklists Basics • for psychomotor processes or products and for personal/social growth • for both processes and products • for analyzing sequences and correct order of actions • for traits present in a product Observation Checklists

30. Checklists Basics • Limitations • indicates present or absent only – no degree specified: an all-or-nothing scale. • Not useful for summarizing impressions as no degree indicated. Observation Checklists

31. Checklists Basics • Perform a careful task analysis to determine component behaviors. • Clearly specify the actions/traits to be observed. • Include in the list common errors where they are likely to occur. • Arrange procedures in the order they are expected to occur. Observation Checklists

32. Checklists Basics • Keep separate checklists for each person. • Transfer results to a master list of all candidates [for making comparisons]. • Give clear directions. • Train observers. Observation Checklists

33. Common Errors in Rating • Personal Bias: general tendency to rate all at approximately the same position on the scale • generosity error - rates all at high end • severity error – rates all at low end • central tendency error – rates all as average Observation Checklists

34. Common Errors in Rating • Halo Effect: our general impression of a person influences the rating of individual characteristics • Logical Error: two characteristics [intelligence and achievement; gifted and poor social adjustment] are rated as more or less related than they actually are. Observation Checklists

35. Common Errors in Rating • Rater’s Attitude: doesn’t view rating as important and doesn’t put much into it. • Not enough time to do a good job: need more extensive opportunities for observation. Observation Checklists

36. Resulting Problems • Puts in doubt a single rating of an individual – Limits the range of an individual’s ratings. • Obscures strengths and weaknesses on different traits because the individual receives similar ratings on all characteristics. • Higher relationship implied than actually exists. • Not done, invalid, and/or flawed results. Observation Checklists

37. Overcoming Problems • Proper design, training, and use of the scales. Observation Checklists

38. Performance Indicators • Quality • Quantity • Waste • Safety • Improvement • Focus: Finished student product • Process: Process student uses to produce product Observation Checklists

39. Quality Indicators • conformity to overall specifications • dimensions, spacing, and/or strength • suitability for intended purpose • general appearance • error rate during performance • efficiency of steps • choice of tools, equipment, and/or materials Observation Checklists

40. Time Indicators • time needed to complete product or perform process • number of products produced Observation Checklists

41. Improvement Indicators • improvement in one or more qualitative characteristics • improvement in one or more quantitative characteristics • reduction in the number of steps for which assistance is required Observation Checklists

42. Waste Indicators • number of unacceptable products produced • amount of excess material; used during performance Observation Checklists

43. Safety Indicators • Degree to which completed product is safe for intended use • Safe handling of tools, materials, and/or equipment • Accident rate during performance Observation Checklists

44. What is QUALITY? • If something occurred occasionally, it would occur __% of the time. • If something occurred sometimes, it would occur __% of the time • If something occurred frequently, it would occur __% of the time • If something occurred usually, it would occur __% of the time • If something occurred often, it would occur __% of the time Observation Checklists

45. Rating Scale Basics • Use for psychomotor and affective domains • give quality, frequency, degree, or level information • for processes or products Observation Checklists

46. Rating Scale Limitations • Subject to a number of biases: leniency, strictness, modesty, halo effect. • Subject to social expectations and, hence, to faking and self-deception. • Subject to interpretation regarding semantic problems. • Subject to criterion inadequacy. Observation Checklists

47. Precautions/Considerations • Begin with a blueprint of the behaviors to ensure valid sampling within the area. • Clearly define traits in most behavioral way possible. • Divide the rating continuum into as many points as needed for clarity: usually between three and seven. Observation Checklists

48. Precautions/Considerations • Clearly define the anchors on the continuum so there is no question as to what each rating means. • Train and motivate the raters to be as accurate and objective as possible. Observation Checklists

49. Sample Rating Scale: Degree of Confidence • 1: little or no; 2 some; 3 great confidence • inquiry teaching 1 2 3 • instructional modules 1 2 3 • simulations 1 2 3 • lecture 1 2 3 • leading a discussion 1 2 3 • questioning 1 2 3 Observation Checklists

50. Defining Steps on a Scale • Numeric Anchors: numbers, often accompanied by verbal cues • Degree of Agreement Anchors: Completely Disagree to Completely Agree; Never to Always • Adjectival Anchors: Bipolar Adjectives define endpoints with numbers in between: valuable 7 6 5 4 3 2 1 worthless passive 1 2 3 4 5 6 7 active Observation Checklists