Performance Assessment OSI Workshop June 25 – 27, 2003 Yerevan, Armenia

1. Performance AssessmentOSI WorkshopJune 25 – 27, 2003Yerevan, Armenia Ara Tekian, PhD, MHPEUniversity of Illinois at Chicago

2. Presentation Outline • Characteristics, types, strengths, and limitations • Five factors to consider when making performance assessment • Checklists and rating scales • Portfolios • Video presentation & exercise (workshop)

3. Performance Assessment (PA) • PA can be distinguished from the traditional paper-and-pencil test by a number of characteristics: • Greater realism of tasks • Greater complexity of tasks • Greater time needed for assessment • Greater use of judgment in scoring Linn & Gronlund, 1995

4. Performance tasks appear in many forms: • Solving realistic problems • Oral or psychomotor skills without a product • Writing or psychomotor skills with a product (Various types of performance may be restricted or extended)

5. Restricted Performance • Highly structured and limited in scope • Examples: • Write a one-page report • Give a one minute speech • Construct a graph form a given set of data • Demonstrate how to sep up lab equipment

6. Extended Performance • Less structured and broad in scope • Examples: • Design and conduct an experiment on a selected topic, present and defend the findings • Take the history of a patient, perform a physical examination, and diagnose and write a management plan.

7. Performance Assessment - Strengths • Can evaluate complex learning outcomes and skills • Provides a more natural, direct, and complete evaluation of some types of reasoning, oral, and physical skills • Provides greater motivation for students by clarifying goals and making learning more meaningful • Encourages the application of learning to “real life” situations

8. Performance Assessment - Limitations • Requires considerable time and effort to use • Judgment and scoring performance is subjective, burdensome, and typically has low reliability • Evaluation must frequently be done individually, rather than in groups

9. Need for Perf. Assessment • The shift from norm-referenced measurement to criterion-referenced measurement • The need for focusing on more complex learning outcomes (reasoning and thinking skills), on using more comprehensive student projects based on “real life” problems, and on engaging students in the activities and in the construction of meaning from them

10. Main factors to consider when making PA • 1. Specifying the performance outcome • 2. Selecting the focus of the assessment (procedure, product, or both) • 3. Selecting an appropriate degree of realism • 4. Selecting the performance situation • 5. Selecting the method of observing, recording, and scoring

11. 1. Specifying the performance outcome • E.g. A research project might include intended learning outcomes as follows: • Selects an appropriate research task • Designs and conducts an experiment • States valid conclusions • Writes a critique of the procedure of findings

12. Typical Action Verbs for restricted performance outcomes • Identify, locate, select, describe • Construct, design, draw, prepare • Demonstrate, measure, perform, set up

13. 2. Selecting the focus of the assessment • Performance assessment can focus on the procedure, the product, or some combination of the two. The nature of the performance frequently dictates where the emphasis should be placed.

14. Assessing the Procedure • Focus the PA on the procedure when: • There is no product, or product evaluation is infeasible (e.g., unavailable or too costly) • The procedure is orderly and directly observable • Correct procedure is crucial to later success • Analysis of procedural steps can aid in improving a product

15. Assessing the Product • PA should be focused on the product when: • Different procedures can result in an equally good product • The procedure is not available for observation • The procedural steps have been mastered • The product has qualities that can be clearly identified and judged

16. 3. Selecting an appropriate degree of realism • Although we cannot expect the duplicate the natural situation in which the learning will later be used, we can strive for performance assessments that approximate “real world” conditions.

17. 4. Selecting the Performance Situation • PA can be classified by the type of situation or setting used: • Paper-and-pencil performance • Identification test • Structured performance test • Simulated performance • Work sample • Extended research project

18. 5. Selecting the Method of Observing, Recording, and Scoring • Whether judging procedures, products, or some combination of the two, some type of guided observation, method of recording and scoring the results is needed. • Systematic observation and anecdotal records • Checklists • Rating scales

19. Systematic observation and anecdotal records • Observations are frequently guided by checklists or rating scales. • An anecdotal record is a brief description of some significant event. It typically includes the observed behavior, the setting in which it occurred, and a separate interpretation of the event.

20. Anecdotal Records • Are likely to be most useful when: • They focus on meaningful incidents • They are recorded soon after the incident • They contain enough information to be understandable later • The observed incident and its interpretation are kept separate

21. Checklists (Advantages) • List of performance for which raters indicate presence or absence of what they have observed • Two purposes: • Descriptors direct the attention of the rater • Boxes provide a means of recording performance judgment

22. Checklists - Focus • Focus on the procedure when there is NO product • Focus on product when: • You have a choice • The procedure is not available for observation

23. Checklists - Limitations • Checklists are for all-or-none decisions • They create a conflict for respondents when the item contains more than one performance attribute • Can force a judgment when there is no basis for the judgment

24. Rating Scales (Advantages) • Contain a list of attributes with a range of responses: e.g. VG, G, ..Poor; SA, A, D, .. • Rating scales direct raters attention to certain performance dimensions & provide a way for recording judgment • Covers areas that are not covered well by other methods (e.g. MCQs) • Flexibility of use • Low cost • Unobtrusiveness

25. Rating Scales - Limitations • Often filled out retrospectively • Lack of agreement. Get many raters for one student. • Associated with certain types of errors • Leniency • Range restriction • Halo effect

26. My Advice • Use checklists rather than rating scales when you have a choice • Use checklists and rating scales immediately after observations • Take care in describing performance attributes • Use tailored descriptive phrases to anchor the points on scale • Include a category like “Not able to rate”

27. More Advice • Provide room for comments to encourage constructive feedback • Don’t ask raters to provide a grade • Use 5 – 7 rating points on the scale • Get multiple ratings of the same student (7 or more) • Use a scoring guide or product scale when rating holistically • Supplement Checklists and rating scales

28. Portfolios • A useful method of collecting, organizing, and evaluating samples of students’ work. • Advantages: • Learning progress over time • Comparing work to past work • Self-assessment skills • Clear communication of learning progress

29. Effective Use of Portfolios • Deciding what to include • Deciding on criteria and standards • Collecting the work samples • Maintaining and using the portfolios • Getting started

30. In conclusion • PAs can provide useful information concerning student achievement but they are subject to all the errors of observation and judgment, such as personal bias, generosity error, and halo effect. • If PAs are to provide valid information, special care must be taken to improve the objectivity, reliability, and meaningfulness of the results.