Developing Assessment Literacy: A workshop on the relationship between instruction, assessment & evaluation

1. Developed & Presented by Edward J. Caropreso, PhD Watson School of Education University of North Carolina Wilmington caropresoe@uncw.edu 910.962.7830 Director of Academics Marine Corps Combat Service Support Schools Camp Johnson, NC January, 2011 Developing Assessment Literacy:A workshop on the relationship between instruction, assessment & evaluation

2. Developing Assessment Literacy: Learning Outcomes • 1. Recognize/value the relationship between components of professional practice (Inst; Asmt; Eval) 2. Recognize/value the relationship between professional vocabulary and practice 3. Recognize/value the use of professional vocabulary • 1. Recognize/value the relationship between goals/objectives, instruction & assessment/evaluation 2. Recognize differences between objectives models 3. Write objectives using a specific model 4. Critique objectives using a specified evaluation system • 1. Recognize/value the role of appropriate information in decision-making 2. Recognize/value the relationship between types of assessment & resulting information 3. State the difference between objective & subjective (non-objective) assessment • 1. State the difference between traditional & alternative assessment models 2. State the difference between models of alternative assessment 3. State the purpose/s for rubrics in assessment 4. Identify appropriate applications of rubrics (in specific assessment settings/contexts) • 1. Describe formative assessment strategies 2. State appropriate applications for formative assessment/s • 1. Recognize/value the need for systematic planning for assessment 2. Evaluate item types for given applications/purposes 3. Construct items to meet specific Los in given learning domains 4. Evaluate items/tests as to purpose/use of assessment information • 1. Describe validity; reliability 2. Explain the relationship between high quality assessment and validity/reliability • 1. Recognize/value the role of instructors as student study supports 2. Recognize the relationship between test item types & specific study strategies

3. Developing Assessment Literacy: Learning Outcomes • 1. Recognize/value the relationship between components of professional practice (Inst; Asmt; Eval) 2. Recognize/value the relationship between professional vocabulary and practice 3. Recognize/value the use of professional vocabulary • 1. Recognize/value the relationship between goals/objectives, instruction & assessment/evaluation 2. Recognize differences between objectives models 3. Write objectives using a specific model 4. Critique objectives using a specified evaluation system • 1. Recognize/value the role of appropriate information in decision-making 2. Recognize/value the relationship between types of assessment & resulting information 3. State the difference between objective & subjective (non-objective) assessment • 1. State the difference between traditional & alternative assessment models 2. State the difference between models of alternative assessment 3. State the purpose/s for rubrics in assessment 4. Identify appropriate applications of rubrics (in specific assessment settings/contexts)

4. Developing Assessment Literacy: Learning Outcomes • 1. Describe formative assessment strategies 2. State appropriate applications for formative assessment/s • 1. Recognize/value the need for systematic planning for assessment 2. Evaluate item types for given applications/purposes 3. Construct items to meet specific Los in given learning domains 4. Evaluate items/tests as to purpose/use of assessment information • 1. Describe validity; reliability 2. Explain the relationship between high quality assessment and validity/reliability • 1. Recognize/value the role of instructors as student study supports 2. Recognize the relationship between test item types & specific study strategies

5. Workshop Contents • Introduction & Context for Training • Knowledge-Practice Model • Professional Vocabulary & Concepts • Context for Instruction • Educational Goals & Objectives • Models of Objectives • Working with Objectives • Context for Assessment & Evaluation • Role of Information for Decision-making • Types of data related to types of decisions • Assessment Model • Types of Assessments • Objective vs Subjective, Etc. • Alternative Assessment: Performance Strategies Complementing Paper Assessment • Formative Assessment: Information about Teaching & Learning to Improve Instruction • Working with Assessment Tools/Strategies: Writing assessment plans; developing assessment items within local work context • Assessment & Test Construction: Technical Features & “Quality” Tools • Validity • Reliability • Supporting Student Learning through Assessment • KSU Study Strategies; Test Preps • Improving Performance; MIT Test Strategies

6. Workshop Contents • Introduction & Context for Training • Knowledge-Practice Model • Professional Vocabulary & Concepts • Context for Instruction • Educational Goals & Objectives • Models of Objectives • Working with Objectives • Context for Assessment & Evaluation • Role of Information for Decision-making • Types of data related to types of decisions • Assessment Model • Types of Assessments • Objective vs Subjective, Etc.

7. Workshop Contents • Alternative Assessment: Performance Strategies Complementing Paper Assessment • Formative Assessment: Information about Teaching & Learning to Improve Instruction • Working with Assessment Tools/Strategies: Writing assessment plans; developing assessment items within local work context • Assessment & Test Construction: Technical Features & “Quality” Tools • Validity • Reliability • Supporting Student Learning through Assessment • KSU Study Strategies; Test Preps • Improving Performance; MIT Test Strategies

8. Workshop Contents • Introduction & Context for Training • Knowledge-Practice Model • Professional Vocabulary & Concepts

9. Developing Assessment Literacy: Learning Outcomes • 1. Recognize/value the relationship between components of professional practice (Inst; Asmt; Eval) 2. Recognize/value the relationship between professional vocabulary and practice 3. Recognize/value the use of professional vocabulary

10. 1. Introduction & Context for Training • Knowledge-Practice Model of Assessment & Evaluation • Professional Vocabulary for Concepts

11. Knowledge-Practice Model Developing Professional Knowledge Knowledge of Learner Achievement Resulting from Instruction Knowledge of Instructional Practice: Strategies & Activites (incl. assessment) Dynamic Reciprocity

12. Knowledge-Practice Model • Developing Professional Knowledge

13. Knowledge-Practice Model(Discussion Activity) • What’s the current status of your professional knowledge? • How have you developed your knowledge of instruction, assessment and evaluation? • What are your expectations for developing your professional knowledge & practice? • What are your professional development alternatives?

14. Professional Vocabulary for ConceptsActivity: Review/id concepts you know well; compare/contrast with colleagues’/groups’ Resources & Databases • New Horizons for Learning • http://www.newhorizons.org/strategies/assess/terminology.htm • Nat’l. Center for Research on Evaluation, Standards, & Student Testing (CRESST @ UCLA) • http://www.cse.ucla.edu/products/glossary.html • Coalition of Essential Schools: Defining Assessment • http://www.essentialschools.org/cs/resources/view/ces_res/124 • System of Adult Basic Education Supports • http://www.sabes.org/assessment/glossary.htm • Beyond Confusion: An Assessment Glossary, by Andrea Leskes • http://ctl.stanford.edu/Tomprof/postings/448.html • Continuous Assessment: Practical Guide for Teachers

15. Professional Vocabulary(Activity) • Review the glossaries • Identify the terms & concepts you know • Mark the ones you know & understand • Compare your list to colleagues’/groups’ • What’s the same; what’s different? • Why are there differences? • What are the implications for professional practice given inconsistencies in the knowledge base?

16. Workshop Contents • Context for Instruction • Educational Goals & Objectives • Models of Objectives • Working with Objectives

17. Developing Assessment Literacy: Learning Outcomes • 1. Recognize/value the relationship between components of professional practice (Inst; Asmt; Eval) 2. Recognize/value the relationship between professional vocabulary and practice 3. Recognize/value the use of professional vocabulary • 1. Recognize/value the relationship between goals/objectives, instruction & assessment/evaluation 2. Recognize differences between objectives models 3. Write objectives using a specific model 4. Critique objectives using a specified evaluation system

18. 2. Context for Instruction • Educational Goals & Objectives • Models of Objectives • Working with Objectives

19. Educational Goals & Objectives • Educational Goals: • “…those human activities which contribute to the functioning of a society (including the functioning of an individual in society), and which can be acquired through learning.” (Gagne, Briggs, & Wagner, 1988, p. 39; cited in Nitko, 2001, p. 23) • Goals are stated in broad terms that give direction and purpose to planning overall educational activities.

20. Educational Goals & Objectives • Educational Goals (or global objectives) : • Broad, complex learning outcomes that require substantial time and instruction to accomplish • Typically encompassing many specific learning objectives which require learner success on most/all to reach the general goal (from Airasian, 2001)

21. Educational Goals & Objectives Educational Goals • UT Examples of Goals vs Objectives • More Examples of Goals vs Objectives • Airasian’s 3-Level Model

22. Models of Objectives • Two Common Models: Mager & Gronlund • Mager Tips • Gagne & Briggs & ABCD Models • A Comparison of Models • Greenberg’s 6 Keys to Successful Performance Objectives

23. Comparison of Formats[Seels & Glasgow (1990). Exercises in Instructional Design. Columbus, Ohio: Merrill.]

24. Working with Objectives • Understanding the Context of Learning • How do you make sense of learning? • How do you identify/describe expectations? • Putting Bloom’s Taxonomy to Work • Three Types of Learning • What Drives Statements of Learning: Active Verbs

25. Working with Objectives: Activity • Identify & State 1-3 TLOs (goals; in your area) • For each TLO (goal), using the course materials, identify & state 1-2 knowledge & 1-2 performance objectives (ELOs) in at least two models we’ve reviewed • Compare your objectives with a colleague’s • Using Greenberg’s Criteria to review your objectives, how do they hold up? • What’s similar to your colleague? • What’s different? • Keep your objectives for later; we’ll use them again!

26. Workshop Contents • Context for Assessment & Evaluation • Role of Information for Decision-making • Types of data related to types of decisions • Assessment Model • Types of Assessments • Objective vs Subjective, Etc.

27. Developing Assessment Literacy: Learning Outcomes • 1. Recognize/value the relationship between components of professional practice (Inst; Asmt; Eval) 2. Recognize/value the relationship between professional vocabulary and practice 3. Recognize/value the use of professional vocabulary • 1. Recognize/value the relationship between goals/objectives, instruction & assessment/evaluation 2. Recognize differences between objectives models 3. Write objectives using a specific model 4. Critique objectives using a specified evaluation system • 1. Recognize/value the role of appropriate information in decision-making 2. Recognize/value the relationship between types of assessment & resulting information 3. State the difference between objective & subjective (non- objective) assessment

28. 3. Context for Assessment & Evaluation • Establishing vocabulary & conceptual framework • Assessment and Evaluation: • What do they mean? • Relationship between assessment, evaluation and decision-making: • The role of time • What about instructional outcomes? • Instruction, assessment and evaluation • Developing Criteria

29. 3. Context for Assessment & Evaluation • Model of Assessment • Identifying & applying appropriate strategies to gather information for specific purposes &/or decisions • Determining appropriate assessment strategies • Appropriately applying strategies

30. 3. Context for Assessment & Evaluation: Assessment Model Purposeful Decision Making Assess Instructional Practices: Methods, Materials, Strategies & Activities Assess Learner Achievement Resulting from Instruction Dynamic & Reciprocal

31. 3. Context for Assessment & Evaluation • Types of Assessments by Purpose • Bloom’s Taxonomy as Context • Aim High • Objective Assessment • Selected vs Constructed Response • Subjective (Non-Objective) Assessment • Writing/Generating Responses • Role & Control of Subjectivity in Assessment • Alternative Assessment • Authentic Assessment • Performance Assessment

32. 3. Context for Assessment & Evaluation [http://vudat.msu.edu/assessment/] • Objective Assessment • Objective assessments (usually multiple choice, true false, matching, short answer) have correct answers. • These are good for testing recall of facts and can be automated. • Objective tests assume that there are true answers and assume that all students should learn the same things.

33. 3. Context for Assessment & Evaluation[http://vudat.msu.edu/assessment/] • Subjective (Non-Objective) Assessment • With subjective assessments, teacher's judgment determines the grade • These include essay tests. • Essay tests take longer to answer and they take longer to grade than objective questions and therefore only include a small number of questions, focusing on complex concepts. • Writing/Generating Responses • Role & Control of Subjectivity: Criteria &/or Rubrics

34. Workshop Contents • Introduction & Context for Training • Knowledge-Practice Model • Professional Vocabulary & Concepts • Context for Instruction • Educational Goals & Objectives • Models of Objectives • Working with Objectives • Context for Assessment & Evaluation • Role of Information for Decision-making • Types of data related to types of decisions • Assessment Model • Types of Assessments • Objective vs Subjective, Etc. • Alternative Assessment: Performance Strategies Complementing Paper Assessment

35. Developing Assessment Literacy: Learning Outcomes • 1. Recognize/value the relationship between components of professional practice (Inst; Asmt; Eval) 2. Recognize/value the relationship between professional vocabulary and practice 3. Recognize/value the use of professional vocabulary • 1. Recognize/value the relationship between goals/objectives, instruction & assessment/evaluation 2. Recognize differences between objectives models 3. Write objectives using a specific model 4. Critique objectives using a specified evaluation system • 1. Recognize/value the role of appropriate information in decision-making 2. Recognize/value the relationship between types of assessment & resulting information 3. State the difference between objective & subjective (non-objective) assessment • 1. State the difference between traditional & alternative assessment models 2. State the difference between models of alternative assessment 3. State the purpose/s for rubrics in assessment 4. Identify appropriate applications of rubrics (in specific assessment settings/contexts)

36. 4. Alternative Assessment: Performance Strategies Complementing Paper Assessment • Alternative • To what? • Portfolio Assessment: An example • Authentic • With respect to what? • Performance • What types?

37. 4. Alternative Assessment : Activity • Alternative • Possible examples? • Portfolio Assessment: Possible examples? • Authentic • Possible examples? • Performance • Possible examples?

38. 4. Alternative Assessment: Alternative Methods to Document Learning • 3 Formal Models of Assessment: • Holistic: • Applying all criteria as a single judgment • Modified Holistic: • Adding levels of judgment • Analytical: • Judgment by criteria; weighting factors; scaling • Valid decision-making: • Matching the “tool” and the instructional outcome • Examples of Rubrics: Rubric Primer • Information Skills • Oral Presentation • Creating Rubrics

39. Workshop Contents • Introduction & Context for Training • Knowledge-Practice Model • Professional Vocabulary & Concepts • Context for Instruction • Educational Goals & Objectives • Models of Objectives • Working with Objectives • Context for Assessment & Evaluation • Role of Information for Decision-making • Types of data related to types of decisions • Assessment Model • Types of Assessments • Objective vs Subjective, Etc. • Alternative Assessment: Performance Strategies Complementing Paper Assessment • Formative Assessment: Information about Teaching & Learning to Improve Instruction

40. Developing Assessment Literacy: Learning Outcomes • 1. Recognize/value the relationship between components of professional practice (Inst; Asmt; Eval) 2. Recognize/value the relationship between professional vocabulary and practice 3. Recognize/value the use of professional vocabulary • 1. Recognize/value the relationship between goals/objectives, instruction & assessment/evaluation 2. Recognize differences between objectives models 3. Write objectives using a specific model 4. Critique objectives using a specified evaluation system • 1. Recognize/value the role of appropriate information in decision-making 2. Recognize/value the relationship between types of assessment & resulting information 3. State the difference between objective & subjective (non-objective) assessment • 1. State the difference between traditional & alternative assessment models 2. State the difference between models of alternative assessment 3. State the purpose/s for rubrics in assessment 4. Identify appropriate applications of rubrics (in specific assessment settings/contexts) • 1. Describe formative assessment strategies 2. State appropriate applications for formative assessment/s

41. 5. Formative Assessment: Information about Teaching & Learning to Improve Instruction • Formative Assessment • Types of Formative Assessment • Objective assessments • Forms/versions of summative assessments that will be implemented for final evaluation of students • Alternative assessments • Forms/versions of summative assessments that could be implemented for final evaluation of students • Used to provide feedback on developing learning • Should not contribute to final grades • May or may not be “graded” assessments • Limited direct impact on final evaluation of student

42. 5. Formative Assessment: Information about Teaching & Learning to Improve Instruction • Observation Strategies • Using explicit formal tools • Informal observations • Questioning Strategies • Taba Tables • Self & Peer Assessment Strategies

43. 5. Formative Assessment: Information about Teaching & Learning to Improve Instruction • Alternative assessments • Used to provide feedback on developing learning • Examples: • Formative Evaluation Form • Summary Evaluation Form • Observation Record Form • Observation Guide Form • Observation Assessment Form • Open-ended Questions

44. Workshop Contents • Introduction & Context for Training • Knowledge-Practice Model • Professional Vocabulary & Concepts • Context for Instruction • Educational Goals & Objectives • Models of Objectives • Working with Objectives • Context for Assessment & Evaluation • Role of Information for Decision-making • Types of data related to types of decisions • Assessment Model • Types of Assessments • Objective vs Subjective, Etc. • Alternative Assessment: Performance Strategies Complementing Paper Assessment • Formative Assessment: Information about Teaching & Learning to Improve Instruction • Working with Assessment Tools/Strategies: Writing assessment plans; developing assessment items within local work context

45. Developing Assessment Literacy: Learning Outcomes • 1. Recognize/value the relationship between components of professional practice (Inst; Asmt; Eval) 2. Recognize/value the relationship between professional vocabulary and practice 3. Recognize/value the use of professional vocabulary • 1. Recognize/value the relationship between goals/objectives, instruction & assessment/evaluation 2. Recognize differences between objectives models 3. Write objectives using a specific model 4. Critique objectives using a specified evaluation system • 1. Recognize/value the role of appropriate information in decision-making 2. Recognize/value the relationship between types of assessment & resulting information 3. State the difference between objective & subjective (non-objective) assessment • 1. State the difference between traditional & alternative assessment models 2. State the difference between models of alternative assessment 3. State the purpose/s for rubrics in assessment 4. Identify appropriate applications of rubrics (in specific assessment settings/contexts) • 1. Describe formative assessment strategies 2. State appropriate applications for formative assessment/s • 1. Recognize/value the need for systematic planning for assessment 2. Evaluate item types for given applications/purposes 3. Construct items to meet specific Los in given learning domains 4. Evaluate items/tests as to purpose/use of assessment information

46. 6. Working with Assessment Tools & Strategies (within local work context) • Writing assessment plans • Who’s responsible for developing plans? • What’s the basis for developing plans? • Developing assessment tools & items • Who develops summative assessments? • Who implements summative assessments? • Who interprets data from summative assessments? • Formative assessments • Who develops formative assessments? • Who implements formative assessments? • Who interprets data from formative assessments?

47. 6. Working with Assessment Tools & Strategies • Developing Objective Tests & Assessments • Planning Assessments • Types of Objective Assessments • Sample Table of Specifications • Judging Assessments

48. 6. Working with Assessment Tools & Strategies • Developing Objective Tests & Assessments • Resources online @ • Writing Test Items • Testing Primer • Matching Items to Objectives

49. 6. Working with Assessment Tools & Strategies • Developing Objective Tests & Assessments • Test Construction Activity (online @ http://people.uncw.edu/caropresoe/)

50. 6. Working with Assessment Tools & Strategies • Developing Subjective (Non-objective) Tests & Assessments • See Test Primer materials (online @ http://people.uncw.edu/caropresoe/)