Don’t Guess . . . Know When (and How) to Assess

1. Don’t Guess . . . Know When (and How) to Assess William F. McComas, Ph.D. Director of Science Education Programs Fellow, Center for Excellence in Teaching Rossier School of Education University of Southern California Los Angeles, CA 90089-0031 mccomas@usc.edu

2. What is Assessment? • Assessment comes from the Latin meaning to “sit with” a learner and verify that the responses really mean what they seem to mean • In most assessment plans such assurance is lacking • Much assessment is suspect in terms of validity and reliability

3. What Can We Assess? • Knowledge • Attitudes • Skills

4. Assessment Plans Must: • Be free of bias • Reflect what is (or should have been) taught • Provide information to enhance instruction (teaching or curriculum) • Reveal misconceptions Adapted from Champagne, Lovitts and Calinger (1990). Assessment in the Service of Instruction. Washington, AAAS.

5. Assessment should: • Be a learning experience for both students and teachers • Involve the evaluation of reasonable standards or benchmarks known in advance to the learners • Be based on a well-designed table of specifications • The table of specifications lists the learning goals associated with the number, kind, and nature of the individual assessment items

6. Timing is Everything • Diagnostic Evaluation (pretest) • Used for gauging prior conceptions on which to base instructional strategies • Formative Evaluation (in process) • Useful for midcourse corrections • Summative Evaluation (posttest) • A way of seeing how well you did as a teacher

7. Sorry, Wrong Number! • Norm-referenced; measures of performance against that of others • Scores are typically reported in percentiles • “Jane was in the 95th percentile on the SAT” • Criterion-referenced; absolute measures • Scores are typically reported in percents • “Bob got 82% of the questions correct” • Problems occur when instructors confuse the two types of assessment goals

8. Major Assessment Issues • High Stakes Test; a single assessment the result of which (positive or negative) will have a large impact on the future of the examinee • Certification, licensure, entrance exams, etc. • Low Stakes Test; a single assessment the result of which does not have a large impact on the future of the examinee • Most teacher-made tests

9. Major Issues in Assessment Design • Reliability; a measure of the consistency of the results of a given instrument (mathematically determined) • Validity; a measure of the degree to which the instrument measures what it is designed to measure (qualitatively determined)

10. What is Validity? Validity refers to the accuracy of a measure A measurement is valid when it measures what it is designed to measure and performs the functions that it purports to perform. Does an indicator accurately measure the variable that it is intended to measure?

11. What is Reliability? Reliability refers to the dependability and consistency of the results provided by an instrument or technique. A measurement is reliable when it produces the same result repeatedly with the same examinees or those from the same sample. Does an indicator repeatedly report the same measure that it is intended to measure?

12. Forms of Assessment Traditional: typical forms of style and substance Enhanced: new forms of style, substance and goals

13. Traditional Assessment • Goals; assignment of grades, student progress reporting and fault finding • Target; learners • Timing; summative • Methods; objective exams at the recall level of the knowledge domain

14. Enhanced Assessment • Goals; assignment of grades, student and program assessment • Target; learners, instructors and the curriculum • Timing; formative and summative • Methods; objective exams and expanded methods targeting KSA at all levels

15. Enhanced Assessment Includes • More data points (not just summative assessment) • More domains (not just knowledge) but attitudes, skills, creativity, etc., • Higher levels of all domains (such as synthesis rather than memorization) • More techniques such as portfolio, authentic assessment and empirically derived exams

16. Why Expand Assessment? • The nature and focus of assessment tell students (and teachers) what is important • Enhanced assessment encourages students to achieve in nontraditional realms (ex; creativity) and in nontraditional ways • Expanded assessment results provides the data necessary to make more thoughtful decisions about the curriculum, instruction, and student progress

17. Multi-Domain Assessment • Knowing and Understanding • Exploring and Discovering (process skills -- particularly in math and science) • Using and Applying Knowledge • Imagining and Creating • Feeling and Valuing (attitudes) • Understanding the nature of the discipline (such as science)

18. Domain: Knowledge & Understanding • Facts • Information • Concepts • Laws (principles) • Explanations and theories • Processes inherent to the discipline

19. Discipline-Based Skills and Processes Primary Science Process Skills • Observing • Using Space/Time Relationships • Classifying (Grouping and Ordering) • Using Numbers (Quantifying) • Measuring • Communicating • Inferring • Predicting

20. Discipline-Based Skills and Processes Integrated Science Process Skills • Controlling and Identifying Variables • Interpreting Data • Formulating Hypotheses • Defining Operationally • Experimenting

21. Domain: Using & Applying • Seeing learned concepts in everyday life • Applying learned concepts to everyday life • Evaluating reports of scientific developments • Making personal decisions based on legitimate knowledge • Becoming involved in science-related pursuits • Taking actions based on what has been learned

22. Domain: Imagining & Creating • Visualizing by producing mental images • Combining objects and ideas in unusual or useful new ways • Producing alternate uses for objects or ideas • Pretending • Converging, Diverging and Synthesizing

23. Domain: AttitudesValues and Feelings • Developing positive attitudes toward science in general and school science • Exploring and discussing both sides of an issue • Expressing personal feelings in a constructive fashion • Making decisions based on values

24. Domain: Understanding the Nature of the Discipline • Appraising the strengths and limitations of the discipline and its methods • Evaluating and applying appropriate knowledge-production modes • Using criteria to place boundaries on the discipline (what is in and what is out) • Understanding the social aspects of work in the field

25. Bloom’s Taxonomy of the Cognitive Domain • Knowledge (Recall of data) • Comprehension (Understanding) • Application (Using data) • Analysis (Separating ideas into their parts) • Synthesis (Building something out of parts) • Evaluation (Making judgments)

26. Technique:Authentic Assessment • These are modes of assessment in which examinees are asked to perform in ways highly related to “real” situations (or situations in which the learning originally occurred) • “Real” situations are those that exist in study, work, or life itself • The best authentic assessments are both faithful (life-like) and comprehensive (wide-ranging)

27. Authentic Assessment Involves: • Worthy problems or questions of importance • Faithful representations of real situations • Options, constraints, and access to resources that are legitimate and not arbitrary • Problems that require a combination of knowledge, judgment and creativity • Judgment standards that are clearly stated in advance Adapted from Wiggins (Nov / 93) Phi Delta Kappan

28. Technique: Empirically-Derived Exams • Determine the standards or bench-marks for achievement • Create an open-ended assessment tool • Collect data and collapse similar responses into typical response items • Now, use the typical responses on a multiple choice type examination

29. Defining science is difficult because science is complex and does many things. But, MAINLY science is: • the study of fields like biology, physics, etc. • a body of knowledge (laws and theories) which explain the world around us • exploring the unknown and making discoveries • carrying out experiments to solve problems • inventing or designing things like computers • an organization of people (scientists) who have ideas for discovering new knowledge • None of these choices fits my basic viewpoint

30. Technique: Performance Assessment • Practical examinations, • Performances, • Exhibits • The task must be legitimate and contextualized, not artificial and remote • in physical science - students measure the temperature of a liquid vs. • in ecology - students determine the chemicals in a sludge sample

31. Technique: Portfolio Assessment “. . . a systematic and organized collection of evidence used by the teacher and student to monitor growth of the student’s knowledge, skills, and attitudes in a specific subject area.” Vavrus, L. (August, 1990). Put portfolios to the test. Instructor. pp. 48-53.

32. What Can Portfolios Contain? • Artifacts; tests, lab reports, photographs, meaningful journal entries • Reproductions; documents about typical events not usually captured (videos of presentations, photos of projects, etc.) • Attestations; something produced by someone else (i.e. notes from the teacher) • Productions; documents prepared for the portfolio (goal statement or reflection)

33. The Portfolio Process • Collect materials for the portfolio • Selectmaterials to be in the portfolio • Reflecton why those materials were selected • Projectmake a presentation based on the portfolio contents • Respectthe contents and presentation • Show the Effectof the portfolio

34. Issues in Enhanced Assessment • How do we evaluate the results of the new schemes? • How do we report the results? • What do we do with the results in terms of advisement and promotion? • What are the solutions to questions of time and expense?

35. Typical Small Toolbox of Assessment Methods Midterms and Finals Multiple Choice Items True False Items Summative Assessment Philosophy

36. Larger Toolbox of Assessment Methods Summative Formative • Graded • End of a Unit • Feedback to students • Not graded • Perhaps on-line • Feedback to all concerned • Immediate feedback • Allow for real time adjustments True/False Multiple Choice Term Papers Labs / Practical exams Minute Paper Empty Outlines Approximate Analogies Etc.

37. Don’t Guess . . . Know you Know How to Assess William F. McComas, Ph.D. Director of Science Education Programs Fellow, Center for Excellence in Teaching Rossier School of Education University of Southern California Los Angeles, CA 90089-0031 mccomas@usc.edu