EDP 303 Presentations

1. EDP 303 Presentations Lawrence W. Sherman, Ph. D PowerPoint Slides for EDP 303: Assessment and Evaluation in Educational Settings. Linn and Miller (2004) text

2. Chapter 1:Educational Testing and Assessment: Context, Issues, and Trends • Accountability demands; • State, national and international assessment programs • National content and performance standards; • Global competition • Fairness of uses and interpretations

3. Chapter 1:Educational Testing and Assessment: Context, Issues, and Trends Accountability demands including state, national and international assessment programs, national content and performance standards, as well as global competition, all contribute to increased demands for testing assessment. These factors have both stimulated and reflected new trends in educational measurement. The increased reliance on testing and assessment as educational reform tools has also raised issues concerning the farness of their uses and their interpretations.

4. Maximum Performance Function: Determines what individuals can do when performing at their best. Example: Aptitude and Achievement Tests Typical Performance Function: Determine what individuals will do under natural conditions Example: Attitude, interest and personality inventories; observational techniques; peer appraisal. Nature of AssessmentChapter 2 issues

5. Fixed Choice Test Function: Efficient measurement of knowledge and skills, indirect indicator. Example: Standardized multiple-choice tests Complex-performance assessment Function: Measurement of performance in contexts and on problems valued in their own right Example: Hands-on laboratory experiment, projects, essays, oral presentation Form of Assessment

6. Tests Used in Classroom Instruction • Placement • Formative • Diagnostic • Summative

7. Function Determines prerequisite skills, degree of mastery of course goals, and or best mode of learning Example: Readiness tests Aptitude tests Pretests on course objectives Self-report inventories Observational techniques Placement

8. Function: Determines learning progress Provides feedback to reinforce learning Corrects learning errors Examples: Teacher-made tests Custom-made tests from textbook publishers, Observational techniques Formative Assessment

9. Function: Determines causes (intellectual, physical, emotional, environmental) of persistent learning difficulties. Example: Published diagnostic tests, Teacher-made diagnostic tests, Observational techniques Diagnostic Assessment

10. Function: Determines end-of-course achievement for assigning grades Certifying mastery of objectives Examples: Teacher-made survey tests Performance rating scales Product scales Summative Evaluation

11. Criterion Referenced Function: Describes student performance according to a specified domain of clearly defined learning tasks (e.g., adds single-digit whole numbers) Example: Teacher-made tests, custom-made tests from test publishers, observational techniques Norm Referenced Function: Describes student performance according to relative position in some known group (e.g., ranks 10th out of 30; top 10 percent) Examples: Standardized aptitude and achievement tests Teacher-made survey tests, Interest inventories, Adjustment inventories Methods of interpreting results:

12. Chapter 3:Instructional Goals and Objectives: Foundations for Assessment What types of learning outcomes do you expect from your teaching? • Knowledge? • Understanding? • Applications? • Thinking skills? • Performance skills? • Attitudes? Clearly defining desired learning outcomes is the first step in good teaching. It is also essential to the assessment of student learning. Sound assessment requires relating the assessment procedures as directly as possible to intended learning outcomes.

13. Chapter 3: Instructional Goals Types of Learning Outcomes to Consider Bloom’s Taxonomy of Educational Objectives • Cognitive Domain • Knowledge and intellectual skills/abilities • Affective Domain • Attitudes, interests, appreciation • Psychomotor Domain • Perceptual and motor skills

14. Chapter 3:Instructional Goals Other sources for lists of objectives: • Professional Association Standards • MCREL • State Content Standards • OHIO

15. Chapter 3:Instructional Goals Some Criteria for Selecting Appropriate Objectives: • Do the objectives include all important outcomes of the course? • Are the objectives in harmony with the content standards of the state or district and with general goals of the school? • Are the objectives in harmony with sound principles of learning? • Are the objectives realistic in terms of the abilities of the students and the time and facilities available?

16. Chapter 4: Validity • When constructing or selecting assessments, the most important questions are: • (1) To what extent will the interpretation of the scores be appropriate, meaningful, and useful for the intended application of the results? and • (2) What are the consequences of the particular uses and interpretations that are made of the results? • (3) A valid test Must be Reliable!

17. Chapter 4: Validity Issues • Nature of Validity • Major Considerations in Assessment Validation • Content Considerations • Construct Considerations • Assessment-Criterion Relationships • Consideration of Consequences • Factors Influencing Validity

18. Nature of Validity • Appropriateness of the interpretation of the results • It’s a matter of degree • Specific to some particular use or interpretation • Is a Unitary concept • Involves an overall evaluative judgment

19. Major Considerations in Validation • Content • How it represents the domain of tasks to be measured • Construct • Interpretation as a meaningful measure of some characteristic or quality • Assessment-Criterion Relationship • Prediction of future performance (criterion) • Consequences • How well rsults accompolish intended purposes and avoids unintended effects

20. Chapter 5: Reliability • Next to validity, reliability is the most important characteristic of assessment results. Reliability • (1) provides the consistency that makes validity possible, and • An unreliable test cannot be Valid! • (2) reliability indicates the degree to which various kinds of generalizations are justifiable. The practicality of the evaluation procedure is, of course, also of concern to the busy classroom teacher.

21. Chapter 5: ReliabilityIssues • Nature of Reliability • Determining Reliability by Correlation Methods • Standard Error of Measurement • Factors Influencing Reliability Measures • Reliability of Assessments Evaluated in Terms of a Fixed Performance Standard • Usability

22. Chapter 6:Planning tests: Timing • Preparation (Planning) • Administration • Grading • Post-Test Analysis!

23. Chapter 6:Planning tests • Objective Tests • A. Supply Type • Short Answer • Completion • B. Selection Type • True-False or Alternative-Response • Matching • Multiple Choice • Performance Assessment • Extended Response • Restricted Response

24. Table of Specifications: Similar to tables 6.2-6.4 in Chapter 6

25. Chapter 7: Simple Forms • Short-Answer • True-false • Matching

26. Short-Answer issuesChapter 7, page 178 • Is this the most appropriate type of item for the intended learning outcomes? • Can the items be answered with a number, symbol, word or brief phrase? • Has text book language been avoided? • Have the items been stated so that only one response is correct? • Are the answer blanks equal in length? • Are the answer blanks at the end of the items? • Are items free of clues (such as a or an)? • Has the degree of precision been indicated for numerical answers? • Have the units been indicated when numerical answers are expressed in units? • Have the items been phrased so as to minimize spelling errors? • If revised, are the items still relevant to the intended learning outcomes? • Have the items been set aside for a time before reviewing?

27. True-False ItemsChapter 7, p. 185 • Is this the most appropriate type of item to use? • Can each statement be clearly judged t or f? • Have specific determiners been avoided? (eg., usually, always, etc.) • Have trivial statements been avoided? • Have negative statements (especially double negatives been avoided? • Have the items been stated in simple, clear language? • Are opinion statements attributed to some source? • Are the t and f items approximately equal in length? • Is there an approximately equal number of true and false items? • Has a detectable pattern of answers been avoided? (eg., t,F,T,F,T,F,etc.) • If revised, are the items still relevant to the intended learning outcome? • Have the items been set aside for a time before reviewing them?

28. Matching ItemsChapter 7, p. 190 • Is this the most appropriate type of item to use? • Is the material in the two lists homogeneous? • Is the list of responses longer or shorter than the list of premises? • Are the responses brief and on the right-hand side? • Have the responses been placed in alphabetical or numerical order? • Do the directions indicate the basis for matching? • Do the directions indicate that each response may be used more than once? • Is all of each matching item on the same page? • If revised, are the items still relevant to the intended learning outcomes? • Have the items been set aside for a time before reviewing them?

29. Chapter 8: Multiple Choice ItemsChapter 8, p. 214 • Is this the most appropriate type of item to use? • Does each item stem present a meaningful problem? • Are the item stems free of irrelevant material? • Are the item stems stated in positive terms (if possible)? • If used, has negative wording been given special emphasis (e.g., capitalized, underlined, etc.) • Are the alternatives grammatically consistent with the item stem? • Are the alternative answers brief and free of unnecessary words? • Are the alternatives similar in length and form? • Is there only one correct or clearly best answer????????? • Are the distracters plausible to low achievers? • Are the items free of verbal clues to the answer? • Are the verbal (or numerical) alternatives in alphabetical (or numerical) order? • Have none of the above and all of the above been avoided (or used sparingly and appropriately? • If revised, are the items still relevant to the intended learning outcomes? • Have the items been set aside for a time before reviewing them?

30. Chapter 10: Measuring Complex Achievement: Essay Questions. Some important learning outcomes may best be measured by the use of open-ended essay questions or other types of “Performance” assessments. Essay questions provide freedom of response that is needed to adequately assess students’ ability to formulate problems; organize, integrate, and evaluate ideas and information; and apply knowledge and skills.

31. Essay Questions Check List • Is this the most appropriate type of task to use? • Are the questions designed to measure higher-level learning outcomes? • Are the questions relevant to the intended learning outcomes? • Does each question clearly indicate the response expected? • Are the students told the bases on which their answers will be evaluated? • Have you conceptualized a “rubric” upon which the response will be scored? • Are generous time limits provided for responding to the questions? • Are students told the time limits and/or point values for each question? • Are all students required to respond to the same question? • If revised, are the questions still relevant to the intended learning outcomes? • Have the questions been set aside for a time before reviewing them?

32. CHAPTER 11: PERFORMANCE-BASED ASSESSMENTS Essay tests are the most common example of a performance-based assessment, but there are many others, including artistic productions, experiments in science, oral presentations, and the use of mathematics to solve real-world problems. The emphasis is on doing, not merely knowing: on PROCESS as well as PRODUCT.

33. Suggestions for constructing performance tasks • Focus on learning outcomes that require complex cognitive skills and student performances. • Select or develop tasks that represent both the content and the skills that are central to important learning outcomes. • Minimize the dependence of task performance on skills that are irrelevant to the intended purpose of the assessment task. • Provide the necessary scaffolding for students to be able to understand the task and what is expected. • Construct the task directions so that the student’s task is clearly indicated. • Clearly communicate performance expectations in terms of the scoring rubrics by which the performances will be judged.

34. Chapter 12: PortfoliosKey Steps in Defining, Implementing and Using Portfolios • Specify purpose • Provide guidelines for selecting portfolio entries • Define student role in selection and self-evaluation. • Specify evaluation criteria • Use portfolios in instruction and communication.

35. Chapter 14: Assembling, Administering, and Appraising Classroom Tests and Assessments Care in preparing an assessment plan and constructing relevant test items and assessment tasks should be followed by similar care in reviewing and editing the items and tasks, preparing clear directions, and administering and appraising the results. Classroom assessments `lso can be improved by using simple methods to analyze student responses, and building a file of effective items and tasks.

36. Chapter 14: Assembling, Administering, • Assembling the Classroom Test • Administering and Scoring Classroom Tests and Assessments • Appraising Classroom Tests and Assessments • Building a File of Effective Items and Tasks

37. STUDENT LEARNING PLANNING DEVELOPMENT APPLICATION Flow Chart of Testing Process

38. PLANNING 1. CLARIFY INSTRUCTIONAL OBJECTIVES: THE CONTENT OR DOMAIN OBJECTIVES. 2. SPECIFY WHAT WILL BE TESTED (BLOOM’S TAXONOMY): KNOWLEDGE COMPREHENSION APPLICATION oAPPLICATION oSYNTHESIS oEVALUATION oANALYSIS 3. DEVELOP A TEST BLUE PRINT:

39. DEVELOPMENT OF THE TEST 4. SELECT THE TEST ITEM FORMATS: TRUE/FALSE MULTIPLE CHOICE MATCHING COMPLETION ESSAY CONTEXT-DEPENDENT ITEMS 5. PREPARE/COMPOSE/WRITE OR SELECT THE ITEMS FROM AN ARCHIVE FOR THE TEST GO TO APPLICATION

40. APPLICATION 8. SCORE THE TEST: MASTERY SCORING? NORMATIVE SCORING? MACHINE SCORE? HAND SCORE? DISCUSS RESULTS WITH STUDENTS 6. ASSEMBLE THE TEST: CONSIDER REASONABLE TIMEING BOUNDRIES COLLECT TEST ITEMS REVIEW TEST ITEMS FORMAT TEST PREPARE DIRECTIONS 7. ADMINISTER THE TEST: PROVIDE ENOUGH TIME HAVE ALL MATERIALS PENCILS PAPER SCANNER SHEETS ETC. RETURN TO STEP 5 AND STORE REVISIONS 9. ANALYZE AND REVISE ITEMS: ITEM DIFFICULTY DISCRIMINATION INDEX REVISE ITEMS AND STORE IN ARCHIVE FOR FUTURE USE

41. Chapter 15: Grading and Reporting Grading and reporting student progress is one of the more frustrating Grading and reporting student progress is one of the more frustrating aspects of teaching—there are so many factors to consider, and so many decisions to be made. This chapter will remove some of the complexity by describing the various types of grading and reporting systems and providing guidelines for their effective use.

42. Chapter 15: Grading and Reporting * Functions of Grading and Reporting Systems • Types of Grading and Reporting Systems • Multiple Grading and Reporting Systems • Assigning Letter Grades • Record-Keeping and Grading Software • Conducting Parent-Teacher Conferences • Reporting Standardized Test Results to Parents