Dr. Kristi Roberson-Scott

1. Evaluation of Student Learning: Test Construction & Other Practical Strategies Faculty Professional DevelopmentFall 2005 Dr. Kristi Roberson-Scott

2. Guiding Principles for Evaluation • Evaluation should relate directly to instructional objectives • Each evaluation activity should be designed to promote student growth • The actual activity should be useful practice in itself • Feedback should be useable by the student • Multiple evaluation strategies should be provided to master achievement of X objective/competency • Student should clearly understand the methods of evaluation for X test or activity

3. Questions to Ask yourself in Designing a Test • What objectives will (should) I be testing? • What types of items will be included in the test? • How long will the test be in terms of time and number of items? • How much will each objective be worth in terms of weighting and number of items?

4. Tests as Diagnostic Tools • Students demonstrate learning • Instructor effectiveness – modify teaching strategies or activities • Assignment of letter grades

5. Different Types of Tests & Learning • Paper & Pencil/WebCT Testing • Limited Choice Questions (MC, T/F, Matching) • Open-Ended Questions (Short Answer, Essay) • Performance Testing • Acquisition of skills that can be demonstrated through action (e.g., music, nursing, etc.)

6. Planning a Test • First step: Outline learning objectives or major concepts to be covered by the test • Test should be representative of objectives and material covered • Major student complaint: Tests don’t fairly cover the material that was supposed to be canvassed on the test.

7. Planning a Test • Second Step: Create a test blueprint • Third Step: Create questions based on blueprint • Match the question type with the appropriate level of learning • Fourth Step: For each check on the blueprint, jot down (might use 3x5 cards) 3-4 alternative question on ideas and item types which will get at the same objective • Fifth Step: Organize questions and/or ideas by item types

8. Planning a Test • Sixth Step: Eliminate similar questions • Seventh Step: Walk away from this for a couple of days • Eighth Step: Reread all of the items – try doing this from the standpoint of a student

9. Planning a Test • Ninth Step: Organize questions logically • Tenth Step: Time yourself actually taking the test and then multiply that by about 4 depending on the level of students • Eleventh Step: Analyze the results (item analyses)

10. Translating Course Objectives/Competencies into Test Items • Syllabus • Specification table- what was taught/weight areas to be tested • Creating a Test Blueprint (see handout) • Blueprint- this is the test plan, i.e., which questions test what concept • Plotting the objectives/competencies against some hierarchy representing levels of cognitive difficulty or depth of processing

11. Thinking Skills • What level of learning corresponds to the course content • Bloom’s Taxonomy of Educational Objectives • Knowledge (see handout) • Comprehension • Application • Analysis • Synthesis • Evaluation

12. Practical Considerations • Representative sample of the course content not random– purposeful based on blueprint • Representative sample of skill or cognitive levels across content • Analyze results by level AND content area

13. Question Arrangement on a Test • Group by question type • Common instructions will save reading time • Limit the number of times students have to change frame of reference • Patterns on test must be logical • Arrange from a content standpoint • Keep similar concepts together • Group by difficulty (easy to hard)

14. Selecting the Right Type of evaluation • How do you know what type of question to use and when? • It depends on the skill you are testing. • Evaluation should always match as closely as possible the actual activity you’re teaching. • Examples: Teaching Speech, should evaluate an oral speech • If testing ability to write in Spanish, better give an essay. • Testing reading –MC, TF • Wouldn’t use MC to test creative writing

15. Question Types verses Cognitive Levels of Learning

16. Constructing the Test • Types of Test Questions: • Multiple-Choice Items • True-False Items • Matching Items • Fill-In, Completion or Short-Answer Items • Essay Questions

17. Multiple Choice Items • Advantages: • Extremely versatile-can measure the higher level mental processes (application, analysis, synthesis and evaluation) • A compromise between a short answer/essay and T/F item • Can cover a wide range of content can be sampled by one test • Disadvantages • Difficult to construct plausible alternative responses

18. Types of Multiple Choice Items • Four Basic Types • Question Type • Incomplete Statement Type • Right Answer Type • Best Answer Type • Which Type is Best? • Question Type vs. Incomplete Statement • Right Answer vs. Best Answer Type

19. Multiple Choice Items • Writing the stem first: • Be sure the stem asks a clear question • Stems phrased as questions are usually easier to write • Stems should not contain a lot of irrelevant info. • Appropriate reading level/terms • Be sure the stem is grammatically correct • Avoid negatively stated stems

20. Multiple Choice Items • Writing the correct response • Use same terms/reading level • Avoid too many qualifiers • Assign a random position in the answer sequence • Read the stem and correct response together • Generate the distractors/alternative responses

21. Multiple Choice Items • Other Tips for Constructing MC Items: • Items should have 3-4 alternatives. • Stem should present a single, clearly formulated problem • Simple, understandable, exclude extraneous words from both stem and alternatives • Include in the stem any word that are repeated in each response • Avoid all of the above (can answer based on partial information) • Avoid none of the above

22. Multiple Choice Items • Alternative responses/distractors should be plausible and as homogeneous as possible • Response alternatives should not overlap • Two synonymous terms (arithmetic average/mean) • Avoid double negatives • None of the following are part of the brain except which one? • Emphasize negative wording • Each item should be independent of other items in the test • Information in the stem of one item should NOT help answer another item.

23. True-False Test Items • Best suited for testing 3 kinds of info.: • Knowledge level learning • Understanding of misconceptions • When there are two logical responses • Advantages: • Sample a large amount of learning per unit of student testing time • Disadvantages: • Tends to be very easy • 50-50 chance of guessing • Tends to be low in reliability

24. Tips for Constructing True/False Items • Tips for constructing True-False Items • Avoid double negatives • Avoid long or complex sentences • Specific determiners (always, never, only, etc.) should be used with caution • Include only one central idea in each statement • Avoid emphasizing the trivial • Exact quantitative (two, three, four) language is better than qualitative (some, few, many) • Avoid a pattern of answers

25. Objective Test Item Analyses • Evaluating the Effectiveness of Items.. • Why? • Scientific way to improve the quality of tests and test items • Identify poorly written items which mislead students • Identify areas (competencies) of difficulty • Item analyses provided info. on: • Item difficulty • Item discrimination • Effectiveness of alternatives in MC Tests

26. Short-Answer Items • Two Types: (Question and Incomplete Statement) • Advantages: • Easy to construct • Excellent format for measuring who, what, when, and where info. • Guessing in minimized • Student must know the material- rather than simply recognize the answer • Disadvantages: • Grading can be time consuming • More than one answer can be correct

27. Short Answer Items • Tips for Constructing Short Answer Items • Better to supply the term and require a definition • For numerical answers, indicate the degree of precision expected and the units in which they are to be expressed. • Use direct questions rather than incomplete statements • Try to phrase items so that there is only one possible correct response • When incomplete statements are used, do not use more than one blank within an item.

28. Essay Questions Types of Essay Questions • Extended Response Question • Great deal of latitude on how to respond to a question. • Example: Discuss essay and multiple-choice type tests. • Restricted Response Question • More specific, easier to score, improved reliability and validity • Example: Compare and contrast the relative advantages of disadvantages of essay and multiple choice tests with respect to: reliability, validity, objectivity, & usability.

29. Essay Items • Advantages: • Measures higher learning levels (synthesis, evaluation) and is easier to construct than an objective test item • Students are less likely to answer an essay question by guessing • Require superior study methods • Offer students an opportunity to demonstrate their abilities to: • Organize knowledge • Express opinions • Foster creativity

30. Essay Items • May limit the sampling of material covered • Tends to reduce validity of the test • Disadvantages • Subjective unreliable nature of scoring • “halo effect” – good or bad student’s previous level of performance • Written expression • Handwriting legibility • Grammatical and spelling errors • Time Consuming

31. Essay Questions • Give students a clear idea of the scope & direction intended for the answer • Might help to start the question with the description of the required behavior (e.g., compare, analyze) • Appropriate language level for students • Construct questions that require students to demonstrate a command of background info, but do not simply repeat that info. • If question calls for an opinion, be sure that the emphasis is not on the opinion but on the way its presented or argued. • Use a larger number of shorter, more specific questions rather than one or two longer questions so that more information can be assessed.

32. Essay Questions • You might • Give students a pair of sample answers to a question of the type you will give on the test. • Sketch out a rubric (grading scheme) for each question before reading the papers OR randomly select a few to read and make up the grading scheme based on those answers • Give students a writing rubric • Detach identifying information and use code numbers instead to avoid letting personality factors influence you. • After grading all the papers on one item, reread the first few to make sure you maintained consistent standards • Be clear to student the extend to which factors other than content (e.g., grammar, handwriting, etc.) will influence the grade.

33. Essay Questions • Tips for constructing Essay Questions • Provide reasonable time limits for each question • “thinking and writing time” • Avoid permitting students a choice of questions • Will not necessarily get a representative sample of student achievement. Only be requiring all students to answer all questions can their achievement be compared • A definite task should be put forth to the student • Critical words: compare, contrast, analyze, evaluate, etc.

34. Scoring Essay Items • Write an outline of the key points (use outline to design a rubric) • Determine how many points are to be assigned to the question as a whole and to the various parts within it. • If possible, score the test without knowledge of the student’s name • Face Sheet • Score all of the answers to one question before proceeding to the next question • Consistent standard

35. Scoring Essay Exams • If possible, score each set of answers within the same time frame • Handwriting, spelling & Neatness • Two separate grades? • Mastery of material • Other

36. Alternative Methods of Assessment • Research/Term Papers • Research Reviews • Reports • Case Studies • Portfolios • Projects • Performances • Peer evaluation • Mastery • Simulations

37. Cheating • Preventing Cheating • Reduce the pressure (multiple evaluations) • Make reasonable demands (length/content of exam) • Use alternative seating • Use alternative forms • Be cautious with extra copies

38. Using Assessment & Evaluation to Improve Student Learning Outcomes • Providing feedback to student • Closing the assessment & evaluation loop • Maximizing student learning

39. Questions?