An Overview of Skills Diagnosis with an application to a Standardized Test

1. An Overview of Skills Diagnosis with an application to a Standardized Test

2. An Overview of Skills Diagnosis • The objective of “Skills diagnosis” is to use the results of an assessment instrument to classify students in terms of how well they have mastered a set of skills or knowledge domains in regard to a specific course of study or content area.

3. Skills Diagnosis Terminology • By “Skills”, we mean a set of unobservable discrete (categorical) attributes for each individual. Examples: a specific set of knowledge domains (geography: capitals, products, ethnic groups, etc.), or a specific set of skills (reading: understanding word meaning, locating information, understanding specific information, connecting information)

4. Skills Diagnosis Terminology • By “Mastery”, we mean the level of achievement on a skill or knowledge domain in terms of ordered categories, such as “below, meets, and exceeds” standards, or “below average, average, and above average”. • By “Diagnosis”, we mean using an assessment instrument to produce a skills profile: lists each skill or knowledge domain and the student’s level of mastery on each.

5. Implementation Framework • Define purpose of the assessment • Define the skills • Develop and Analyze the items • Select an appropriate model • Fit model to data & evaluate results • Develop score reports

6. Define Assessment Purpose • Discrete classification vs. Continuous scaling • Impacts test design and item development • Formative vs. Summative vs. Benchmark • Kinds of feedback from the assessment (skill granularity; relationship to standards) • Timing of feedback • Users and uses of the feedback

7. Define the Skills • Purpose of the assessment • Number of skills • Level of granularity • Relationship to standards • Number of categories of “mastery” • Skill correlations and ordered relationships • Communication w/ teachers, students, & parents

8. Item Development & Analysis • Purpose of the assessment: Purely diagnostic? • Diagnostically developed items? • Item analysis • No. of skills per item • Skill interaction: Compensatory vs. Conjunctive • Skill difficulty vs. Item difficulty • Q matrix coding

9. Q Matrix • Relates items to attributes • User-specified Q matrix is merely an initial estimate

10. Conjunctive vs. Compensatory • Conjunctive: Successful application of all skills required for successful item performance • Compensatory: Successful application of some skills makes up for unsuccessful application of others, resulting in successful item performance. • Disjunctive: Success application of any one skill yields successful performance on item

11. ModelSelection • Conjunctive vs. Compensatory • Number and type of item parameters • Difficulty level of the item • How well an item discriminates between masters and non-masters of the skills measured by the item

12. Fit model to data & evaluate results • Interpret estimated model parameters. • Item parameters • Skill mastery population proportions • Re-evaluate skill definitions • Compare model-predicted statistics to observed statistics (model fit) • Individual scores • Item-pair correlations • Estimate classification accuracy

13. Example: ETS LanguEdge RC Test(By Prof. Eunice Jang, Univ. of Toronto) • Two forms: 37 & 39 multiple-choice items • ETS prototype for New TOEFL • Field test data -- 1350 examinees per form • Applied as diagnostic Pretest & Posttest in a summer English program.

14. Purpose of the Assessment • To aid instruction & learning with estimated skill mastery profiles on agreed upon skills • Simple Mastery vs. Non-mastery low-stakes decisions desired • Needed understandable skills & easy to interpret feedback on skills and scores

15. Examples of LanguEdge RC Skills: • Deduce word meaning from context (CDV) Deducing the meaning of a word or a phrase by searching and analyzing text and by using contextual clues appearing in the text. Analyze and evaluate relative importance of information in the text by distinguishing major ideas from supporting details. • Determine word meaning out of context (CIV) Determine word meaning out of context with recourse to background knowledge • Summarize major ideas from minor details (SUM) Education Example

16. Plots of Performance Difference Between Masters and Non-masters Form 1

17. Model Fit: Score Distribution

18. Comparing Students’ Skill Mastery over Time

19. Comparing Students’ Skill Mastery over Time

20. How to Interpret Skill Mastery • Nine primary reading skills are assessed in this • reading comprehension test. Please review skill • descriptions and example questions attached • to this scoring report. • The graph on the left side shows your probable • mastery standing of each skill. • The grey region indicates that your probable • mastery standing cannot be determined. • There may be some measurement error associated • with the classification. • This diagnostic information can be more useful • when used in combination with your teacher’s • and your own evaluation of your reading skills. DiagnOsis scoring reportStudent Name: Yoshi LanguEdge Reading Comprehension Test 1 1 Review Your Answers Score You earned 21 out of maximum 41 points. 8 points from 12 easy questions 8 points from 17 medium questions 5 points from 8 hard questions Scoring Correct answer to questions with 4 choices = Plus 1 point Wrong or omitted answer = No point Q13 & 25: 3 correct = 2 points, 2 correct=1 point Q37: 5 correct=3 points, 4 correct=2 points, 3 correct = 1 point Key √ Correct o Omitted + Plus partial points e = Easy, m = Medium, h = Hard (Difficulty is based on 1372 students’ performance on this test) 2 Improve Your Skills Needs improvement Not determined Mastered

21. Review Your Answers Score You earned 21 out of maximum 41 points. 8 points from 12 easy questions 8 points from 17 medium questions 5 points from 8 hard questions

23. Not all example questions are equally informative in assessing related skills. Questions are listed in the order from most informative to least informative for your review. • indicates that these skills are weak areas you need to improve. ‘?’ indicates that your mastery is not determined. 4 5 Yoshi DiagnOsisscoring report Primary Skill Descriptions and Example Questions

24. Making Progress Hi! Find your scores on the left side. Note that the two tests differ at difficulty levels. Minor differences in your scores may be associated with some measurement error. Pay close attention to the summary of reading skills mastery on the next page. Ask yourself the following questions: Which skill is consistently below 0.4? Which skill is consistently above 0.6? Which skill shows a significant improvement or remained stable? With two tests, now you can have more confidence about your strengths and weaknesses in reading ability. Use this information wisely. Estimation of your skill mastery standing is not perfectly accurate. And not all example questions provided in the skill descriptions are equally informative in assessing intended skills. DiagnOsis scoring reportStudent Name: Yoshi LanguEdge Reading Comprehension Test 2 Review Your Answers Your Scores

25. Yoshi • Skill 1: Word meanings in context You can determine which option word or phrase has the closest meaning to the author’s intended meaning by searching and analyzing surrounding text and making use of clues appearing in the text. (1, 7, 24, 5, 29, 18) • Skill 2: Vocabulary knowledge You can determine word meanings using your knowledge of vocabulary instead of heavily relying on textual clues. (3, 17, 37, 32, 27) • Skill 3: Grammatical relationships of words/phrases You can comprehend grammatical relationships of words/phrases across sentences. Typical questions assessing this skill involves identifying words or phrases that particular pronouns refer to OR determining where a new sentence can be inserted without logical or grammatical problems. (38, 25, 31, 12, 17, 24, 22) • Skill 4: Information explicitly stated in the text You can comprehend information that is explicitly stated in the text by searching across sentences within a paragraph and matching words/phrases with information that is clearly stated in the text. (23, 25, 6, 9, 2, 30, 12)

26. Yoshi • Skill 5: Information NOT explicitly stated in the text You can comprehend information that is not clearly stated in the text and determine which paraphrased option most accurately preserves the author’s intended meaning. (36, 16, 10, 4, 19, 7, 8, 34, 20, 14, 21, 22, 15) • Skill 6: Inference about arguments and author’s purpose You can make inferences about arguments that are not explicitly stated in the text or determine the author’s underlying purpose of mentioning particular phrases in text. (33, 34, 11, 35, 28, 15, 37 ) • Skill 7: Negation You can determine what information is true or not true by searching across paragraphs and identifying key information. (19, 9, 33, 14) • Skill 8: Summarizing major ideas You can identify major ideas by distinguishing them from nonessential information across paragraphs. (8, 13, 2, 6, 28, 14, 26) • Skill 9: Organization of the text and major contrasts You can determine major contrasts and arguments presented across paragraphs by recognizing the organization of the text which often contains the relationships such as compare/contrast, cause/effect, or alternative arguments. (19, 21, 39, 23, 30)

27. Implications • Diagnostic results welcomed by teachers & students • Teachers need instruction on how to integrate diagnosis results with lesson plans • Model seems to be working • Need to explore compensatory model

28. Summary • Skills diagnosis is a Team Effort. • Cognitive psychologists • Psychometricians • Content-matter experts • Need more validity studies • How to define “mastery”? • How to aligntests with classroom teaching? • How to designdiagnostic tests? • How to report “scores”?

29. Skill granularity & refinement: • 32  16  9 • Theoretical defensibility • Sufficient number of tasks per skill • Compatibility with modeling assumptions • Inter-rater agreement on skill coding • Avg. of 2 skills/item (8 items/skill) • Unidimensional IRT analysis verified assumed skill difficulties, except for one skill

30. Model Selection • Conjunctive model: Correct application of all required skills is presumed necessary for Correct item response. • Item parameter for each item/skill combination • Ratio of performance of skill non-master to performance of skill master (desired to be small) • Ability model • Dichotomous, mastery/non-mastery on each skill • Estimate is posterior probability of mastery (ppm) • ppm < .4  non-master ppm > .6  mastery

31. Fit model to data & Evaluate results • Joint calibration of two forms • Mastery proportions range: .38-.56; .10 indeterminate • Interpret item parameters: deleted 13 item/skill combinations (about 9%) • Items did not discriminate for some assigned skills. • Mastery estimation: Skill mastery consistently improved or remained same pretest to posttest.

32. Classification Accuracy Rate

33. DiagnOsis Summary report for teachersTeacher: Nancy, B Language Reading Comprehension Test 1 Review Your Students’ Answers See individual students’ report cards to review raw responses and scoring guideline Individual Students’ Scores Review Students’ Skill Mastery

34. Reading Comprehension Skills • Skills-based approach to RC is prominent in the literature. • No general agreement on a particular set of RC skills – depends on the setting.

35. Skills Set Construction for LanguEdge RC • Preliminary Analyses • Task analysis and Textual features • ETS content codes • Nonparametric IRT dimensionality analyses with field test data (3 skill clusters) • Think-aloud Protocols (18 processes)