1 / 68

Review and Validation of ISAT Performance Levels for 2006 and Beyond

Objectives for State Testing Review Committee. Review cut score validation processReview modifications to grade 8 mathematics cut scoresReview grade 5 mathematics cut score proposalReview new ISAT reporting scale proposal. Purpose of the Validation Panel. Review cutoff values in relationship toPerformance definitionsAssessment frameworksRepresentative item difficultiesDistrict/school performance profilesJudge the reasonableness of the proposed cutoffs.

jalila
Download Presentation

Review and Validation of ISAT Performance Levels for 2006 and Beyond

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. Review and Validation of ISAT Performance Levels for 2006 and Beyond MetriTech, Inc. Champaign, IL Presented January 4, 2006 at Springfield to State Testing Review Committee members.Presented January 4, 2006 at Springfield to State Testing Review Committee members.

    2. Objectives for State Testing Review Committee Review cut score validation process Review modifications to grade 8 mathematics cut scores Review grade 5 mathematics cut score proposal Review new ISAT reporting scale proposal

    3. Purpose of the Validation Panel Review cutoff values in relationship to Performance definitions Assessment frameworks Representative item difficulties District/school performance profiles Judge the reasonableness of the proposed cutoffs

    4. Process Review development of ISAT performance levels Review equating process to SAT-10 vertical scale Consider proposed cutoffs in relationship to frameworks, definitions, data Evaluate the reasonableness of the outcomes

    5. Constraints NCLB requirements for continuity in school/district accountability Grades 3, 5, 8 frame possibilities for grades 4, 6, 7

    6. Outcome

    7. ISAT/SAT-10 Bridge Study

    8. Bridge Study Purposes SAT-10 will be the basis for the enhanced ISAT vertical scales Vertical scale will be the basis for establishing intermediate grade performance levels (reading/math grades 4, 6, 7)

    9. Description of the Equating Sample 9793 SAT-10 Records 9 districts 47 schools grades 3, 4, 5, 7, 8

    10. ISAT/SAT-10 Data Record Match

    11. Descriptive Statistics for Matched Samples: Reading Tests

    12. Descriptive Statistics for Matched Samples: Mathematics Tests

    13. Descriptive Statistics for Matched Samples: Science Tests

    14. Statistical Comparison of Matched Sample SAT-10 Means With SAT-10 Spring Norms

    15. Statistical Comparison of Matched Sample ISAT Means With ISAT 2005 Test Population

    16. Correlation Between Corresponding ISAT/SAT-10 Ability Estimates (1) SAT-10 subtests corresponding to each ISAT test are as follows: ISAT reading-SAT-10 Reading Comprehension, ISAT mathematics-SAT-10 Problem Solving, ISAT science-SAT-10 Science. (2) For the science samples, there were large numbers of matched records for students who had not taken the SAT-10 science test. Thus, the larger numbers of cases excluded as outliers. (1) SAT-10 subtests corresponding to each ISAT test are as follows: ISAT reading-SAT-10 Reading Comprehension, ISAT mathematics-SAT-10 Problem Solving, ISAT science-SAT-10 Science. (2) For the science samples, there were large numbers of matched records for students who had not taken the SAT-10 science test. Thus, the larger numbers of cases excluded as outliers.

    17. Equating Methods Rasch (outliers included) Rasch (outliers excluded) Equipercentile Equipercentile with linear smoothing

    18. Equating Outcomes Rasch/Equipercentile results similar with respect to cut points Rasch: 635.6 Equipercentile: 635.3 Outlier impact Tended to eliminate higher ISAT/lower SAT-10 scores Inconsistent with past practice

    19. Constants For Transforming ISAT Scales to the SAT-10 Vertical Scales

    20. Table for Transforming ISAT Scores to SAT-10 Vertical Scales

    21. Transformation of the Performance Level Cut Scores

    22. ISAT Performance Category Score Ranges on the Current Scales

    23. ISAT Cut Scores Expressed on the SAT-10 Vertical Scales (Lower Bounds)

    24. Reading Cut Scores Expressed on the SAT-10 Vertical Scale

    25. Science Cut Scores Expressed on the SAT-10 Vertical Scale

    26. Mathematics Cut Scores Expressed on the SAT-10 Vertical Scale

    27. Adjustment to the Grade 8 Mathematics Cut Scores

    28. Large Discrepancies Across Grades (Grade 3: 79%; Grade 5: 73%; Grade 8: 54%) Multiple Panels Used in the Original Derivation (162, 149) Discrepancies in National Percentile Ranks Reasons for Reexamining the Cut Scores

    29. National Percentile Ranks Corresponding to Each ISAT Cut Score

    30. Estimating Performance Level Cut Scores for Intermediate Grades

    31. Reading Cut Scores With Interpolations for All Grades

    32. Mathematics Cut Scores With Interpolations for All Grades

    33. The Evaluation Process Review performance definitions and assessment frameworks Review difficulty-ordered item booklets Review district/school performance profiles Make initial judgments Large group discussion Final judgments

    34. Difficulty-Ordered Item Booklets Represent a range of probable performances by students at different levels Primarily represent the Below Standards/ Meets Standards performance range

    35. Difficulty-Ordered Reading Booklets

    36. Difficulty-Ordered Mathematics Booklets

    37. District 1 Performance Profile

    38. District 1 Performance Profile

    39. District 1 Performance Profile

    40. District 1 Performance Profile

    41. District 2 Performance Profile

    42. District 2 Performance Profile

    43. District 2 Performance Profile

    44. District 2 Performance Profile

    45. School 1 Performance Profile

    46. School 1 Performance Profile

    47. School 1 Performance Profile

    48. School 1 Performance Profile

    49. Evaluation Worksheet

    50. Evaluation Outcomes

    51. Evaluation Outcomes

    52. Panel Recommendations for Grade 5 Exceeds Cut Score in Mathematics Large Discrepancies Across Grades— Grade 3: 34%; Grade 5: 12%; Grade 8: 17% (before adjustment, but about 29% after adjustment) Multiple Panels Used in the Original Derivation —Current cut score uses higher of two panels’ recommendations. Other panel would have about 33%. Exceeds items in the meets performance range Discrepancies in National Percentile Ranks

    53. National Percentile Ranks Corresponding to Each ISAT Cut Score

    54. Options to Consider Anchor existing grade 3 and adjusted grade 8. Interpolate intermediate grades Anchor on adjusted grade 3 and adjusted grade 8.

    55. Defining the 2006 ISAT Reporting Scales

    56. Enhanced ISAT Vertical Scale Linear transformation of the SAT-10 vertical scale Unit size Range consideration

    57. Scenario Unit size approximately twice that of the current scale Anchor lower end of grade 3 (reading, mathematics) or grade 4 (science) scale at 120 How would things have looked in 2005 if the new reporting scale were used?

    58. Reading

    59. Mathematics

    60. Science

    61. Reading Means

    62. Mathematics Means

    63. Science Means

    64. Score Distribution (3, 5, 8)

    65. Score Distribution (3, 5, 8)

    66. Score Distribution (4,7)

    67. Constants for Transforming SAT-10 Scales to New ISAT Reporting Scales Characteristics of New Scales Reading: 50 scored multiple-choice items; 1 extended-response item scored on a 4-point scale. Weighting of ER item: 2.21 Raw Score Scale: 0-59 (versus 75 in the past) Math: 65 scored MC items; 1 ER item scored on 12-point scale; 2 SCR items scored on 2-point scale Weighting of ER items: .72 Raw Score Scale: 0-77 (versus 82 in past)Characteristics of New Scales Reading: 50 scored multiple-choice items; 1 extended-response item scored on a 4-point scale. Weighting of ER item: 2.21 Raw Score Scale: 0-59 (versus 75 in the past) Math: 65 scored MC items; 1 ER item scored on 12-point scale; 2 SCR items scored on 2-point scale Weighting of ER items: .72 Raw Score Scale: 0-77 (versus 82 in past)

    68. Constants for Transforming Existing (1999) ISAT Scales to New (2006) ISAT Reporting Scales

More Related