Assessment Directors Meeting August 10, 2006

1. Assessment Directors MeetingAugust 10, 2006 Judy W. Park Assessment & Accountability Director Utah State Office of Education July 2006

2. Welcome Introductions District Sharing Cards

3. District Assessment Handbook

4. Put Students in the Drivers Seat • Clear Core Curriculum • Clear Test Purpose • Quality Assessments • Error Free • Available on time • Trained Administrators

5. Put Students in the Drivers Seat • Testing Environment • Standard Administration • All tools available • Immediate Results • Quality Results Interpretation • Test Results are clearly connected to preparing for the next test

6. Put Students in the Drivers Seat • Clear Core Curriculum • Clear Test Purpose • Quality Assessments • Trained Administrators • Testing Environment • Immediate Results • Quality Results Interpretation • Test Results are clearly connected to preparing for the next test

7. Fall Road Trip • District Responsibility • Invite A & A • Host meeting – computer lab setting • Audience – Determined by District • School Administrators? • School Test Coordinators? • School Counselors? • Teachers? • Agenda • Put Students in the Driver’s Seat • Assessment & Accountability Literacy • Standard Test Administration & Test Ethics • Understanding Summative Test Reports • Using U-PASS & other data to inform instruction & improve student achievement

8. Test Development • August – Blueprint • September - Item Development • October – Item Review (Content) • November – Bias & Sensitivity Review • January – Construct Pilot Test Form Take Test • February - Data Review • March – Construct Operational Test Form Take Test • May – Standard Setting, Scoring, Reporting Review Test Results

9. Test Development August – Blueprint Just as a blueprint of a house delineates the house’s framework, the test blueprint outlines the test’s framework.

10. Decision Guidelines for Blueprint Construction • The standards and objectives selected for the blueprint must represent the entire curriculum. • Only those objectives which can clearly be assessed in a multiple-choice format are selected for the blueprint. • The number of test items assigned to each objective reflects the depth and breadth of the indicators within the objective. • The following questions should be asked: What proportion of the items on a test should represent each standard of the core curriculum? What proportion of the items should represent each objective?

11. Decision Guidelines for Blueprint Construction • The total number of questions on a test must be large enough to provide a fair sample of student performance across the standards and objectives. • We need to determine the appropriateness of making inferences about a student’s proficiency with respect to the standard and objective based upon “x” number of items. • Items are aligned to the standards and objectives and not the indicators.

12. Test Aligns to Standards and Objectives • Item writers first refer to the core curriculum, specifically to the portions of the core assessed on a CRT. • Standards are very broad, educational goals. • Objectives are broad statements of what a student should be able to do. • Indicators are the descriptions of how students demonstrate that they can perform the objective. • There are multiple indicators for each objective, which provide a broad spectrum of content skills assessed on each year’s test. • Indicators provide a reference for composing test items.

13. Caveats • Every blueprint shows: • The Standard • The Objective • The number of questions on the CRT for each objective • Only those objectives which can clearly be assessed in a multiple-choice format are selected for the blueprint. • The number of test items assigned to each objective reflects the depth and breadth of the indicators within the objective.

14. Assessment Updates • IOWA • UBSCT • Math CRT • UTIPS • NAEP

15. Accountability Update • Applied Math 1 & 2 • Not included in accountability calculations • August 15? • Electronic Reports on ftp site • Research disk • August 20 – 30 • CRT Reports & Profiles electronic copy • Contact Sharon Marsh if you want hard copies • September 20 • Cumulative Student Report electronic copy • Contact Sharon Marsh if you want hard copies

16. Accountability Update Accountability Meetings for U-PASS, AYP, AMAO • Sevier District Office – August 18 • Granite District Office – August 22 • Weber District Office – August 23 • Nebo Learning Center – August 25

17. U-PASS 2006 Reports • May Board Meeting • Request to explore changes • June Board Meeting • Changes approved

18. U-PASS 2006 Reports • Participation 95% • Status 80% • Progress 190 • Confidence Interval 95% Projection - Identify 12.7% of schools

19. U-PASS High School Task Force June 5, 2006 Minutes from this meeting are in red

20. 2006 Timeline • April 2006 • Meeting to review designs and make decisions • May – July • Draft business rules • August Assessment Directors Meeting • Share business rules • August – October • USOE run all of the schools through the business rules • Districts may replicate analyses in their districts • USOE run multiple scenarios for status/progress • November 14—Task Force Meeting! • Meeting to review and clean up the business rules • Review data and the multiple scenarios • Make initial decisions for status & progress

21. 2007 Milestones • January - February 2007 • USOE Rerun all reports and fix all business rules • Districts replicate the rules for their districts • March 2007—Task Force Meeting! • Meeting to review data and business rules • Final planning meeting to approve business rules and cutscores • April 2007 • Develop initial school designations • Meeting for districts to review all school designations • Make any final adjustments to business rules and calculations • July 15 • Run all AYP/U-PASS/AMAO Reports • August 15 • Release all Accountability to districts for 30 day review • September 15 • Release Accountability reports to the public

22. U-PASS Purpose • To identify schools in need of assistance to meet state standards • This is a “minimum competency” goal. It will be important to keep this in mind as we consider what values are reflected in the model.

23. General Principles • The accountability system should be based on multiple indicators to provide a more complete picture of schools • But should not overwhelm schools’ data capacity • The measures should be combined using a compensatory framework to improve the reliability and validity of the decisions • Every group should be required to meet an acceptable level of status or progress, but not both

24. Subgroups • The same approach for holding subgroups accountable (i.e. the “super-subgroup”) in the 3-8 system will be used in the high school system as well

25. A Stratified Approach • Long-term plan that focuses on valuing the “most important” things that high schools should be doing • Short-term plan that addresses things that can be measured fairly across schools and not create a huge data burden

26. The Short-term reality • While the long-term system is appealing, we are probably several years away from implementing such an approach • We need to find a short-term solution that meets the requirements of the law, supports the long-term view, is fair, and can be implemented in 2007

27. Short-term proposal • This approach focuses on a specific “checkpoint” to identify schools most in need of assistance to meet state standards • Participation, growth, and status should be required components of the system • Subgroups must be held to the same standards as the full school

28. Participation • Will be calculated using: • UBSCT--10th graders attempting all 3 subtests/all 10th graders • CRTs—all CRT test takers/all students enrolled in CRT courses • DWA—all students tested/all 9th graders Agreed!!!!

29. HS Status Components • 10th grade UBSCT scores in—reading, writing, math • CRTs—counting all those scores earned by : • ELA--9, 10, 11 • Math CRTs—Algebra, Geometry • It was suggested including all math CRT scores whenever they take it • Science—Earth Science, Biology, Chemistry, Physics • All 9th grade Science CRT would count • All 10th grade Science CRT would count • Only the 11th grade CRT for students who did not take the science CRT in both 9th and 10th. (For the kids who have not received the 2 credits by 11th grade, their 11th grade science CRT will count) • Denominator= 11th graders and below with 2 CRT credits or fewer

30. HS Status Components • Math Courses—full year credit for 9th and 10th grade students enrolled in a non CRT course • Students earning credit/total number of students enrolled in non CRT classes in 9th and 10th grade • What is a non CRT course? (Concurrent enrollment, Remedial math, CTE courses?) • DWA—9th grade • UAA—in 9th and 10th grades • Graduation/completion rate • Attendance (18 or more absences/year)???? • The 3-8 system has attendance as 15 or more absences per year.

31. UBSCT Status • The denominator will be all 10th graders in the school • The numerator will be the count of students passing the first administration

32. Graduation rate • Same calculation method as AYP • Will move to a longitudinal system when the data system allows for it • What are the targets for graduation? • Same as AYP? • Will this simply be incorporated into the status calculation? If so, what metric will we use? • For example, given the current status calculation plan a difference in grad rates between 80 and 95% might NOT have a noticeable effect on overall status. • Should graduation rate be a separate, conjunctive indicator? Graduation rate was not discussed – we ran out of time.

33. Status Computation Questions • How do we want to weight the various status components? • All content areas weighted the same? • Do math courses count the same as CRT scores? • Graduation? Separate or incorporated? • UBSCT weight This was not discussed – we ran out of time

34. Growth components • ELA--longitudinal growth from grades 8-11 • For Science and Math: • Use 8th grade scores as a pretest for the 10th grade CRT • For UBSCT: • Use the same subject CRT score in 8th grade as a pretest score for the UBSCT subject area test and measure student longitudinal growth • These would all be measured using a value table approach similar to the 3-8 assessments • We could build a UBSCT improvement statistic—evaluate the difference between initial pass rate and ultimate pass rate • This was not discussed – we ran out of time

35. U-PASS High School Next Steps Committee Meetings will continue

36. AYP • New Amendment • An LEA is identified for improvement only when it misses AYP in the same subject and in all grade spans for two consecutive years, or the other academic indicator in all grade spans for two consecutive years. • New Report

37. Standard Testing Administration Testing Ethics • What are the concerns? • Lack of Training? • Inappropriate behavior? • What are the solutions?

38. District Sharing

39. New Accommodations Policy

40. Utah Academic Language Proficiency Assessment UALPA

41. Proficiency Levels A P Pre-Emergent B E Emergent C I Intermediate D A Advanced E F Fluent

42. Students to be assessed • All P (pre-emergent) students (A) • All E (emergent) students (B) • All I (intermediate) students (C) • No A (advanced) students (D) • No F (fluent) students (E) • No Native Speakers tested

43. Students to be assessed • Students double tested to bridge scores from IPT to UALPA • The same student must take the IPT and the UALPA in the same time frame Minimal Requirement • Double test 8% of P (A) students • Double test 3% of E (B) students • Double test 5% of I (C) students

44. Placement Test • IPT used for Placement 06-07 year • UALPA used for Placement in 07-08 year?

45. Test Materials Student Booklet • 1 booklet for each grade span • K - flip chart • Speaking – flip chart • One flip chart for 1 – 6 • One flip chart for 7 – 12 • 1-2 consumable booklet • 3-6, 7-8, 9-12 student booklet • Booklet includes listening, reading, writing

46. Test Materials Answer document for each student • K • Teacher completed • 1-2 consumable booklet • 3-6, 7-8, 9-12 • Speaking – teacher completed • Listening, reading, writing - Student completed

47. Student Booklet

48. Answer Document

49. Test Materials • Test Administration Manual (TAM) • Includes basic training information • Includes all administration information • Examiner script • Includes scoring guides for • Speaking – teacher completed • K – teacher completed • Coordinators Manual • Test distribution information • Test return information • Scoring information

50. TEST DESIGN