Monitoring and Pre-Scoring

1. Monitoring and Pre-Scoring Virginia Alternate Assessment Program (VAAP) Virginia Department of Education

2. Purpose The purpose of this presentation is to provide guidance to school divisions on monitoring and conducting pre-scoring activities for Virginia Alternate Assessment Program (VAAP) Collections of Evidence.

3. Monitoring and Pre-Scoring Plans • Monitoring and pre-scoring plans can be essential to early detection and prevention of issues within the Virginia Alternate Assessment Program (VAAP) Collection of Evidence (COE). • School divisions are encouraged to develop and implement monitoring and pre-scoring plans to ensure VAAP COE are developed under testing conditions. • Ensure scoring rules outlined in the VAAP Implementation Manual and the Virginia Alternate Assessments Administrator’s Manual are adhered to.

4. Monitoring Periodic and systematic review of evidence-based assessments in the development process with opportunities for feedback and intervention. Why Monitor? • To create better collections of evidence(COE). • To increase the alignment between the student’s level of achievement and the evidence presented in the COE. • To catch potential problems early.

5. What Monitoring is Not? • Checking with the teacher • Flipping through a collection of evidence • Review without feedback Prevent the “MAD DASH” before the COEs are due

6. Why collections of evidence are not successful • Missing or incomplete Student Evidence Identification (SEI) tags • Ungraded or incorrectly graded evidence • Use of unacceptable evidence • Missing evidence to support the Aligned Standards of Learning (ASOL) • Use of evidence not aligned with the ASOL • Evidence does not support the level of performance for the ASOL

7. Monitoring Decisions • Who will monitor? • How will you train them? • When will monitoring occur? • Who will receive feedback? • How will feedback be given? • What are your intervention options?

8. Components of Monitoring • Trained Reviewers • Feedback Loops • Intervention Options

9. Trained Reviewers • Trained reviewers may include: • Persons with Content Knowledge and Special Education Experience • School Test Coordinator • Assigned or Hired Staff • Building Level Teams • Central Office Level Teams

10. Feedback Loops • Teachers preparing collections • Principals • Building Level Coordinators (VAAP) • Trainers/Technical Assistance Staff • Service Providers

11. Intervention Options • Attendance at local training • Access to state and local ASOL resources or VAAP assessment resources • Peer or small group sessions • Individualized consultation and support

12. A Sample Division Monitoring Plan • School-based Review Teams created and trained • Teachers turn in collections of evidence every nine-weeks • All collections of evidence are reviewed by team according to division pacing chart • Team reports shared with principal and central office • Central office assigns Content Teams to address concerns with evidence.

13. A Sample Monitoring Plan

14. Pre-Scoring A detailed review of evidence-based assessments prior to submission for scoring. Why Pre-score? • To provide a last check to ensure that all the scoring rules have been followed • To ensure that no last-minute detail has been overlooked • To provide an opportunity to make changes before the COE is submitted for scoring

15. What Pre-scoring is Not? • A “mad dash” to create a COE • The time to provide instruction

16. Pre-scoring Decisions • Who will pre-score? • How will you train them? • When will pre-scoring occur in relation to the submission date? • Who will receive feedback? • How will feedback be given? • What are your intervention options?

17. Sample Pre-scoring Plan Building Level Review • Review of COE using the 2019-2020 • Teacher Checklist • Administrator Checklist • Completed Collection of Evidence Check list • School-based oversight teams review • Building Administrator review and sign-off • Return to Teacher to address concerns Central Office Review • Review of COE using the 2019-2020 • Administrator Checklist • Completed Collection of Evidence Check list • Reviewed by central office teams • Return to schools for correction of identified errors or omissions

18. Tools for Pre-Scoring VAAP Implementation Manual • Teacher Checklist for Collections of Evidence p.37 – 39 • Administrator Checklist for Collections of Evidence p.40 – 41 • Completed COE Submission Checklist p.42 - 43

19. Media Considerations • Do videotapes and audiotapes have transcripts? • Are videotapes and audiotapes clearly labeled with SEI tags? SEI tags should be on the case, not on the DVD. • Is equipment/software needed to review the media clearly identified on the case?

20. Organizational Considerations • Is the evidence organized according to the scoring worksheet? • Are all required state and local forms completed?

21. VAAP COLLECTIONS OF EVIDENCE

22. Collection of Evidence (COE) • The VAAP provides the student the opportunity to demonstrate what he or she knows through a non-traditional method of testing. • By completing products and work samples that demonstrate his/her understanding or skill, under testing conditions the student proves that he/she knows the content.

23. Collections of Evidence All persons who are responsible for collecting, reviewing, scoring, or handling the COE, or authorized to have access to the COE, must ensure that the student’s information remains confidential and that the evidence is created under testing conditions and maintained in a secure manner.

24. Testing Conditions • Evidence collected for the VAAP must be completed in the presence of the teacher or other school personnel and under testing conditions. • All curricular materials related to the content and test-taking strategies that might influence student performance or provide an unfair advantage when the student is completing work samples, interviews, and other types of evidence must be removed from the site. • These materials include, but are not limited to: class notes, study guides, graphic organizers, maps, charts, posters, projections, computer programs, textbooks, dry-erase board displays, chalkboard displays, and bulletin board displays.

25. Testing Conditions (2 of 2) • In addition, directions or examples on work samples must not include hints, clueing, or prompts that provide answers or an unfair advantage for the student. • Students may not have access to any items not approved for use during testing. These items include, but are not limited to: cell phones, electronic devices including wearable technology, class notes, course content review materials, test-taking strategy reminders, notes, “cheat sheets,” textbooks, and other curriculum materials.

26. Secure Measures • All COE must be kept in a secure location when not in use; • Establishing and maintaining appropriate monitoring activities for COE while in development; • The School Division Personnel Test Security Agreement for the Virginia Alternate Assessment Program has been signed by all staff who have access to the COE;

27. Secure Measures (2 of 2) • The Affidavit of Student Performance has been signed by all staff involved with implementation and development of COE; and • Ensuring evidence submitted in the COE was completed under testing conditions.

28. Violations of Confidentiality • COE left unattended or in a location accessible to persons not involved with the development of the collection; and • COE left in an inappropriate location such as a vehicle or home.

29. Components of a Collection of Evidence • A COE submitted for scoring to the local school division must contain the following components in this order: • A signed2019-2020 VAAP Affidavit of Student Performance ( this includes a signed Interpreter’s Affidavit.) • Any locally required documentation • A 2019-2020 VAAP Content Area Cover Sheet • Evidence that demonstrates individual achievement on ASOL selected from each reporting category for a specific grade level and content area.

30. VAAP Affidavit of Student Performance Ensures that all the work samples included in the COE were completed independently by the student, in the presence of a teacher or other school personnel, and under testing conditions in which the student did not have access to hints, clueing, or prompts that would provide answers.

31. VAAP Affidavit of Student Performance (2 of 2) • Persons who have supervised the student while completing evidence are required to sign an affidavit. • The principal or designee who reviewed the COE prior to submission for scoring must also sign the affidavit.

32. Examples of VAAP Affidavit of Student Performance Administrator Signature is dated after the spring 2019 administration has concluded. Has the collection of evidence been reviewed by an administrator? Dates missing from Teacher Signature

33. Examples of VAAP Affidavit of Student Performance (2 of 3) Date missing for Administrator’s Signature. Missing Administrator’s Signature and Date

34. Examples of VAAP Affidavit of Student Performance (3 of 3)

35. VAAP Content Area Cover Sheet • All evidence for an assessed content area should be organized using the 2019-2020 VAAP Content Area Cover Sheet. • A separate Content Area Cover Sheet must be submitted for each content area. • The Level of Performance is not indicated in the content area of History/Social Science.

36. VAAP Content Area Cover Sheet (2 of 2) • In addition to providing student information, the cover sheet should • Identify the content area • Identify the Reporting category; and • Identify the ASOL and bullet • The student’s level of performance must be indicated on the Content Area Cover Sheet for the content areas of Reading, Writing, Mathematics, and Science.

37. VAAP Content Area Cover Sheet and SEI Tag Discrepancy

38. Student Evidence ID (SEI) Tag VAAP CONTENT AREA: READING ASOL: 3E-RW BULLET: C LEVEL OF PERFORMANCE: III Red Student Evidence ID (SEI) Tag VAAP CONTENT AREA: MATHEMATICS ASOL: 7M-NSCE 2 BULLET: LEVEL OF PERFORMANCE: Student Evidence ID (SEI) Tag VAAP CONTENT AREA: ASOL: 8S-ECO 2 BULLET: D LEVEL OF PERFORMANCE: Scoring Rule 4Incomplete Student Evidence Identification (SEI)Tags If there is no SEI Tag or if the SEI Tag is mislabeled or incomplete, the evidence cannot be considered in scoring. Top: SEI Tag is missing the ASOL number. Middle: SEI Tag is missing the bullet. Bottom: SEI Tag is missing the content area and the level of performance.

39. VAAP Evidence

40. VAAP Collections of Evidence Various types of acceptable evidence submitted: • Work Samples Interviews • Audio Data Charts/Graph • Video Captioned Photographs • Anecdotal Record

41. VAAP Evidence (2 of 2) Anecdotal records, interviews, and other evidence that are “observations” of student skills and/or knowledge should be written in such a manner that the scorer can determine the level of individual achievement of the ASOL demonstrated.

42. VAAP Reading, Writing, Mathematics, and Science • For Reading, Writing, Mathematics, and Science, teachers must select Aligned Standards of Learning (ASOL) only from the grade level in which the student is enrolled. • For the content areas listed above, the teacher must use the Level of Performance system to indicate whether the student: • demonstrated the ASOL with significant support and modification(Level I), • partially demonstrated the ASOL (Level II), or • fully demonstrated the ASOL (Level III).

43. Selecting ASOL The Levels of Performance provide flexibility for instruction and assessment. Examine the ASOL: • Consider all the skills listed in the stem and the bullet • Is the student able to fully demonstrate all the requirements of the ASOL? • Are there specific skills of the ASOL that the student can demonstrate? • Are significant supports and modifications needed to assist the student in demonstrating the ASOL?

44. Selecting ASOL and Level of Performance Determine the Level of Performance: If the student • can demonstrate all the skills required by the ASOL, consider Level of Performance III. • can demonstrate some of the skills required by the ASOL, consider Level of Performance II. • needs significant support and modification to show understanding of the skills required by the ASOL, consider Level of Performance I.

45. Evidence must show an understanding of: • Characteristics of moving objects • Interactions of moving objects • Moving objects have kinetic energy Science ASOL Example 5S-FME 1 The student will investigate and understand characteristics and interactionsof moving objects. Key concepts include d) moving objects have kinetic energy.

46. Reading ASOL Example • Evidence must show an understanding of: • Main idea AND • Use a fictional text 4E-CF 1 The student will b) determine the main idea of a fictional text

47. Math ASOL Example • Evidence must show an understanding of: • Adding double-digit whole numbers AND • Subtracting double-digit whole numbers 4M-NSCE 4 The student will b) Add and subtract double-digit whole numbers

48. History ASOL Example • The student will need to show evidence of: • History describes events and people of other times and places. • Identify the people and events honored by the holidays of : • Thanksgiving Day • Martin Luther King Jr Day • President’s Day • Independence Day (Fourth of July) • HS-H 1 The student will recognize that history describes events and people of other times and places by • b) identifying the people and events honored by the holidays of Thanksgiving Day; Martin Luther King, Jr., Day; Presidents’ Day; and Independence Day (Fourth of July).

49. Levels of Performance The scoring event coordinator does NOThave the flexibility to change or adjust the Levels of Performance on the SEI Tag. The Levels of Performance are not considered technical errors.

50. Hints, Clues, and Prompts