What is High-Quality Assessment? Linking Research with Practice

What is High-Quality Assessment? Linking Research with Practice Santa Clara County Office of Education June 23, 2014 Karin K. Hess, Ed.D. khess@nciea.org or karinhessvt@gmail.com

Presentation Overview • Clarify understandings of cognitive rigor/DOK – using sample assessments & rubrics • Use the Hess Validation Tools & Protocols (Module 3) to examine technical criteria for high quality assessments: Formative & Performance • Review tools & strategies to discuss & plan future assessment activities and support to teachers • Karin’s coaching tips…

Rubric Design & Formative Tools • Revisit Handout from this morning: “What I need to do” rubric (citing evidence of proficiency) • Handout 2a: Find a half • Handout 2b: Hess Cognitive Rigor Matrix – Math-Science • Handout 2c: What will this formative assessment uncover? • Work in small groups to analyze the assessment

What do we mean by high-quality performance assessment? • At your tables, brainstorm examples of performance assessments – any content area (e.g., arts, writing, science) or real world assessments (driver’s test, marriage planning, etc.) • Have a recorder write them down • You have only 3 minutes

Turn & talk: Select one PA from your list and answer these questions: • What is it actually assessing (skills & concepts)? • What makes it a PA? • What evidence is captured in the assessment that distinguishes poor from best performances? • What makes it a “good” performance assessment? • You have 5 minutes

Let’s generalize… • With regard to skills & concepts assessed ______ • What makes something a PA? ______ • The kind of evidence that will distinguish poor from exemplary performances _______ • What makes it a “good” performance assessment? _________

What we know (from research) about High Quality Assessment: • Defined by agreed-upon standards/ expectations • Measures the individual’s learning & can take different forms/formats • Measures the effectiveness of instruction and appropriateness of curriculum • Is transparent: • Students know what is expected of them and how they will be assessed • Assessment criteria are clear and training is provided to educators and reviewers/raters. • Communicates information effectively to students, teachers, parents, administration and the public at large

Simply put, HQ assessments have… • Clarity of expectations • Alignment to the intended expectations (skills, concepts) • Reliability of scoring and interpretation of results • Attention to the intended rigor (tasks & scoring guides) • Opportunities for student engagement & decision making • Opportunities to make the assessment “fair” & unbiased for all • Linked to instruction(opportunity to learn)

2. The DOK Matrix Instructional Paths Instruction & Assessment Decisions… Selected Response DOK 4 Extended Thinking Constructed Response Each standard has an assigned Depth of Knowledge. DOK 3 Reasoning and Thinking DOK 2 Skills and Concepts Performance Tasks The DOK determines the cognitive level of instruction. DOK 1 Recall and Reproduction -Explain how concepts or ideas specifically relate to other content domains. Explain, generalize or connect ideas using supporting evidence (quote, text, evidence) Remember Explain relationships Summarize State central idea Recall, locate basic facts, definitions, details, events Devise an approach among many alternatives to research a novel problem Use concepts to solve non-routine problems and justify Select appropriate words for use when intended meaning is clearly evident. Apply Understand Use context for word meanings Use information using text features Analyze multiple sources or multiple text Analyze complex abstract themes Analyze . Analyze or interpret author’s craft (e.g., literary devices, viewpoint, or potential bias) to critique a text Evaluate relevancy, accuracy and completeness of information across texts or sources Cite evidence and develop a logical argument for conjectures based on one text or problem Evaluate Create . Develop a complex model or approach for a given situation Develop an alternative solution Synthesize across multiple sources/ texts Articulate a new voice, theme, or perspective

GOAL: Each “validated” assessment will demonstrate: • Clarity of expectations for the student and teacher(s) • Alignment (task & scoring) to the intended standards: content & performance/DOK • Provide opportunities for student engagement • Provide opportunities to make the assessment “fair” & unbiased for ALL students

First we consider alignment… • It’s really about validity – making decisions about the degree to which there is a “strong match” between grade level content standards + performance and the assessment/test questions/tasks • And making valid inferences about learning resulting from an assessment score

“Validity is a matter of degree, rather than all or none.”Robert Lynn, 2008

Alignment (validity) Questions: • Is there a strong content matchbetween assessment/test questions/tasks and grade level standards? • Are the test questions/tasks (and the assessment as a whole) more rigorous, less rigorous, or of comparable rigor (DOK) to grade level performance standards?

Task Validation Protocol Handout # 3(K. Hess, 2013) • Table Groups review the technical criteria and descriptions on pages 3-4 in the protocol at your tables • What’s one aspect you feel you (or teachers you work with) now do well in most local assessments? • What’s one aspect you feel you (or your teachers) need to understand more deeply as you work with them?

Uses of the assessment task validation tools & protocols • Develop new assessments • Analyze existing assessments • Validate a revised assessment or new assessment prior to broader administration (or purchase) • Provide OBJECTIVE feedback to assessment developers • Promote collaboration and a shared understanding of high quality assessment

Local Validation Teams represent the diversity of the school • Administrator/Leader/Coach • All* content areas represented • All/most grade levels (grade spans)* represented • PLUS Representation from special education, fine arts, HPE, CTE, foreign language, ELL, etc. *decisions may differ depending on school configurations and staffing, but diversity in teams is critical, especially including special educators

Frequency of Validations? • Initially learning & debriefing the process together serves as calibration - so everyone is on the same page – “developing a shared understanding” of what high quality assessment looks like • School teams set up their schedules – once each month, every other month, as needed, highest priority, etc. • Team members may rotate on-off so more (all) staff are involved over time

Getting ready for validation • Grade level or department teams develop the assessments using the Basic Validation Protocol (e.g., a gr 2 team might develop a common math assessment for all gr 2 classes/schools) • Developers put the assessment on the local (school/district) validation calendar • Validation teams prioritize order of validations – common assessments, major assessments first, second round review after getting feedback, etc.

Validation Materials • Each team member needs (electronic) validation protocols (Handout: Module 3, pages 3-4) • Each person needs a copy of the cover page with the assessment and scoring rubric/answer key (Handout: Module 3, pages 5-6) • There may be additional materials – e.g., anchor papers, examples that do not need to copied for everyone but may be helpful to see during the review • Each person needs a content specific DOK reference sheet (Handout: Module 1, tools #1, #2, or # 3)

Validation Protocols [1] • Each time, preview norms for working together • I am… • I am NOT… • Choose a recorder – to keep an electronic record & provide a copy of feedback for the assessment developers • Date and list validation panel names on the “official copy” (this can be set up ahead of time) • Individually, take 5-10 minutes to read through & make notes before any discussion

Sample norms (Source: adapted from Powell, WY) I AM • Keeping electronic devices on vibrate/off • Listening to understand other points of view • Respecting everyone as a professional • Focusing on the issues • Avoiding side conversations • Encouraging everyone having a turn to speak • Refraining from judgmental statements • Representing the best interests of all students • Asking clarifying questions • Demonstrating a commitment to the process (attending meetings, on time, etc.) • Others? I AM NOT • Using killer phrases • Preparing my next remark instead of listening • Sounding apologetic • Engaging in unrelated activities • Using negative gestures/body language • others?

Optional -Validation Protocols [2] • Should the authors present the task at the start? (especially if 2nd round) – there are pros & cons to this • Go over what is on the cover page/what is included and what the purpose of the assessment is • 2-5 minutes to explain the materials in the packet – no interruptions from validation panel • Panel then asks any clarifying questions only • The is NOT for depth of understanding, just to know/clarify what is there BEFORE silently reading & discussing

Validation Protocols [3] • Make notes individually before discussion • Choose a task manager/ timekeeper to keep things moving – reads each indicator on the Validation Protocols • Have a process to reach consensus (fist 5, thumbs up, etc.)- be sure to involve each person! • Choose 2 people to give feedback to the authors/developers & “rehearse” comments • DEBRIEF! Did we honor norms? What went well/needs to be refined next time?

Giving Feedback • Use descriptive language, NOT judgmental language • While you may wonder about instructional pieces, comments/suggestions about instruction are probably not appropriate • Your job is NOT to redo the assessment! Keep feedback crisp & to the point (e.g., pose a question)- it is the developer’s job to decide what to do next to strengthen the assessment tasks.

Giving Feedback (continued) • Well-written, clear feedback guides assessment developers to make a stronger assessment in the end. • Place your positive (and descriptive) comments under the feedback section (Module 3, page 7): What makes this a HQ (high quality) assessment?

Examples of Feedback (noted on page 7) • We were unable to locate… • We think this might be DOK2, not DOK3 because…what do you think? • We were not clear what the student is expected to do or to produce. Did you mean…? • This might be better aligned to this standard … • As hard as it will be, avoid saying “we liked…” This implies you did not like other things and your job is NOT to like the assessment. • Include the “HQ” positives! The directions are clear; students have authentic choices; etc.

Debrief each time! • Did the validation team honor the norms at all times? • Do we need to modify/revise norms? • What went well? • What could have gone better? • What will we do differently next time? • Who/when will we meet with authors to give feedback?

What is High-Quality Assessment? Linking Research with Practice