Contingencies for the use of effective educational practices: Developing Utah’s Alternate Assessment

1. Contingencies for the use of effective educational practices: Developing Utah’s Alternate Assessment Tim Slocum Wing Institute First Annual Summit April 2006

2. Main Topics • Contingencies and evidence-based special education • Designing an alternate assessment that can bring powerful NCLB contingencies to bear on promoting evidence-based special education.

3. Contingencies and Evidence-Based Practices

4. Contingencies • A contingency is a relationship between behavior and consequences. • If a certain behavior occurs, then a certain consequence will occur. • Many contingencies are “natural” - that is, not intentionally constructed • Physical – Effects of jumping out of a window on falling • Social – Effects of speaking unclearly on interaction • Other contingencies are intentionally constructed to influence behavior • Parenting – Family rules • Education – Classroom motivation systems • Legislation and policy – Laws and regulations

5. Contingencies and Evidence-Based Practices • When we consider factors that may promote or undermine a culture of evidence-based practices, we should consider contingencies that may make such a culture more or less relevant. • What legislative and policy contingencies provide a context for evidence-based practices?

6. Effects of Contingencies • Contingencies can be very powerful for changing behavior. • When powerful reinforcers are given contingent on a particular desirable behavior, changes in that behavior can be dramatic. • However, if contingencies are not carefully constructed, they can produce unintended outcomes. • A teacher may offer a reward contingent on increased oral reading rate. This may result in increased oral reading rate, but also increased errors.

7. Contingencies – Defining Targets • There are two ways to define behaviors in a contingency: • By their form – specifying what must be done: • Examples • Study 2 hours per night… • Reading First – a federal program that specifies forms of behavior (research-based reading instruction) necessary to get a reward (funding). • This requires a specific form of behavior but leaves the outcome unspecified. • Unintended outcomes can come from completing the behavior in ways that do not produce the intended outcomes. • Consider all the ineffective “study behaviors” that a student might engage in. • Consider how mandated processes (IEP, functional analysis, implementation of research-based practices) become formalized and fail to achieve intended purposes.

8. Contingencies – Defining Targets • By their results – specifying what must be achieved: • Examples • Get a GPA above 3.75… • No Child Left Behind – a federal program that specifies outcomes (AYP) necessary to avoid punishment (public scorn, loss of control, loss of funding) • This specifies what is to be accomplished but leaves the process unspecified. • Unintended consequence can come from ways of producing the narrowly defined outcomes without producing the intended outcomes. • Getting a high GPA by taking easy classes • Raising test scores by excluding lower performing students.

9. Contingencies • So, whether we aim at what a person must do (form) or what they must accomplish (achieve), we must be very careful about defining what constitutes fulfilling the target of the contingency. • When contingencies specify difficult behaviors and promise strong consequences, the chances of unintended consequences are most serious.

10. Contingencies and NCLB • We can understand NCLB as a set of contingencies that specify results but leave specific forms of behavior unspecified. • NCLB has produced a lot of behavior oriented toward achieving adequate yearly progress (AYP). • Some of this behavior is desirable: • Searching for and implementing evidence-based practices, • Focus on effectively teaching core subject matter. • Some of this behavior is undesirable: • Implementing narrow test-preparation programs, • Excluding lower performing students, • Reduced focus on important outcomes that do not affect AYP.

11. Contingencies and Evidence-Based Practices • When we talk about promoting an evidence-based culture in special education, we must consider contingencies. • Contingencies that demand effectiveness provide a context for an evidence-based culture. • With such contingencies, many of the most serious barriers to implementation of evidence-based practices are reduced or eliminated. • In the absence of such contingencies, an evidence-based culture is likely to be less relevant. • Effective implementation of evidence-based practices requires substantial effort from many players. • In the absence of strong contingencies, this kind of effort is unlikely. • When we think about each part of the Wing model of evidence-based education, we should consider the contingencies for key actors at each point.

12. Alternate Assessment

13. Contingencies and Alternate Assessment • NCLB is designed to include all students in a school’s AYP – and that is a very good thing. • Given the strength of NCLB’s contingencies, behavior that is not relevant to meeting its demands is likely to be reduced in value. • Exclusion from contributing to AYP does not just imply avoiding the pain and hassle of test-taking. • Exclusion makes the student’s educational progress irrelevant to the powerful contingencies of NCLB. • Those who are not counted, don’t count. • NCLB includes a provision specifying that students with significant cognitive disabilities (SCD) take an alternate assessment that will contribute to AYP.

14. Utah’s Alternate Assessment • We (Karen Hager and I) became interested in whether an alternate assessment could be constructed that would take advantage of the strong contingencies of NCLB to: • motivate effective educational practices and • improve educational outcomes of students with SCD. • We worked with the State of Utah to attempt to construct an effective alternate assessment. • We realized that validity of the alternate assessment would be critical. • Validity is a technical term for the degree to which an assessment successfully accomplishes the purposes for which it is being used. • Validity is critical for producing intended outcomes and avoiding unintended outcomes from contingencies.

15. Importance of Validity • The contingency of NCLB is based on test scores, not important student learning per se. • To the degree that alternate assessment results are valid, the contingencies of NCLB could have very positive impacts. • To the degree that alternate assessment results are invalid, the contingencies of NCLB could be very detrimental.

16. Defining the Construct We began with the question, “What are we trying to measure?” • Federal mandate – AA must assess: • Domains of Language Arts and Math forstudents with SCD. • Our values – AA should assess: • Skills that will contribute to independence and effectiveness of students with SCD. • The construct that drove the development of Utah’s Alternate Assessment: • Language Arts and Math skills that will contribute to students’ independence and effectiveness for students with SCD.

17. Sources of Invalidity • There are 2 main sources of invalidity: • Construct-Irrelevant Variance – the assessment may be influenced by factors that are not part of the target construct. • May result in strengthening the wrong behaviors. • Construct Under-representation – the assessment may ignore some important component of the construct. • May result in failing to strengthen some appropriate behaviors.

18. Construct Irrelevant Variance • Sources of Construct Irrelevant Variance • Direct assessment of factors other than student behavior • E.g., degree of inclusion, attendance, and other issues • Assessment of skills in such a way that outcomes are heavily influenced by how the teacher documents student behavior. • E.g., some forms of portfolio assessment • Inclusion of skills that are not important. • E.g., academic skills that are not functional • Problems with administration • E.g., Low fidelity of implementation and low reliability • Trivial tasks • E.g., assessments on which all students will be “proficient” no matter what their skills.

19. Construct Irrelevant Variance • Implications of Construct Irrelevant Variance • These are sources of unintended effects of the contingency • The contingency may strengthen behaviors relevant to these other factors at the expense of actually improving the learning of important skills.

20. Construct Under-representation • Sources of construct under-representation • Some important skills are not assessed. • E.g., actual differences in critical communication skills do not impact test results • Implications of under-representation • The powerful contingencies of NCLB may reduce focus on these important skills.

21. Construct Under-representation • NCLB appears to be very powerful in changing behavior of teachers and administrators. • If this power is to benefit students with significant cognitive disabilities, Alternate Assessments must be highly valid. • That is, they must reflect actual student performance on important tasks and must exclude other factors as much as possible.

22. Design of Utah’s Alternate Assessment

23. Standardization among diversity • The greatest challenge in designing an alternate assessment is to: • Produce meaningful, standardized outcomes • For a highly diverse group of students. • This requires a balance of standardization and individualization.

24. Standardization in UAA • Standardization is necessary to enable those not directly involved in the administration to interpret scores. • Standardized bank of tasks • UAA includes approximately 115 tasks in language arts and math • Standardized administration procedures • Each task requires direct observations of student performance • Tasks are to be embedded in naturally occurring routines in natural settings • Any communication system or assistive technology the student typically uses is acceptable during the assessment

25. Standardization in UAA • Performance criteria standardized for each task: • Performance criteria are designed to reflect a “functional” level of performance, i.e., a level that makes the skill meaningful/useful • Students perform the task independently (e.g., no physical, verbal, or gesture assistance) • Measurement tasks specify assessment of generalization: Three performances in situations that vary in relevant variables (e.g., people, settings, materials)

26. Individualization in UAA • Individualization is necessary to ensure relevance to specific students. • IEP team selects targets from list of standardized tasks • IEP team selects content on some tasks • E.g., specific sight words to be assessed

27. Individualization in UAA • IEP team adapts the task to some individual circumstances • IEP team specifies the natural routine during which to assess • IEP team selects appropriate generalization settings • E.g., particular individuals, settings, materials

28. Defining Adequate Yearly Progress • AYP for general education students • Standards set by grade level • AYP for students with SCD • Grade level not relevant basis for defining standards • Tasks that define AYP must be identified individually. • In UAA, IEP team selects relevant tasks from task bank. • Given a relevant task, performance level necessary to declare AYP is standardized.

29. Example UAA Task Description

30. Example UAA Task Description

31. Example UAA Task Description

32. Example UAA Task Description

33. Example UAA Task Record

34. Example UAA Task Record

35. Example UAA Task Record

36. Accessing UAA • The entire manual for Utah’s Alternate Assessment and the Technical Manual can be found at: • http://www.usoe.k12.ut.us/sars/Data.htm