Overview of measurement, assessment, and evaluation

1. Overview of measurement, assessment, and evaluation July 31, 2003 Department of Behavior Analysis

2. Today’s agenda • Welcome (5 minutes) • Please remember to click • Bonus for reminding me about Free/Writes • Questions & answers (5 minutes) • Student presentations (45 minutes) • Difficult to discriminate stimuli • Model concept analyses • Overview of Assessment Department of Behavior Analysis

3. Some terms… • Measurement – the gathering of quantitative information • Assessment – the pacing of information within a context • Evaluation – making a decision based on assessment information Department of Behavior Analysis

4. Two Main Types of Assessment: • Criterion referenced • Compares the student’s performance to a specified mastery level • Instructional objectives • Norm referenced • Compares the student’s performance on a task to a group of students • Age equivalent, grade equivalents, percentile ranks Department of Behavior Analysis

5. 3 Levels of Assessment: • Micro level • Assessment conducted on an on-going basis related to specific instructional objectives • Meta level • Assessment conducted on a regular basis related to broad skill clusters within curriculum areas • Macro level • Large-scale assessment to determine overall impact of educational intervention Department of Behavior Analysis

6. Intervals Corresponding to Levels of Assessment

7. Intervals Corresponding to Levels of Assessment Micro

8. Intervals Corresponding to Levels of Assessment Meta

9. Intervals Corresponding to Levels of Assessment Macro

10. The Micro Level of Assessment: • Every skill being established or practiced is monitored • Monitoring occurs each time a skill is taught • Collects data along the same dimension as the instructional objective • Is used to inform instructional decisions Department of Behavior Analysis

11. Quality Intervention requires: • Continuous MeasurementStudent responding to intervention practices is continuously measured. • Visual Displays- A relevant dimension of the behavior to be taught is graphically displayed.- Graphic displays (indicating the progress of the student) are always available for viewing.- Graphs are used to make sound decisions about tactics for individuals. Adapted from work by R.D. Greer Department of Behavior Analysis

12. Quality Intervention requires: • Responsiveness Intervention driven by the moment to moment responses of each individual student and guided by existing research findings. Adapted from work by R.D. Greer Department of Behavior Analysis

13. The Micro Level of assessment • Effects of instructional decisions are tracked relative to each skill • Monitoring continues until essential learning outcomes are demonstrated • Skill retention • Skill endurance • Skill application • Skill stability Department of Behavior Analysis

14. The Meta Level of Assessment: • Clusters of interrelated skills are assessed together • Assesses student’s performance on a range of skills within a curriculum area • E.g., reading, mathematics, social skills, language • Helps determine whether intervention is being effective • Performance is compared to typical peers • Also called Curriculum Based Assessment (CBA) Department of Behavior Analysis

15. To assess effectiveness of: Sounds instruction Words instruction We might measure the number of words read correctly per minute from prose samples To assess effectiveness of: Expressive labeling of objects instruction Spoken sentence structure (semantics) instruction We might measure the number of words and sentences spoken spontaneously during a five minute language sample Examples of CBA Measures: Department of Behavior Analysis

16. An Example:

17. Another Example:

18. Another Example:

24. The Macro Level of assessment: • Skill clusters are assessed together on complex tasks • Uses norm-referenced, standardized, published tests • e.g., Woodcock-Johnson, Preschool Language Scale-3, Photo Articulation Test, Child Development Inventory, etc. • Allows comparison of client progress relative to the norm group • Emphasis is placed on rate of progress • pre/post intervention • compared to typical peers Department of Behavior Analysis

25. An example of the importance of RATE of progress “Emma has made significant progress since her last evaluation in her motor skills. She has gone from an age equivalent of 30 months to 54 months in her gross motor development.” - School Psychologist, in a triennial evaluation report Did Emma progress as a result of educational intervention? Department of Behavior Analysis

26. E m m a - G r o s s M o t o r D e v e l o p m e n t 8 0 T y p i c a l C h i l d ' s P r o g r e s s ( 1 m o n t h p e r m o n t h ) 7 0 E m m a ' s P r o g r e s s - B a s e l i n e ( . 8 3 6 0 m o n t h s p e r m o n t h ) E m m a ' s P r o g r e s s - D T T + p r e s c h o o l 5 0 ( . 7 5 m o n t h s p e r m o n t h ) Performance Age Equivalent 4 0 3 0 2 0 1 0 0 4 1 7 22 25 28 40 43 46 49 61 64 67 70 73 10 13 16 19 52 55 58 31 34 37 C h r o n o l o g i c a l A g e B a t e l l e D e v e l o p m e n t a l I n v e n t o r y An example of why RATE of progress is so important...

27. Legend Age in Months Daniel's Progress - .76 months of gain per month Typical Child's Progress - 1 month gain per month Therapy Progress - 3.2 months of gain per month Daniel’s Expressive Language Progress with Intervention 50 45 40 35 30 Skill Age Equivalent (in Months) 25 20 15 10 5 0 5 7 9 1 3 11 17 19 25 27 37 39 13 15 21 23 29 31 33 35 41 Chronological Age (in Months) Preschool Language Scale - 3

28. 60 Legend Age in Months 50 Daniel's Baseline Progress - .63 months gain per month Typical Child's Progress - 1 month gain per 40 month Daniel's Therapy Progress - 4 months of gain in one month Skill Age Equivalent (in Months) 30 20 10 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 Chronological Age (in Months) Preschool Language Scale - 3 Daniel’s Auditory Comprehension Progress with Intervention

29. Daniel’s Articulation Progress with Intervention 1.12 Standard Deviations from Peers - 0.21 Pre-Therapy (11/00) Current (05/00) Photo Articulation Test

30. You should begin working on… • You will be responsible for completing a report related to assessment of children with autism. Your report is to follow APA style guidelines and should have the following features: • 1. Describe at least three published assessment tools used with children with autism. In your description, be sure to include descriptions of the skills and repertoires assessed by the tool, whether the tool is normatively or criterion referenced, a description of the referencing process used by the tool developers, and critical features of children for whom the tool would be appropriate (age, pre-requisite skills required, cultural background, etc.) • 2. For one of the assessment tools you describe, list the assessment items from the tool and identify the type of verbal operant tested (if the test items are, in fact, testing verbal behavior). For example, if a test item requires the student to answer the question, "Who lives in the Hundred Acre Wood?" by saying, "Winnie the Pooh", you would classify this as testing the intraverbal repertoire. NOTE: you will make me very happy if you also identify the type of intraverbal response tested (sequelic, codic, or duplic). As another example, if the test item asks the student to say "Winnie the Pooh" when shown a picture of Winnie the Pooh, you would classify this as testing the tact repertoire. If the test item is not testing verbal behavior, then categorize the item according to the taxonomy system you learn in class. Department of Behavior Analysis

31. Tonight’s readings… • White, O.R. (1984). Selected issues in program evaluation: Arguments for the individual. In B.K. Keogh (Ed.) Advances in Special Education: Documenting Program Impact. Greenwich, CT: JAI Press. • Vargas, E.A. (1988). Verbally-governed and event-governed behavior. The Analysis of Verbal Behavior, 6, 11-22. • Tiemann, P.W. & Markle, S.M. (1990). Principles (chapter 6). Analyzing Instructional Content. Champaign, IL: Stipes Publishing Company • Tiemann, P.W. & Markle, S.M. (1990). Strategies (chapter 7). Analyzing Instructional Content. Champaign, IL: Stipes Publishing Company • Tiemann, P.W. & Markle, S.M. (1990). Sets of related principles--some issue to consider (appendix 10). Analyzing Instructional Content. Champaign, IL: Stipes Publishing Company • Tiemann, P.W. & Markle, S.M. (1990). Guidelines for the analysis of a set of related principles (appendix 11). Analyzing Instructional Content. Champaign, IL: Stipes Publishing Company Department of Behavior Analysis

32. Tonight’s homework… • complete the in-text questions for Tiemann & Markle, chapters 6 and 7 (due 7/25/03) • identify four examples of instances of rule-governed behavior (due 7/28/03) that children with autism often need to learn from each of the following areas: • academic skills • social skills • self-help skills • leisure skills • of the examples you identified above, analyze one from each of the four areas using the example formats Tiemann & Makle provide in appendix 11 ;please be prepared to present at least one of these in class tomorrow Department of Behavior Analysis