1 / 67

Making Data Work for Title I Directors

Making Data Work for Title I Directors. Using Achievement Data to Effectively Inform Both Targeted Assistance and Schoolwide Programs 9 th Annual Title Programs Conference June 14 – 17, 2011. Agenda . Title I Evaluation Requirements Purpose of Evaluation Understanding Assessment

mercury
Download Presentation

Making Data Work for Title I Directors

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Making Data Work for Title I Directors Using Achievement Data to Effectively Inform Both Targeted Assistance and Schoolwide Programs 9th Annual Title Programs Conference June 14 – 17, 2011

  2. Agenda • Title I Evaluation Requirements • Purpose of Evaluation • Understanding Assessment • Detecting Differences • Measuring Outcomes • Relationships Between Variables • Practical Applications for Schoolwide and Targeted Assistance Programs • Questions

  3. How Does Title I Measure Up? Federal government program assessment

  4. Federal Government Program Assessment • Program Assessment Rating Tool (PART)* • 25 questions to measure effectiveness of all federally funded programs on the following: • Clarity of purpose and design • Strategic planning • Management • Results • Classifies programs as Effective, Moderately Effective, Adequate, or Ineffective *Information obtained from ExpectMore.gov

  5. Federal Government Program Assessment *Data obtained from ExpectMore.gov

  6. Federal Government Program Assessment • Title I Results: • Moderately Effective* • Ambitious goals • Well-managed • Likely need to improve efficiency or address other problems in program’s design or management • http://www.whitehouse.gov/omb/expectmore/summary/10003320.2006.html *Information obtained from ExpectMore.gov

  7. Data and Assessment Requirements for Title I Title I evaluation requirements

  8. Title I Data and Assessment Requirements • Comprehensive Needs Assessment • Must be based on academic achievement information about all students in the school, including: • Racial and ethnic groups • Children with disabilities • Children with limited English proficiency • Children with low incomes • Migrant students • Information must reflect achievement relative to the state standards

  9. Title I Data and Assessment Requirements • Comprehensive Needs Assessment • Help the school understand the subjects and skills for which teaching and learning need to be improved • Identify specific academic needs of students and groups of students who are not yet meeting state standards

  10. Title I Data and Assessment Requirements • Evaluation • Schools operating schoolwide programs must annually evaluate the implementation of and results achieved by Title I programs using data from state assessments and other indicators of academic achievement. • Determine whether the program has increased achievement of students, particularly students who had been farthest from achieving standards • Results should guide any revisions to the program plan

  11. Title I Data and Assessment Requirements • Evaluation of Program Components: • Schoolwide reform strategies • Instruction by highly qualified teachers • Parental Involvement • Additional support • Transition plans

  12. Purpose of Evaluation • Accountability • Objectivity • Comparability • Decision-Making tool • Allocation of resources • Determine areas of need • Measure progress

  13. Defining Constructs, Test Theory, and Measurement Properties Understanding assessment

  14. Understanding Assessment • Operationally defining constructs of interest • Academic Achievement • Intelligence • Attitudes toward school • Parental Involvement • Attendance • Disabilities/Learning Disorders • Language ability • Access to services

  15. Understanding Assessment • Commonly used approaches to measurement: • Standardized tests • IQ tests • Surveys • Checklists • Rating scales • Structured interviews • Count variables

  16. Understanding Assessment • Test Theory • Complete/Perfect measurement is not possible • Items are drawn from an infinite pool that represents complete information for any construct • Collection of items provides an observed score • TRUE score = observed score + error • Goal of assessment: systematically minimize error to consistently detect meaningful measures that are as close to “True” score as possible

  17. Understanding Assessment • Test Theory • Items should discriminate; Item Characteristic Curve (ICC)-plots ability vs. probability of correct response

  18. Understanding Assessment • Reliability • Degree of consistency of measurement • Validity • Degree of accuracy & representativeness of measurement • Instrument can have high reliability but low validity • Valid instrument must be reliable

  19. Understanding Assessment • Sampling distributions • Each observed score is a sample statistic drawn from a distribution of multiple observations (actual or potential)

  20. Understanding Assessment • Confidence intervals • More accurate in terms of interpreting and communicating results

  21. Understanding Assessment • Interpreting results • Multiple measures always best • Keep the goal of testing in mind • External contributing factors (testing conditions) • Population considerations (culture, language, etc.) • Consider subscales when reported • Provide data on specific domains of interest • Useful for focused evaluation

  22. Understanding Assessment • Descriptive Statistics • Produce quantitative summaries of numbers • Describes a population or sample • Central tendency • Variability • Linearity/Non-linearity • Inferential Statistics • Hypothesis testing • Allows for prediction and examining real world relationships (cause & effect) • T-test • ANOVA • Linear modeling

  23. Evaluating Achievement Outcomes for Significant Differences Between Groups Detecting differences

  24. Detecting Differences • Why is it important to understand group differences? • Achievement gaps • Differences in availability and/or usability of resources • Title programs are aimed at reducing these gaps for academically at-risk students • Why is it important to use data to examine differences? • Objectivity • Consistency • Accuracy • Monitoring growth, improvement, changes over time

  25. Detecting Differences • Group differences in evaluation terms • Basic differences in groups or populations • T-test for statistical significance (compare group means for significant variability) • Differences in patterns and relationships • ANOVA, Linear tests (compare direction, association, and slope or rate of change in groups)

  26. Detecting Differences

  27. Detecting Differences • Using various data sources to determine meaningful differences • Multiple assessments, informants, data points over time, etc. • Provides a clear, overarching view • Compare measures to: • Validate your group comparisons • Negate or shed light on weaknesses from a single source

  28. Detecting Differences • Disaggregate • Slicing data to see what the picture looks like for different subgroups hidden within an average or basic percentage • Break down statistics into smaller components based on groups you are interested in comparing • Schools within a district • Teachers within a school • Race/ethnicity • ELL status • Disability status • Free/Reduced lunch status

  29. Detecting Differences • Disaggregate • Start with simple statistics: averages, percentages Average Score: 143 African American 121 Native American 132 Asian 154 Latino 127 White 160

  30. Detecting Differences • Cross-Tabluate • Dicing data to further examine group differences; allows for multiple group categories to be considered at once • Useful for demonstrating how an education system advantages/disadvantages different groups of students, and how the situation might be improved

  31. Detecting Differences Percent Passing 8th Grade Math Test =50%

  32. Monitoring the Effectiveness of Interventions Measuring outcomes

  33. Measuring Outcomes • Why is it important to measure outcomes? • Program evaluation • Improve student programs • Fiscal responsibility • Why is it important to use data to monitor outcomes? • Objectivity • Consistency • Accuracy • Monitoring growth, improvement, changes over time

  34. Measuring Outcomes • Measuring outcomes in evaluation terms • Basic pre-post comparison • T-test • Antecedent monitoring and process evaluation • Comparison of relationships between outcomes • Linear models • Do outcomes vary for different groups or students? • Reporting outcomes • Raw score change • Percentage change

  35. Measuring Outcomes

  36. Measuring Outcomes • Longitudinal Data • Looking at data over a period of time (weeks, months, years) • Observing multiple time points can point out important patterns that cannot be detected with one or two measurements

  37. Measuring Outcomes Student Achievement on State Tests: A Longitudinal Analysis

  38. Identifying Important Connections Relationships between variables

  39. Relationships Between Variables • Why is it important to understand relationships between variables? • Identify patterns • Knowledge about what to expect for students • Identify areas to target for improvement • How does data help us examine relationships between variables? • Measurements represent constructs of interest • Objective evidence to support claims and provide ideas • CANNOT identify causes of outcomes

  40. Relationships Between Variables • Relationships in Evaluation Terms • Scatterplots • Track scores or outcomes across levels of another variable to uncover connections • Allow us to examine real-world connections and understand relationships, form hypotheses • Allow us to visualize impact of cutoff scores and group criteria

  41. Relationships Between Variables Table/List Data Scatterplot

  42. Relationships Between Variables

  43. Relationships Between Variables • Important considerations • Type of variables (continuous, no groups or categories) • Scales of variables • Cutoffs • Context of analysis, potential outcomes or decisions • Limitations

  44. Using Data to Inform Title I Programs Practical applications

  45. Practical Applications • Example 1: Combining Data Sources to Identify Group Differences for Hispanic Students • Strategy: • Is there an achievement gap for Hispanic students in my system? Is it higher or lower than those at the state or national level? • Analyze data from various sources to provide support/rationale for your Title I plan • Multiple levels (local, state, national) • Multiple measures (different assessments) • Over time where available

  46. Practical Applications • Examine Longitudinal National Data • Large sample, over time • Trend in NAEP mathematics average scores for 9-year-old students, by race/ethnicity* *Data obtained from NAEP Long Term Trend Data

  47. Practical Applications • Trend in White – Hispanic NAEP mathematics average scores and score gaps for 9-year-old students* *Data obtained from NAEP Long Term Trend Data

  48. Practical Applications • State 4th Grade CRCT Scores – Percentage of Students at Each Performance Level: Comparison by Race/Ethnicity* • DNM • Meets • Exceeds *Data obtained from GaDOE 2009-10 State Report Card

  49. Practical Applications • Average Math ACT Score for H.S. Seniors by Subgroups at the State and National Levels • State • Nation *Data obtained from GaDOE 2009-10 State Report Card

  50. Practical Applications • LEA 4th Grade CRCT Scores – Percentage of Students at Each Performance Level: Comparison by Race/Ethnicity* • DNM • Meets • Exceeds *Data obtained from GaDOE 2009-10 LEA Report Card

More Related