1 / 19

Root Cause Analysis of Achievement Test Data: An Introduction

Root Cause Analysis of Achievement Test Data: An Introduction. Kim Kappler Hewitt, Ph.D. What is “root cause analysis” (RCA)?. RCA is the practice of identifying the factors that cause or contribute to an outcome. Why do Root Cause Analysis?.

qiana
Download Presentation

Root Cause Analysis of Achievement Test Data: An Introduction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Root Cause Analysis of Achievement Test Data:An Introduction Kim Kappler Hewitt, Ph.D.

  2. What is “root cause analysis” (RCA)? RCA is the practice of identifying the factors that cause or contribute to an outcome.

  3. Why do Root Cause Analysis? • By addressing the symptoms of a problem and not the causes, we will not see the change we want. • Example: Taking over-the-counter pain medicine for a broken wrist (instead of having the break addressed). • If we make assumptions about the cause(s) instead of rigorously and systematically investigating and considering the true causes, we risk making the wrong changes that may have no effect on the problem or--worse yet--increase the problem. • Example: Assuming my plant is wilting due to lack of water, watering it thoroughly for two weeks, and noting that it has gotten worse (real issue was soil acidity).

  4. RCA Basic Questions • Four basic questions guide the RCA process: • Identify the focus: What happened (identify the outcome that is the focus of the RCA)? • Identify the cause(s): Why did it happen? (This is the heart of the RCA process.) • Develop a plan: What will be done to change the outcome (prevent the outcome from recurring)? • Monitor the plan: How will we monitor change and determine the success of the change?

  5. RCA can be… • Messy: Rarely is RCA as simple as 2 + 2 = 4. There is often no clear, obvious cause that immediately jumps out OR there are many possible cause(s) such that it is difficult to identify the meaningful one(s). • Tedious: One must ask “why” and then to each answer to that question ask “why” again and again until the fundamental cause(s) are identified. • Uncertain: You may not know “for sure” that you have identified the accurate root cause(s). Work to find the most compelling explanation(s) for the outcome. • Imperfect: Even a carefully executed RCA may not lead to the desired change in effect.

  6. Symptom/assumption Approach Female students aren’t as interested in science. Male students always do better on the science test than females. We see this pattern in other buildings in our district as well. This is how students are wired. We can’t change this. Root Cause Approach Problem: Achievement gap by gender in science. Identify root cause(s): Look at various data, get familiar with the research/literature on the issue, and pose hypotheses regarding the cause(s). Develop a plan to address the cause(s). Monitor implementation of the plan. Symptom v. Root Cause Approaches Outcome: Male student proficiency rates on the 8th grade science test are 12 percentage points higher than female students’. Additionally, 18% of males scored “advanced” on the science test, while 3% of females did.

  7. Sample questions to apply RCA to achievement data • Is there adequate alignment between the state standards and what is taught? • Is there adequate alignment between what is taught and what is tested (and how it is tested)? • Is there adequate alignment between the state standards and instructional materials? • Do teachers have adequate content knowledge? • Are teachers using effective pedagogy? Assessment practices? • Are there adequate resources for struggling students? • Are there adequate resources for students who need enrichment/extension? • Are content, process, and product being differentiated effectively? • Are students engaged? • Are there equity issues (associated with achievement gaps)? • Are there issues of student attendance and/or mobility?

  8. RCA Steps • Step 1: Identify the focus: What happened (identify the outcome that is the focus of the RCA)? • Be specific. Focus the problem/issue as much as possible. • Ask clarifying questions, as needed.

  9. RCA Steps • Step 2: Identify the cause(s): Why did it happen? (This is the heart of the RCA process.) • Brain-storm possible causes. At this point, list all manner of possible causes (do not evaluate or eliminate any). • Consider what data can confirm/disconfirm each possible cause; then collect and analyze that data to eliminate possible causes that are not supported by the data. • Identify the cause(s) that is the most compelling explanation, based on all available data.

  10. RCA Steps • Develop a plan: What will be done to change the outcome (prevent the outcome from recurring)? • Include: • steps to take • a timeline • resources that will be needed • person/people responsible for each step.

  11. RCA Steps • Step 4: Monitor the plan: How will we monitor change and determine the success of the change? • Detail the following (indicating the timeline for each and assigning a person/people responsible for each): • What assessments/data will be used to determine if the plan is being implemented with fidelity? • What assessments/data will be used to determine if the plan is being being effective at addressing the root causes? • What assessments/data will be used to determine if the plan is being successful at changing the outcome?

  12. Let’s do a sample problem:Outcome: Male student proficiency rates on the 8th grade science test are 12 percentage points higher than female students’. Additionally, 18% of males scored “advanced” on the science test, while 3% of females did.

  13. RCA Steps • Step 1: Identify the focus: What happened (identify the outcome that is the focus of the RCA)? • Be specific: Male student proficiency rates on the 8th grade science test are 12 percentage points higher than female students’. Additionally, 18% of males scored “advanced” on the science test, while 3% of females did. • Ask clarifying questions: • How enduring is this pattern (for how many years has this achievement gap been in place)? • Is this pattern occurring across classroom? • Is this pattern occurring across subgroups of female students (Normally high-performing students? Students of various races/SES backgrounds?)? • Is this pattern occurring across science standards (i.e., are there some science standards on which female students are performing similarly to male students, such as life science of scientific ways of knowing)? • Is this pattern occurring across grade levels (i.e., What does the 5th grade science data say? Is this a phenomenon specific to 8th grade, or is it something that may be starting earlier in students’ schooling and continuing or compounding in the 8th grade?

  14. Clarifying questions answered. RCA Steps • Step 1: Identify the focus: What happened (identify the outcome that is the focus of the RCA)? • Be specific: Male student proficiency rates on the 8th grade science test are 12 percentage points higher than female students’. Additionally, 18% of males scored “advanced” on the science test, while 3% of females did. • Ask clarifying questions: • How enduring is this pattern (for how many years has this achievement gap been in place)? >4 years • Is this pattern occurring across classroom? Yes, but more prevalent in classrooms with male teachers. • Is this pattern occurring across subgroups of female students (Normally high-performing students? Students of various races/SES backgrounds?)? Yes, but to varying degrees (low SES and students of “average ability” most affected). • Is this pattern occurring across science standards (i.e., are there some science standards on which female students are performing similarly to male students, such as life science of scientific ways of knowing)? Yes, across standards. • Is this pattern occurring across grade levels (i.e., What does the 5th grade science data say? Is this a phenomenon specific to 8th grade, or is it something that may be starting earlier in students’ schooling and continuing or compounding in the 8th grade? Yes, but more pronounced in 8th.

  15. RCA Steps • Step 2: Identify the cause(s): Why did it happen? (This is the heart of the RCA process.) • Brain-storm possible causes. At this point, list all manner of possible causes (do not evaluate or eliminate any). • Consider what data can confirm/disconfirm each possible cause; then collect and analyze that data to eliminate possible causes that are not supported by the data. • Identify the cause(s) that is the most compelling explanation, based on all available data.

  16. Main Causes RCA Steps • Step 2: Identify the cause(s): Why did it happen? (This is the heart of the RCA process.) • Brain-storm possible causes. At this point, list all manner of possible causes (do not evaluate or eliminate any). • Consider what data can confirm/disconfirm each possible cause; then collect and analyze that data to eliminate possible causes that are not supported by the data. • Identify the cause(s) that is the most compelling explanation, based on all available data.

  17. RCA Steps • Develop a plan: What will be done to change the outcome (prevent the outcome from recurring)? • Include: • steps to take • a timeline • resources that will be needed • person/people responsible for each step.

  18. RCA Steps • Step 4: Monitor the plan: How will we monitor change and determine the success of the change? • Detail the following (indicating the timeline for each and assigning a person/people responsible for each): • What assessments/data will be used to determine if the plan is being implemented with fidelity? • What assessments/data will be used to determine if the plan is being being effective at addressing the root causes? • What assessments/data will be used to determine if the plan is being successful at changing the outcome?

  19. Tools • Chart template: http://www.mindtools.com/pages/article/worksheets/RCAtemplate.pdf • Fishbone diagram template: http://www.isixsigma.com/index.php?option=com_k2&view=item&id=1416:the-cause-and-effect-aka-fishbone-diagram&Itemid=200 • Appreciation: Keep asking, “So what?” in response to facts. See http://www.mindtools.com/pages/article/newTMC_01.htm. • 5 Whys: http://www.mindtools.com/pages/article/newTMC_5W.htm • Drill down: http://www.mindtools.com/pages/article/newTMC_02.htm

More Related