HSAG Performance Improvement Training Statistical Testing

1. HSAG Performance Improvement Training Statistical Testing Presented by Donald Grostic, MS Health Services Advisory Group, Inc. January 11, 2011

2. Agenda • Background • Why do statistical testing? • When is statistical testing used? • Which PIP Activities and Evaluation Elements does statistical testing apply? • What statistical test do I use? • Chi-square test of inference • How is the numerator and denominator for the rate of home-delivered meals within 3 days used in a Chi-square test? • Website recommendation for Chi-square test • How do I interpret the results of the Chi-square test? • Question and answer session

3. Background • Why do statistical Testing? • The CMS protocols require that statistical testing be used to prove that any improvement in rates is real. • Without statistical testing, a PIP would not meet the CMS protocols.

4. Background (continued) • When is statistical testing used? • After collecting annual baseline data, completing a causal barrier analysis, implemented interventions and collecting an annual first remeasurement. • At this point, statistical testing should be completed to compare the annual baseline rate to the annual first remeasurement rate. • Statistical testing would need to be performed again between the annual remeasurement 1 rate and the annual remeasurement 2 rate after a causal barrier analysis is completed.

5. Background (continued) • Which PIP Activities and Evaluation Elements include statistical testing? • Activity VIII Evaluation Element 7: Identifies statistical differences between initial measurement and remeasurement. • Activity IX Evaluation Element 4: There is statistical evidence that observed improvement is true improvement. • Activity X Evaluation Element 1: Repeated measurements over comparable time periods demonstrate sustained improvement, or that a decline in improvement is not statistically significant.

6. Chi-square Test • What statistical test should I use? • For rates that are calculated using a numerator and denominator, the Chi-square test, Fisher’s exact test or the Z test for proportions should be used used. • For example if the home-delivered meals within 3 days measure for the baseline year had 20 clients receiving meals within 3 days out of 42 total clients, the rate would be 20/42 = 0.47x100 = 47.6 %. • If in the annual first remeasurement had 60 clients receiving meals within 3 days out of 78 total clients, the rate would be 60/78 = 0.76.9x100 = 76.9%.

7. Chi-square Test (continued) • How is the numerator and denominator for the rate of home-delivered meals within 3 days used in a Chi-square test? Hypothetical data showing Chi-square test Time Frame(X) HD Meals 3 Days(Y)Baseline R1 Total Yes (numerator) 20 60 80 No 22 18 40 Total (denominator) 42 78120

8. Chi-square Test (continued) • Website recommendation for a Chi-square test. • http://faculty.vassar.edu/lowry/tab2x2.html Time Frame (X) HD Meals 3 Days(Y)Baseline R1 Total Yes (numerator) 20 60 80 No 22 18 40 Total (denominator) 42 78 120 Note: Only enter numbers with arrows, the remaining numbers are calculated by the website.

9. Chi-square Test (continued) • Chi-square value = 10.55 • P-value = 0.001162

10. Chi-square Test (continued) • How do I interpret the results of the statistical test? • If the p-value is less than 0.05, then the change in rates is statistically significant. • Since the p-value is equal to 0.001162, which is less than 0.05, there is a statistically significant improvement in the rate of home-delivered meals within 3 days between the baseline rate (47.6%) and first remeasurement rate (76.9%).

11. Questions ?