Performance evaluation, Capability Analysis and monitoring with Low Level Count data

1. Erwin M. Saniga Dept. of Bus. Admin. University of Delaware Newark, DE 19716 302-831-2555 sanigae@be.udel.edu James M. Lucas J.M. Lucas and Associates 302-368-1214 5120 New Kent Road Wilmington, DE 19808 Jamesm.lucas@worldnet.att.net Darwin J. Davis Dept. of Bus. Admin. University of Delaware Newark, DE 19716 302-831-2555 davisd@be.udel.ed Presenter: Erwin Saniga Performance evaluation, Capability Analysis and monitoring with Low Level Count data

2. Bank Vendor Vendor A D Vendor Vendor B C Purpose: To examine the performance of alternative methods to study processes where quality is measured by counts and counts are low. Example • Large Wilmington (DE) area credit card bank • Processes credit card applications • Four vendors process these applications • Wish to implement a vendor certification and quality improvement program

3. We will show in this presentation: • When exploring this type of data for performance evaluation or process capability analysis, four different types of plots can reveal different things about the process • A traditional sequence plot • Adding a Shewhart UCL and a method to detect improvements to the traditional sequence plot • A CUSUM plot • Adding a “V-mask” to the CUSUM plot

4. Actual results for the four vendors • Each point represents the number of defectives resulting from an inspection of 50 random credit card applications that were processed during the day. These were taken at the end of the day. • Each credit card application can be processed correctly or incorrectly.

5. Average vendor performance Vendor A average p = 0.0246 Vendor B average p = 0.0510 Vendor C average p = 0.0247 Vendor D average p = 0.0294 Analyst questions: What caused the spikes at various points in time for each vendor? What caused the sequence of “good” (zero defective) samples for various vendors? Why is Vendor B doing poorly when compared to Vendors A, C, and D? Are these substantive differences? If the data were available in real time and we could plan the data collection we might: Investigate the cause of a spike or run of good points immediately Keep a diary or log of variables identified during a focus group meeting of employees, managers, etc. Vendor Analysis

6. CUSUM SEQUENCE (DIAGNOSTIC) PLOTS • Let Xj = the actual number of defectives observed in the jth sample • The ith CUSUM is then: • where k is the reference value. • We use a reference value of k=1.25 which is the average count of defectives in a sample of 50 for the three “good” vendors.

7. Process Averages for CUSUM Sequence Plot • The CUSUM sequence plot can identify “good”, “average” or “bad” regimes. • Regime average is determined by the slope (in this case, slope = 0 implies 1.25 defects). • The average count from periods L to M is given by: • Vendor A Example Regime 1 – Days 46 to 122 Ave. count = 1.25 + [-27.5 – (-0.25)]/[122 – 46 + 1] = 0.896 Regime 2 – Days 176 to 199 Ave. count = 1.25 + (6.25 – (-21.75)/(199 – 176 + 1) = 2.417

8. CUSUM Sequence Plot Summary • There are various “regimes’ noted by the CUSUM sequence plots that are not immediately recognizable from the traditional sequence plot of the counts. • Some of these regimes indicate notable “good” or “bad” performance. CUSUM sequence plot questions: • What are the reasons for the good performance in certain regimes? • What are the reasons for the bad performance in certain regimes? • What happened on the particular days a change point was observed? If the data were available in real time we might keep a diary or log of variables possibly associated with performance and investigate these.

9. Vendor A Comparison A comparison of the CUSUM sequence plot with the traditional sequence plot Traditional sequence plot: spikes at 35, 128, 157, 169, 193, and 197 zero counts from 200 to 207 consecutively CUSUM sequence plot: “Average” process on days 1-14 “Good” process on days 15 to 32 “Bad” process on days 33 to 45 “Good” process on days 46 to 122 “Average” process on days 123 to 175 “Bad” process on days 176 to 199 “Good” process on days 200 to 207

10. Summary of plots • Traditional sequence plots are good for detecting shocks to the system and rare events(transient shifts) • CUSUM sequence plots are good for detecting regimes or periods of good, bad, and average behavior(persistent shifts)

11. Other Questions For traditional sequence plots: • Are the spikes unusual when compared to what might happen under pure chance? • Are the zero count sequences unusual in the sense that they indicate the process has improved? For CUSUM sequence plots: • Are the regimes we observed unusual when compared to what might happen under pure chance?

12. Traditional Sequence Plot • Shewhart chart provides a simple and effective way to “signal” spikes as being significantly unusual. • Control Limits for the Shewhart chart: where p is the average proportion defective.

13. The Upper Control Limit • For our data with an average count of 1.25 (p=0.025, n=50) we have UCL =4.56 (the Shewhart 3 sigma limit) • In our examples we use UCL = 5 (signal at 6) to ensure the ARL in control is sufficiently large. For UCL= 4 (signal at 5) ARL in control = 123 For UCL= 5 (signal at 6) ARL in control = 662 • (Generally, for low count data adding 1 to the UCL yields a more desirable in control ARL)

14. The Lower Control Limit • LCL = -2.06 (no lower control limit) Solutions for the lower side (detecting improvement) • Ryan(1989) and Schwertman and Ryan(1997) suggest equal tail probability methods. • These do not work well when P(0) is high: In control ARL is low or, equivalently, false alarm probabilities are high. • Acosta-Mejia (1999) suggests counting successive results below a modified centerline • Dominated by CUSUM methods in terms of ARL. Does not work well for P(0) large. • CUSUM methods (e.g. Reynolds and Stoumbos (1999, 2000) • Optimal for a particular shift in terms of ARL yields • Harder to design and use than special CUSUMs we will provide.

15. Count the Zeros: Special CUSUMs The simplest low side CUSUMs Method 1: Signal if k in a row samples have zero defectives Method 2: Signal if 2 in t samples have zero defectives Properties: • Optimal for large shifts • Easy to use • Easy to design • Easy to understand

16. Low Side Schemes • Comparison of our special CUSUM (k in a row) and the CUSUM.

17. The Computational CUSUM yi = the observed count For the high side Si = Max {0, Si-1 - kH + yi } If Si > hH in a signal is given For the low side Si = Max {0, Si-1 + kL - yi } If Si > hL in a signal is given For our example we use: kH = 2 hH = 4 kL = 0.75 hL = 5.25

18. Asymmetric V-Mask Slope = kL – reference value hL Last Cusum Value hH Slope = kH – reference value

19. Comparison of Signals

20. Comparison of Signals

21. We have shown… • Four different types of plots can reveal different things about a process • A traditional sequence plot • Adding a Shewhart UCL and a method to detect improvements to the traditional sequence plot • A CUSUM plot • Adding a “V-mask” to the CUSUM plot