1 / 38

Predictors of customer perceived software quality

Predictors of customer perceived software quality. Paul Luo Li (ISRI – CMU) Audris Mockus (Avaya Research) Ping Zhang (Avaya Research). Need to View Quality from the Customer’s Perspective. … We translate these advanced technologies into value for our customers …

bona
Download Presentation

Predictors of customer perceived software quality

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Predictors of customer perceived software quality Paul Luo Li (ISRI – CMU) Audris Mockus (Avaya Research) Ping Zhang (Avaya Research)

  2. Need to View Quality from the Customer’s Perspective … We translate these advanced technologies into value for our customers … -IBM (#9 on the Fortune 500) … Our strategy is to offer products, services and solutions that are high tech, low cost and deliver the best customer experience. -HP (#11 on the Fortune 500) • … We deliver unparalleled value to our customers. • Only by serving our customers well do we justify our existence as a business • -Avaya (#401 on the Fortune 500)

  3. What Would be Ideal • Predict customer perceived quality • Using customer characteristics • For each customer Key idea: Focus on the customer

  4. How do I plan deployment to meet the quality expectations of the customer? How do I target improvement efforts? How do I allocate the right resources to deal with customer problems Predict customer experience for each customer Identify possible causes of problems Predict customer interactions Possible Applications of Predictions

  5. How do I plan deployment to meet the quality expectations of the customer? How do I target improvement efforts? How do I allocate the right resources to deal with customer problems Predict customer experience for each customer Identify possible causes of problems Predict customer interactions Solutions for Software Producers

  6. How do I plan deployment to meet the quality expectations of the customer? How do I target improvement efforts? How do I allocate the right resources to deal with customer problems Predict customer experience for each customer Identify possible causes of problems Predict customer interactions To Improve Customer Perceived Quality

  7. Gaps in Current Research • Prior work examined: • Software defect prediction for a single customer (Musa et al. 1987, Lyu et al. 1996) • Software defect prediction for modules or features (Jones et al. 1999, Khoshgoftaar et al. 1996) Is not scalable

  8. Not Focused on Customers • Prior work examined: • Software defect prediction for a single customer (Musa et al. 1987, Lyu et al. 1996) • Software defect prediction for modules or features (Jones et al. 1999, Khoshgoftaar et al. 1996) Tell us nothing about a specific customer

  9. Does not Capture other Aspects of Customer Perceived Quality • Prior work examined: • Software defect prediction for a single customer (Musa et al. 1987, Lyu et al. 1996) • Software defect prediction for modules or features (Jones et al. 1999, Khoshgoftaar et al. 1996) Does not predict other aspects of customer perceived quality that are not code related.

  10. Research Contributions • Predict software defects for each customer in a cost effective manner • Predict other aspects of customer perceived quality for each customer • Empirically validate deployment, usage, software, and hardware predictors

  11. Rest of This Talk • The setting • Customer interactions (outputs) • Customer characteristics (inputs) • Results • Conclusion

  12. Empirical Results from a Real World Software System • Avaya telephone call processing software system • 7 million+ lines of C/C++ • Fixed release schedule • Process improvement efforts • Tens of thousands of customers • 90% of Fortune 500 companies use it • Professional support organization

  13. Data Used are Commonly Available • Customer issue tracking system • Trouble ticket database • The equipment database • Change management • Sablime database Data sources available at other organizations e.g. IBM and HP Data collected as a part of everyday operations

  14. At Other Organizations • Customer issue tracking system • Trouble ticket database • The equipment database • Change management • Sablime database Data sources available at other organizations e.g. IBM and HP Data collected as a part of everyday operations

  15. Customer Interactions (Outputs) • We assume customer interaction == customer perceived quality • Five customer interaction (Chulani et al. 2001, Buckley and Chillarege 1995) within 3 month of deployment • Software defects: high impact problem • System outages: high impact problem • Technician dispatches • Calls • Automated alarms Important for Avaya and likely for other organizations as well

  16. Examine Customer Installations Number of deployments 5 1 Months after general availability

  17. Capture Characteristics of Each Installation Customer 1: Deployed first month, a Large system, Linux… Customer 2: Deployed first month, a Small system, Windows… Customer 3: Deployed first month, a Large system, Proprietary Os… Customer 4: Deployed first month, a Small system, Linux… Customer 5: Deployed first month, a Large system, Linux… Number of deployments 5 1 Months after general availability

  18. Analyze Using Statistical Analysis Customer 1: Deployed first month, a Large system, Linux… Customer 2: Deployed first month, a Small system, Windows… Customer 3: Deployed first month, a Large system, Proprietary Os… Customer 4: Deployed first month, a Small system, Linux… Customer 5: Deployed first month, a Large system, Linux… Number of deployments 5 Similarities Differences 1 Months after general availability

  19. Category of Predictors (Kinds of Inputs) • We examine: • Deployment issues • Usage patterns • Software platform • Hardware configurations • Prior work examines: • Software product • Development process Common sense issues, but lack empirical validation

  20. Category of Predictors (Kinds of Inputs) • We examine: • Deployment issues • Usage patterns • Software platform • Hardware configurations • Prior work examines: • Software product • Development process Key idea: From the customer’s perspective, they are not good predictors (i.e. do not vary for a single release)

  21. Specific Predictors (Inputs) • Total deployment time • deployment issues • Operating system • software platform, hardware configurations • System size • hardware configurations, software platform, usage patterns • Ports • usage pattern, hardware configurations • Software upgrades • deployment issue

  22. Predict for each customer (outputs): Software defects System outages Technician dispatches Calls Automated alarms Using Logistic regression and Linear regression Using predictors (inputs): Total deployment time Operating system System size Ports Software upgrades For a real world software system Recap

  23. Example: Field Defect Predictions

  24. Predictors

  25. Nuisance Variables

  26. All Predictors are Important

  27. The Most Important Predictor • Total deployment time (deployment issue) • Systems deployed half way into our observational period are 13 to 25 times less likely to experience a software defect

  28. May Enable Deployment Adjustments • Total deployment time (deployment issue) • Systems deployed half way into our observational period are 13 to 25 times less likely to experience a software defect • May be due to software patching, better tools, more experienced technicians

  29. Another Important Predictor • Total deployment time (deployment issue) • Operating system (software platform, hardware configurations) • Systems running on the proprietary OS are 3 times less likely to experience a software defect compared with systems on running the open OS (Linux) • Systems running on the commercial OS (Windows) are 3 times more likely to experience a software defect compared with systems running on the open OS (Linux)

  30. May Allow for Targeted Improvement or Improved Testing • Total deployment time (deployment issue) • Operating system (software platform, hardware configurations) • May be due to familiarity with the operating system • May be due to operating system complexity

  31. More Results in Paper • The complete results and analyses for field defects • Predictions for other customer interactions

  32. Validation of Results and Method • We accounted for data reporting differences • Included indicator variables in the models to identify populations (e.g. US or international customers) • We independently validated the data collection process • Independently extracted data and performed analyses • We interviewed personnel to validate findings • Programmers • Field technicians

  33. Summary: Identified Predictors of Customer Perceived Quality • We identified and quantified characteristics, like time of deployment, that can affect customer perceived quality by more than an order of magnitude

  34. Summary: Modeled Customer Interactions • We identified and quantified characteristics , like time of deployment, that can affect customer perceived quality by more than an order of magnitude • We created models that can predict various customer interactions and found that predictors have consistent effect across interactions

  35. Summary: Deployment is Important for High Reliability • We identified and quantified characteristics , like time of deployment, that can affect customer perceived quality by more than an order of magnitude • We created models that can predict various customer interactions and found that predictors have consistent effect across interactions • We learned that controlled deployment may be the key for high reliability systems

  36. Improve Customer’s Experiences • You can target improvement efforts • You can allocate the right resources to deal with customer reported problems • You can adjust deployment to meet the quality expectations of your customers

  37. Predictors of customer perceived software quality Paul Luo Li (paul.li@cs.cmu.edu) Audris Mockus (Avaya Research) Ping Zhang (Avaya Research)

  38. Predicted Number of Calls Match Actual Number of Calls Calls for the next release Calls Predictions are made here Time

More Related