Predictor of Customer Perceived Software Quality

1. Predictor of Customer Perceived Software Quality By Haroon Malik

2. Introduction • Predict the customer’s experience within first three months • Quantifying the relative importance of various processes and product factors on customer experience • Deployment issues • Usage patterns, • Software platforms & • Hardware platform.

3. Introduction • Come up with a model that can be easily adapted and used at other organization with little or more tailoring.

4. Driving force • Techniques already exists to predict how many faults remains in unchanging software system, changes or module will have defect and even how much effort defect repairs will require. • Many researches examined the effect of software contents and development process on measure of customer perceived quality.

5. Driving force • Most of which ignore the • Hardware configurations, • Software platform, • Usage patterns & • Deployment issues • End users experience the software typically experience the quality of the entire “Solution”. • Past researches do consider the importance of these factors but no solid work is done to validate the claim or quantify these factors.

6. The Software project • Call processing software for AVYA telephony systems • Established product • Seven million lines of code mostly in C and C++ • Multiple releases are in field and are being used by tens of thousands of customers. • Used by clients whose business depend on the high availability of the product.

7. Capturing Interaction • Four database are considered for capturing customer interaction measures • Customer issue tracking system • Trouble ticket database • The equipment database • Change management • Sablime database Database-1 Database-2

8. The Process • Avaya uses a tiered support process. • Trouble Ticket • Half of the over 4 million tickets created in 2003 are related to product analyzed in this project. • Equipment Database • Software release • Number of licensed ports • 4 million systems listen in equipment databases.

9. Customer perceived quality models Model Model Model • Business activity distraction ---- Negative effect • To major aspect of customer perceived quality: • Impact of problem occurrence • Frequency of problem occurrence • Interest • Rare high-impact problems • Equipment services outages • Malfunction resulting in software modification Model Model Model Model Model

10. Customer perceived quality models Model Model Model • Frequent low impact problems • Technician dispatches • Customer calls • Alarm reports Model Model Model Model Model

11. Factors to predict • Factors • Deployment issues • Usage patterns • Software platform • Hardware configurations • Prior work examines • Software product • Development process lack empirical validation They are not good predictors (Static. do not vary for a single release)

12. Predictor Measures ? • Value Predictors • System size. • Operating system • Ports • Total Deployment time • Software Upgrades • Nuisance Predictors • US of international installation • Service contracts • Missing configuration information

13. System size • This predictor measures • Hardware configuration factors • Software platform factors and • Usage patterns factor. • Introduced “LARGE” variable indicator in the model • Small and medium systems have: • Fewer customer interactions • Few settings to configure • Fewer systems to interface with • Likely not to be involved in business critical application requiring 7x24 uptime. • Less likely to experience and report issues.

14. Operating System • Operating system predictor measures: • Software platform factor & • Hardware configuration factor • Considered • An open Linux • proprietary • Commercial windows • Very small number Os systems used NT/Win2000 • Off-the-shelf operating system introduce unnecessary complexity and configuration issue.

15. Ports • The port predictor measures • Usage pattern factor & • Hardware configuration factors • The number of ports indicates how many licensed end points are supported by the system. • Model encoded the log number of ports with log(nPort) variable

16. Total deployment time • Deployment time predictor measure the deployment issues • Anticipation: fewer customer interaction as the total deployment time increases. I need it on Timeeee!!!!

17. Software upgrades • The software upgrades predictor measures the deployment issues factor. • Encoded the existence of upgrade using an indictor variable called “Upgr”. • Upgrade serves to keep the machine running properly by incorporating the latest fixes and refinements to the system • Upgrades have clearly defined purpose of making the system more stable, so expect to have that effect.

18. Nuisance Factors • These factors are likely to identify peculiarities of data reporting and collection process, but not necessarily differences in underlying customer perceived quality. • US of International installation • Service contracts • Missing configuration information

19. Predict for each customer (outputs): Software defects System outages Technician dispatches Calls Automated alarms Using Logistic regression and Linear regression Using predictors (inputs): Total deployment time Operating system System size Ports Software upgrades For a real world software system Factors Vs Predictors

20. Results • First fit the models to test the relationships hypothesized previously. • Use the models to predict customer interaction for the next major releases. • Present full results only for two measures.

21. Software Failure Nuisance Factors

22. Most important predictor • Total Deployment time: • Customers who installed the application early may have detected malfunctions that are fixed by the time later customers install their systems • The individuals performing the installation and configuration may have acquired more experience, have access to improved documentation • The lesson from this relationship is that customers that are less tolerant of availability issues should not be the first to install a major software release “never upgrade to dot zero release.”

23. Another Important Predictor • Operating system (software platform, hardware configurations) • Systems running on the proprietary OS are 3 times less likely to experience a software defect compared with systems on running the open OS (Linux) • Systems running on the commercial OS (Windows) are 3 times more likely to experience a software defect compared with systems running on the open OS (Linux)

24. Customer calls

25. Predicting customer call traffic • Ports and NAPorts are not available • know from talking to the customer call center that they have estimates of how many calls a call representative can handle so our prediction of the number of calls in a month can be used to plan staffing. • The take away is that they the predictions are accurate and can be used to plan various activities

26. Validations • Accounted for data reporting differences • Included indicator variables in the models to identify populations (e.g. US or international customers) • Independently validated the data collection process • Independently extracted data and performed analyses • Interviewed personnel to validate findings • Programmers • Field technicians

27. Best Contributions • Identified and quantified characteristics, like time of deployment, that can affect customer perceived quality by more than an order of magnitude • created models that can predict various customer interactions and found that predictors have consistent effect across interactions • We learned that controlled deployment may be the key for high reliability systems