INFO 630 Evaluation of Information Systems Prof. Glenn Booker

1. INFO 630Evaluation of Information SystemsProf. Glenn Booker Week 3 – Quality Management and Earned Value INFO630 Week 3

2. Quality Management System • Every organization should have a Quality Management System (QMS) • The QMS describes the quality organization (members, duties, and to whom they report) and activities which they need to perform • Quality Assurance, Quality Management, and Quality Control are often used interchangeably for the QMS’ name INFO630 Week 3

3. Typical QMS Activities • Activities by a quality management organization typically include • Process Evaluation – to determine if process is clear, and consistent with other related processes • Process Assessment - help ensure processes meet quality standards (CMMI, ISO 9000) INFO630 Week 3

4. Typical QMS Activities • Process Verification – to ensure that processes are actually being used • Product Evaluation – to inspect documents or other products (SW, HW) to verify their quality • Think of the tag ‘Inspected by 19’ in new clothes – that’s a product evaluation INFO630 Week 3

5. QMS Activities • Quality control for manufacturing, assembly, shipping, receiving, etc. • Support definition, collection, and analysis of metrics for each organization • The quality organization should be able to stop any activity if the product quality is inadequate (what defines inadequate?) INFO630 Week 3

6. Quality Road Map for Software Maintenance • An organization’s Quality Management System may be guided by some sort of Quality Road Map (high level plan) • The Quality Road Map is the basis for each release’s Quality Plan • The Road Map is the highest level corporate strategy for improving the product (Warning – IBM terminology here…) INFO630 Week 3

7. Quality Road Map • The Road Map should systematically introduce more comprehensive quality activities and measures • Let each team develop own quality goals • Consider process, product, people, and tools; how will each contribute to quality? • FAA uses 3-year overall plan, with one-year goals to help meet it INFO630 Week 3

8. Supplier Buy-in • Your quality efforts must be matched by your suppliers • This must be stated in the Quality Plan or equivalent • Hence the quality measurements need to fulfill the overall goals of the organization, and define objectives for each release or product, and for major suppliers INFO630 Week 3

9. Tracking, Measurement, Analysis • These are key to making quality management work • Software development must become quantitatively predictable to be regarded as a science • Large volume of data not needed; instead, focused, accurate, useful data is better INFO630 Week 3

10. Customer Problems • Problems include defects, user errors, usability problems, unclear instructions, duplicates of known problems, etc. • Usually express as Problems per User Month (PUM) • PUM = (Problems reported) / (total license-months) INFO630 Week 3

11. Customer Satisfaction • Often measured with 5-point Likert scale (with optional Net Satisfaction Index (NSI)): • Very Satisfied (NSI = 100%) • Satisfied (NSI = 75%) • Neutral (NSI = 50%) • Dissatisfied (NSI = 25%) • Very Dissatisfied (NSI = 0%) INFO630 Week 3

12. Customer Satisfaction • Note that average NSI is a weak metric, since it hides the type of distribution • The average NSI for half Very Satisfied and half Neutral customers looks the same as for all Satisfied • Like measuring the mean of a distribution, without measuring the standard deviation (width of the bell curve) INFO630 Week 3

13. In-Process Quality Metrics • Defect density during automated (machine) system testing • Good for comparing different releases of one product, or very similar products • Defect arrival rate • # of defects found vs. time (e.g. week, month) • Cumulative defects found vs. time • Look for stability to decide done with testing INFO630 Week 3

14. In-Process Quality Metrics • Clarify actual defects, vs. reported problems, vs. backlog of defects • Actual defects measures how many known defects have been found in the code • Reported problems includes code defects, user errors, etc. • Backlog shows how fast defects fixed, as well as found INFO630 Week 3

15. In-Process Quality Metrics • “Phase-based defect removal pattern” tracks defects during the entire life cycle • Ways to identify defects include requirements and design reviews, code inspections, formal verification, and all phases of testing • Want most defects found early in project (key theme) INFO630 Week 3

16. Software Maintenance Metrics • Focus on fixing existing problems, not changing features or requirements • Fix backlog = number of outstanding known problems • Backlog Management Index = (# closed problems) / (# new problems) * 100 • BMI > 100 means backlog of problems is being reduced INFO630 Week 3

17. Software Maintenance Metrics • New, closed, and the status of problems can also be tracked • Fix response time, measured by the “mean time problems are open” • Percent delinquent fixes = % of problems which exceeded desired response time • Note that this implies an active goal, since the expected fix time has been defined INFO630 Week 3

18. Fix Quality • Fixes are, in general, not all perfect - they may create new problems! • Fix quality = % of fixes which are defective (i.e. result in new problems) • Fix propagation rate = average number of new problems resulting from a fix INFO630 Week 3

19. What is Earned Value? • Earned Value is a method for planning and tracking the cost, schedule, and technical accomplishments of a project • Earned Value focuses on comparing the ACTUAL cost, schedule, and technical accomplishments to their BUDGETED (or planned) values INFO630 Week 3

20. Cost EarnedValue Schedule Technical Earned Value Earned Value balances measurement of cost, schedule, and technical performance. INFO630 Week 3

21. Why do we care? • Some organizations are contractually required to use earned value to report their progress • Payments are then based on the accomplishments claimed under earned value; hence earned value reporting claims can be legally binding! INFO630 Week 3

22. Foundation • Earned value requires having a written plan which describes each task, before that task is performed. • The plan must define, for each task: • Schedule (start, stop, and calendar duration) • Cost (or earned value) it is worth, and • Earned value measurement method (see later) INFO630 Week 3

23. Budget Basics • The total amount of money available on a contract is the Contract Budget Base (CBB) • Wise managers leave a Management Reserve (MR) of about 10-15%, and budget the rest to the project’s Performance Measurement Baseline (PMB) • Hence CBB = MR + PMB INFO630 Week 3

24. Budget Basics • Most mortals (project staff) are only aware of the PMB, and do not know the extent of the MR • The PMB may be broken into the undistributed budget (a more visible reserve than MR), plus distributed budget (which is the sum of all tasks’ cost accounts) • PMB = undistributed budget + distributed budget INFO630 Week 3

25. Budget Basics INFO630 Week 3

26. Data Element Term Acronym Scheduled Work Budgeted Cost for Work Scheduled BCWS Earned Value Budgeted Cost for Work Performed BCWP Actuals Actual Cost of Work Performed ACWP Authorized Work Budget At Completion BAC Forecasted Cost Estimate At Completion EAC Work Variance Schedule Variance SV Cost Variance Cost Variance CV Completion Variance Variance At Completion VAC Earned Value Terminology INFO630 Week 3

27. The Plan • The project plan produces a curve for the BCWS (budgeted cost of work scheduled) through its planned completion. This curve is typically S-shaped, and is plotted as cost versus time (e.g. $K versus months). • The total expected cost of the task, located at the end of the BCWS curve, is the BAC (budget at completion) INFO630 Week 3

28. Budgeted Cost of Work Scheduled BAC INFO630 Week 3

29. Tracking Progress • As the task is being performed, measure the amount of money actually being expended (ACWP, actual cost of work performed) • Keep track of the tasks which have actually been worked on, and look up how much earned value you get credit for that work (BCWP, the budgeted cost of work actually performed) INFO630 Week 3

30. Tracking Progress • Plot the ACWP, BCWP, and BCWS over time • Notice that ACWP and BCWP can only be shown to date, whereas the BCWS covers the entire project’s schedule • BCWS is the original project plan • ACWP is how much real money has been spent • BCWP is how much real work has been accomplished with that money INFO630 Week 3

31. Now Earned Value Example INFO630 Week 3

32. Analysis • At any time during the project: • Cost Variance (CV, in $) = ACWP - BCWP • Schedule Variance (SV, $) = BCWS - BCWP • Schedule Slip (time) = (current date) minus (date when current value of BCWP was found on the BCWS curve) • Positive numbers are bad for all of these; negative is good INFO630 Week 3

33. Projection • The ACWP curve is used to estimate when the task will actually be completed • The projected ACWP curve is called Estimate To Completion (ETC) • The end of the projected ETC curve is the Estimate At Completion (EAC) • That’s the expected end of the project (in $ and schedule) INFO630 Week 3

34. BAC EAC Notice the ACWP line becomes the ETC line after the present date Projected Project Completion INFO630 Week 3

35. Project Completion • Compare BAC (the originally planned completion of the project) and EAC (the current estimated completion of the project) • Projected Program Delay (schedule overrun) = Time difference between BAC and EAC • Variance At Completion (VAC, cost overrun) = Cost difference between BAC and EAC INFO630 Week 3

36. T/N is Time Now Earned Value Summary $ INFO630 Week 3

37. Earned Value Measures Key measures we’ll use are SPI and CPI INFO630 Week 3

38. Term Symbol Formula Checklist Actions Ratio of work accomplished against money spent (Efficiency Rating: Work Done for Resources Expended) Risk Exposure Performance Index ARWP REWP REPI Ratio of work accomplished against what should have been done (Efficiency Rating: Work done as compared to what should have been done) REWS REWP Risk Schedule Performance Index RSPI TFA EAC Ratio of Total Funds Available against total Estimated funds expended At Completion; or Total Funds Available against the Budgeted cost At Completion multiplied by the Cost Performance Index = Management Reserve Ratio MRR TFA BAC * = CPI More Earned Value Measures INFO630 Week 3

39. What are “good” values? Per the “U.S. Marine Corps Acquisition Procedures Handbook,” June 1997 INFO630 Week 3

40. Projection • Another method for projecting the end of the actual project is • Plot CPI and SPI over time • Based on the trends seen, estimate the CPI and SPI at the end of the project • EAC(cost) = BAC(cost) / CPI • EAC(schedule) = BAC(schedule) / SPI INFO630 Week 3

41. Example - Railroad Project • Given the following project: build 4 miles of railroad track in 4 weeks for $4 million • Actual performance to date is: After 3 weeks, $2 million has been spent and one mile of track has been built • Pure cost analysis would only show that the project has spent $1 million less than expected ($3M - $2M) • Is this good? INFO630 Week 3

42. Example - Railroad Project INFO630 Week 3

43. Example - Railroad Project EAC BAC INFO630 Week 3

44. Example - Status Analysis • Today’s earned value status: • BCWS = $3M (expected expenditure & work) • ACWP = $2M (actual expenditure) • BCWP = $1M (actual work accomplished) • Calculate • Cost Variance (CV) = ACWP - BCWP = $1M • Schedule Variance = BCWS - BCWP = $2M • Schedule Slip = (3 weeks) - (1 week) = 2 weeks INFO630 Week 3

45. Example - Status Analysis • Earned value analysis: • Planned $1M/week expenditure for 4 weeks • Hence planned $3M after 3 weeks (BCWS) • Earned value of only $1M (BCWP is 66% behind the plan) • Actual expenditure of $2M to date (ACWP has 100% overrun, given how much was accomplished) INFO630 Week 3

46. Example - Projected Outcome • Find EAC using SPI and CPI • Only are given one data point for status, so use: • SPI = BCWP/BCWS = $1M / $3M = 0.333 • CPI = BCWP/ACWP = $1M / $2M = 0.500 INFO630 Week 3

47. Example - Projected Outcome • BAC was 4 weeks and $4 million • EAC is • EAC(cost) = BAC(cost)/CPI = $4 million / 0.500 = $8 million • EAC(schedule) = BAC(schedule)/SPI = 4 weeks / 0.333 = 12 weeks • Hence projected cost overrun is $4M, and schedule overrun is 8 weeks INFO630 Week 3

48. Measurement Methods • The methods used to take credit for earned value are of key significance • It is critical that earned value claims are accurate and consistent • Credit, or Earned Value, for performing each task may be obtained in many different ways, depending on the type of task. INFO630 Week 3

49. Milestone Events Weighted Milestone Fixed Formula Percentage Complete Estimates Percentage Complete and Milestone Gates Equivalent Completed Units Earned Standards Apportioned Relationship to Discrete Work Level of Effort Measurement Methods INFO630 Week 3

50. Milestone Events • Milestone events earn 100% credit when: • The event has occurred, AND • All follow-up work is done (e.g. preparing and distributing minutes after a formal review) AND • All action items are closed • Until all of the foregoing activities are completed, no earned value may be taken INFO630 Week 3