Assessing the Value of Process Improvement

1. Assessing the Value of Process Improvement Jim Pederson Software Process Improvement Network – Feb, 2006

2. Overview of the Problem • CMMI directly and indirectly requires evaluation of process and technology improvements. • Link to organizations strategic direction • Plan-Do-Check-Act cycle validates significance • Commitment of business resources typically requires projection of positive Return on Investment (ROI) and rightfully so. • Analyzing ROI is inherently difficult • Costs not segregated • Multiple variables at work at once • Improvement real but not effectively deployed • Cost avoidance is difficult to definitively value • In some cases, individual changes can be impossible to assess • Strategic initiatives not linked directly to productivity • Improvement has potential value only when implemented with other related improvements (part of a broader system).

3. Process ROI – A Questionable History • Some high level data supports the notion that process investment improves efficiency but the case for this is not overwhelming. • Based heavily on defect cost avoidance • Not statistically compelling – inference only • Somewhat shaded by self interest • Most/All organizations have significant negative experiences with process initiatives gone bad. • Overran budget • Didn’t work • Wasn’t used • Had no significant impact • Program/Project personnel that support and champion process are uncommon. • Task oriented and frequently accustomed to working around processes • Tend to want quick and dramatic impact

4. Common Internal Project Approval Process • Internally funded projects, regardless of type, tend to follow a relatively common approval and tracking process. • Develop and define proposed project • Calculate proposed savings and anticipated costs • Prioritize and approve projects • CMMI (OID, CAR) depict an expanded process • Define and Quantify the Problem • Conduct root cause analysis • Develop Plan based on RCA • Track implementation and assess changes in process performance • The standard business model has some serious shortcomings • Doesn’t quantify initial problem or opportunity or even validate that it exists. • Doesn’t relate plan to root cause – assumed correlation • Doesn’t validate outcome or impact (or hold anybody responsible) • Primarily utilizes intuitive as opposed to analytical decision making • Does a poor job of filtering “make work” projects that are more the result of salesmanship than real need.

5. The Basics of ROI Calculation • Determine Cost to Implement Change • Development Costs • Implementation Costs • Management and Support Costs (can be recurring) • Determine Benefits • What programs/projects will use the improvement? • Identify recurring and non-recurring cost savings • Determine common infrastructure impact • Determine Duration • Number and duration of instances • Will the change be overcome by technology or a later change. • Calculate Outcome • Factor in risks • May need to consider time value of money

6. ROI in the Perfect Scenario • In the ideal scenario, there would be a before and after efficiency metric and a single change impacting a consistent process. • Regularly maintained and validated metrics shows clear improvement related specifically to a defined process/tool change • Process is related directly to control account and earned value management system • In reality, the conditions to support this rarely exist even if the change was directly related to an efficiency improvement • Cost impact not readily identifiable through accounting structure • Numerous other causal factors or independent variables at work at the same time. • Variance in the way the change is incorporated into programs/projects

7. Problem 1 – Process/Data Instrumentation • In order to show a clear value correlation, data must be available on the output being measured and the output must be linked to earned value. • Engineering outputs are frequently, but not always, available as data but may not be timely if the cycle time to implement spans multiple reporting periods. • Common outputs related to productivity calculations are things like drawings/models generated, lines of code generated, spec pages produced, etc.. • Assessing percent complete of in process work is difficult • Control accounts generally do not divide tasks in a manner that directly correlates to assessing targeted improvements. • Tendency is to keep the number of managed accounts down to minimize related overhead • Sometimes open fields can be used to at least collect actual charges but this will not address planned and scheduled performance • Modeling improvements using a form of time and motion analysis is the fallback alternative but lacks the same fidelity. • Analysis shows improvement is possible/probable but doesn’t demonstrate that improvements were actually realized.

8. Problem 2 – Projecting Deployment • Determining to what extent an improvement will impact new and future programs is difficult and requires the acceptance of a significant degree of uncertainty. • New program/project start ups • Impact on acquiring new work (in some cases) • Degree and rate of deployment on legacy programs • External business factors can have a great deal of impact on this topic. • Changes in markets • New competitors • Mergers • Technology Changes • Ultimately the best that can be done to mitigate this problem is to state the assumptions and ground rules, keep them consistent, and update forecasts as conditions change.

9. Problem 3 – All Things Don’t Remain Equal • A basic assumption for scientific experimentation that we all learn early on in school is “all other factors remain constant”. • Unfortunately in business, this is almost never true. • An improvement can be real but still see a decrease in measured performance. • Change in project scope/complexity • Increased risk • Personnel variances • Management • Schedule compression, etc.. • Likewise, a planned improvement can have no impact or even a negative impact but measured performance can improve. • To assess process and technology changes in a real world environment, the impact of causal factors need to be isolated from each other. • Assessing all variables separately isn’t practical • Regression Analysis • Design of Experiments (DOE) with limited opportunistic data sets.

10. Problem 4 – Assumptions on Project Management • A basic assumption that is central to parametric estimating is that effort is a function of work scope and productivity. • Schedule compression or elongation can effect planned productivity values • This assumes that assigned resource levels can be readily adjusted based on amount and type of work • An alternate theory, Theory of Constraints (TOC), holds that resources tend to remain relatively constant and productivity improvements are obtained principally through management of work flow and schedule. • A case can be made using data that TOC is a better reflection of reality than scope and rate based planning • The degree to which TOC is true or false varies with the organization but there is always some degree of truth to it and it cannot be readily dismissed • If productivity metrics change principally with the scope/size of a project, this is a good indication that TOC assumptions characterize the organization • To the extent that TOC concepts apply, process or technology improvements that should theoretically improve productivity may not. • This is especially true late in a project’s life cycle where testing and rework dominate the project’s total effort

11. Problem 5 – Addressing Cost Avoidance • Cost avoidance is often a contentious topic in formulating cost projections. • Theoretically, cost avoidance should be reflected in efficiency improvements of impacted control accounts • For this reason, accounting rules frequently prohibit cost avoidance as a stand alone justification in developing ROI projections • This pattern of thinking tends to require that resources be redeployed or reduced (BCWS be reduced) before any true savings can be established • Defect costs are the most common form of cost avoidance and pose unique challenges. • It generally isn’t practical to track defect costs discretely in the labor accounting system • Estimates based on characterization variables (phase, severity, type, etc.,.,) can give useable accuracy with minimal effort. • For cost avoidance to represent real savings, the impacted resources must be used for something else.

12. Assessing Process Infrastructure Improvement • The previous scenario addressed recurring costs incurred on programs. • Infrastructure improvements are generally the easiest to measure and focus on organization overhead and non-recurring program cost reductions. • Infrastructure tends to be a fixed cost and doesn’t require assumptions about future program/project population and duration • Cost avoidance is generally a reduction in recurring overhead labor and/or dollars (i.e. licensing costs, facility costs, ADPE costs, etc..) • Reduction in fixed infrastructure set up costs on programs • Infrastructure improvements, although easier to plan, implement, and measure, are less popular amongst process specialists in most instances. • Addresses the cost as opposed to value of process/technology infrastructure • Can lead to reductions in budget allocated to process/technology infrastructure • Many opportunities exist • Software costs • Program setup costs • Procedure/Process consolidation

13. Complex Improvements • There are many improvements that do not lend themselves well to costing at all and pose a risk of gross miscalculation. • Grouped changes • Strategic changes • Administrative or compliance driven changes. • These types of changes can generally not be reduced to an efficiency metric associated with streamlining process steps. • Determining what the baseline condition for comparison is often not clear • Change had to be made to remain consistent with competition • Change required by customer or regulator • Complex improvements tend to be more organizational than programatic and act as enablers.

14. Strategic Initiatives • Strategic initiatives (corporate goals) will routinely impact process and technology based on perceived industry direction and customer perception. • Assessing the value of these things is very difficult in the first place but we know these factors are real and relate closely to business survival • Customer perceptions can actually be wrong yet they drive procurement decisions • Determining these factors is largely subjective and done at a high organizational level • Process planning reacts to these high level inputs and is decomposed in multiple actions linking closely to business objectives (CMMI requirement) • Because the problem or opportunity is extremely difficult to quantify in the first place and partially dependent on what competitors do, calculating an accurate ROI is extremely difficult. • In some cases the ROI can be extremely large (up to and including long term business survival) and in other cases there may be none at all. • Some process changes/actions have to be regarded as strategic only and should be regarded as exceptions to normally processing per the CAR model.

15. Irreducible Complexity • A complex system is typically made up multiple components that perform a function only within the context of a larger system. • If a component is removed, the component is useless • If a component is removed, the system is probably useless • This is clear to see in deliverable systems but can be equally true of processes and environments. • Sometimes several changes can be linked together and they have value only as an integrated set – each individual change could be seen as an enabler but has inherent value by itself. • In practice, this scenario is relatively common especially when dealing with technology (tools/environment) • Process also will generally have to be modified in order to take advantage of a tool or technology upgrade. • In assessing the value of multiple process or technology changes that are interrelated, they should be valued as an integrated set. • There is no need to allocate value to individual elements and would be highly arbitrary anyway.

16. Program Deployment • It is not uncommon for a process or technology change to have real potential value but for the change not to be deployed on a program in a manner that that the change realizes any value. • Process changes, especially those involving technology and automation, replace as opposed to overlay process steps. • When the process change is handled by a program as being additive it will increase the work load and also reinforces images people may have that process add costs as opposed to reduce cost. • The question then arises as to whether the value of a planned improvement should be assessed assuming as planned deployment or should be adjusted based on deployment issues. • Deployment should be part of overall planning and ineffective deployment could be associated with bad planning • On the other hand programs/projects are relatively autonomous and project decisions can overcome good planning done at a organizational level • Planning a growth path and deployment curve is reasonable alternative that considers resistance to change, learning curve, and complexity

17. Forecasting and Managing Development • Some process actions involve a relatively short and simple installation process and can be handled much like an action item. • Others are not and need to be managed as projects. • Need specialized resources • Extend over an extended period of time • Require dedicated funding to accomplish (out of scope) • Process specialists aren’t necessarily good project managers • Larger process tasks need to be estimated and managed as projects • ALL development and DEPLOYMENT costs need to be included in the estimate and plan • Under scoping will cause some projects to go forward that shouldn’t leading to negative ROI • Projects that show signs of significant overruns early should be re-evaluated. • Implementation of CMMI related process improvements that require separate dedicated funding will often require support from a funding mechanism not directly related to CMMI infrastructure.

18. The Dangers of Large Process Initiatives • Large scale process improvements typically involve a large number of part time resources and can accumulate costs very quickly. • Part time representatives from many areas with differing priorities • Project leader has little authority over participants (low accountability) • Implementation cycle time and risks are difficult to predict • Communication across multiple organizations or sites can be extensive and very complex. • Mixing of business cultures is a common problem • Differing agendas including obstruction • Getting agreement may make outcome too generic to be useable by anyone • Large scale improvement projects are generally justified based on centralizing overhead costs • Despite attractiveness of centralizing OH, CMMI and history tend to favor an incremental methodology using smaller pilots.

19. Strategic and Political Factors • Strategic Initiatives may flow down to different organizations as independent actions leading to multiple approaches. • The roll of a core type organization is difficult to define in some respects but needed to avoid recurring the same costs multiple times. • Responsibility for performance should reside with the performing organization, which is why the task is normally assigned to a project or program • If multiple projects or programs approach the same objective independently, however, they will incur learning curve and infrastructure type costs multiple times and tend to make the same mistakes repeatedly • Process specialists will be more able to address issues concerning process, metrics, or infrastructure type technology than will project personnel in most instances • Core groups need to have a recognized role in program/project goals and objectives but this needs to be in support only and stop short of full ownership. • Share knowledge and common resources • Don’t dictate exactly common approach