1 / 68

Characterizing the Impact of Requirements Volatility on Systems Engineering Effort

Characterizing the Impact of Requirements Volatility on Systems Engineering Effort. Mauricio E. Peña Dr. Ricardo Valerdi March 8, 2011. Agenda. 1:30 – 1:45 pm Introductions & objectives 1:45 – 2:00 pm COSYSMO overview and updates

fred
Download Presentation

Characterizing the Impact of Requirements Volatility on Systems Engineering Effort

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Characterizing the Impact of Requirements Volatility on Systems Engineering Effort Mauricio E. Peña Dr. Ricardo Valerdi March 8, 2011

  2. Agenda 1:30 – 1:45 pm Introductions & objectives 1:45 – 2:00 pm COSYSMO overview and updates 2:00 - 2:30 pm COSYSMO requirements volatility extension: introduction and research results 2:30 – 3:30 pm Requirements volatility measures: definitions & counting rules 3:30 – 3:45 pm Break 3:45 – 4:45 pm Survey Exercise: expected level of volatility and impact on systems engineering effort 4:45 – 5:00 pm Discussion

  3. Objectives of the Workshop • Learn about COSYSMO and the latest research updates • Explore the causes of requirements volatility and its impact on systems engineering effort • Obtain feedback on a proposed requirements volatility extension to COSYSMO • Reach agreement on definitions and rating scales for requirements volatility • Provide an opportunity for participants to exchange lessons learned on requirements volatility and influence the direction of future research

  4. COSYSMO Overview & Updates

  5. CSSE Parametric Cost Models • The Constructive Systems Engineering Cost Model (COSYSMO) was developed by the USC Center for Software and Systems Engineering (CSSE) in collaboration with INCOSE and Industry affiliates • COSYSMO is the first generally-available parametric cost model designed to estimate Systems Engineering effort • Built on experience from COCOMO 1981, COCOMO II • Most widely used software cost models worldwide • Developed with Affiliate funding, expertise, data support

  6. COSYSMO Operational Concept # Requirements # Interfaces # Scenarios # Algorithms Size* Drivers COSYSMO Effort Effort Multipliers • Application factors • 8 factors • Team factors • 6 factors Calibration WBS guided by EIA/ANSI 632 *Each weighted by complexity: easy, nominal, and difficult Source: Valerdi, R. (2005)

  7. Scope of the Model ISO/IEC 15288 – Common framework for describing the life cycle of systems EIA/ANSI 632 – Integrated set of fundamental systems engineering processes Source : Draft Report ISO Study Group May 2, 2000

  8. Life Cycle Phase Definition • Conceptualize phase focuses on identifying stakeholder needs, exploring different solution concepts, and proposing candidate solutions. • The Development phase involves refining the system requirements, creating a solution description, and building a system. • The Operational Test & Evaluation Phase involves verifying/validating the system and performing the appropriate inspections before it is delivered to the user • The Transition to Operation Phase involves the transition to utilization of the system to satisfy the users’ needs. Source: Valerdi, R. (2005)

  9. Parametric Cost Estimating Relationship • Where: PMNS = effort in Person Months (Nominal Schedule) A = constant derived from historical project data Size = determined by computing the weighted average of the (4) size drivers E = represents diseconomy of scale n = number of cost drivers (14) EM = effort multiplier for the ith cost driver. The geometric product results in an overall effort adjustment factor to the nominal effort Source: Valerdi, R. (2005)

  10. Application Factors Requirements understanding Architecture understanding Level of service requirements Migration complexity Technology Maturity Documentation Match to Life Cycle Needs # and Diversity of Installations/Platforms # of Recursive Levels in the Design Team Factors Stakeholder team cohesion Personnel/team capability Personnel experience/continuity Process maturity Multisite coordination Tool support COSYSMO Cost Drivers Source: Valerdi, R. (2005)

  11. Cost Driver Rating Scales Source: Valerdi, R. (2005)

  12. COSYSMO 2.0 Operational Concept Source: Fortune (2009)

  13. COSYSMO 2.0 Model Form Source: Fortune (2009)

  14. Reuse Category Weights Source: Fortune (2009)

  15. COSYSMO 2.0 Implementation Results • Across 44 projects at 1 diversified organization • Using COSYSMO: • PRED(.30) = 14% • PRED(.40) = 20% • PRED(.50) = 20% • R2 = 0.50 • Using COSYSMO 2.0: • PRED(.30) = 34% • PRED(.40) = 50% • PRED(.50) = 57% • R2 = 0.72 • Result: 36 of 44 (82%) estimates improved Source: Fortune (2009)

  16. COSYSMO Extension:Requirements Volatility

  17. Importance of Understanding Requirements Volatility • Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects1 • Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans2 • The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure3 System developers often lack effective methods and tools to account for and manage requirements volatility Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393

  18. Requirements Volatility is Expected • Changes to requirements are a part of our increasingly complex systems & dynamic business environment • Stakeholders needs evolve rapidly • The customer may not be able to fully specify the system requirements up front • New requirements may emerge as knowledge of the system evolves • Requirements often change during the early phases of the project as a result of trades and negotiations Requirements volatility must be anticipated and managed Sources: Kotonya and Sommerville (1995); Reifer (2000)

  19. Questions that will be covered • How do we account for the impact of requirements volatility on functional size and effort estimates? • What counting rules should be used for requirements volatility? • How do we distinguish between typical changes/iterations and unplanned changes that require more effort than originally projected? • How do we prevent from counting the elaboration of requirements as volatility (added requirements)? • Is it feasible to track the volatility of other size drivers (interfaces, operational scenarios, algorithms)?

  20. Requirements Volatility Definitions • Requirements volatility is typically defined as the change in requirements (added, deleted, and modified) over a given time interval • Also known as: • Requirements creep: An increase in scope and number of system requirements • Requirements churn: Instability in the requirements set – requirements are modified or re-worked without necessarily resulting in an increase in the total number of requirements Source: MIL-STD-498

  21. Requirements Volatility Metrics (1 of 3) Volatility expressed as a % of the total number of requirements Systems Engineering Leading Indicators Guide, Version 2.0, 2010

  22. Requirements Volatility Metrics (2 of 3) • Volatility metrics track the number and type of changes over time Source: Hammer, T., Huffman, L., and Rosenberg, L. (1998).

  23. Requirements Volatility Metrics (3 of 3)

  24. Systems Engineering Reviews and Acquisition Lifecycle Phases A Requirements baseline is established when the functional requirements and characteristics of the system are formally documented and placed under configuration control Source: DoD Instruction 5000.02 (2008)

  25. Requirements and Evolutionary Acquisition Process Source: DoD Instruction 5000.02 (2008)

  26. Requirements Trends as a Leading Indicator of Project Performance Evaluates trends in the growth, change, completeness and correctness of the system requirements. It helps to determine the stability and completeness of the system requirements which could potentially impact project performance Requirements Growth Trends Corrective Action Taken Number of Requirements Planned # of Requirements Actual # of Requirements Projected # of Requirements SRR PDR CDR Time Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010

  27. Implications to COSYSMO • During the development of COSYSMO, volatility was identified as a relevant adjustment factor to the model’s size drivers • However, there was insufficient data to incorporate volatility effects into the initial version of the model • The primary objective of the research is to complete the requirements volatility extension to COSYSMO within the existing structure and scope of the model Source: Valerdi (2005)

  28. COSYSMO Volatility Factor Source: Fortune (2009)

  29. Overview of Research to Date • Data were collected through surveys and interviews of subject-matter experts in four different workshops • A variety of industries were represented with an emphasis on aerospace & defense • Exploratory survey administered at: • Workshop # 1: 2010 USC-CSSE Annual Research Review • Workshop # 2: 2010 LAI Knowledge Exchange Event • Survey exercises were conducted at: • Workshop # 3: 2010 Practical Software and Systems Measurement (PSM) Users Group Conference • Workshop # 4: University of Southern California 25th Annual COCOMO Forum

  30. Observations from the Literature and Exploratory Research • Requirements volatility is correlated with an increase in project size and cost • The level of requirements volatility is a function of the system life cycle phase and does not necessarily behave linearly • The cost / effort impact of a requirements change increases the later the change occurs in the system life cycle • The impact of requirements volatility varies depending on the type of change: added, deleted, or modified • Removing a requirement may not necessarily result in a net decrease in engineering effort and cost A causal model diagram was developed that relates technical, organizational and contextual project factors to requirements volatility

  31. Requirements Volatility Causal Model Diagram Contextual / Environmental Changes Changes in Org / structure/ Process Changes in COTS Products Experienced Staff Poor Understanding of the System & Customer Needs Requirements Process Maturity Changes in Co-dependent Systems Technology Maturity Customer-driven Scope Change Requirements Volatility Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)

  32. Impact of Requirements Volatility +/- +/- +/- +/- +/- +/- Quality Rework / Defects Project Cost Requirements Volatility Customer Satisfaction Project Effort Number of System Requirements Project Schedule Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)

  33. Exploratory Survey • Developed to gather the perspectives of subject-matter experts on the causes, impacts, and expected level of requirements volatility for a given system of interest • Organizations represented: • The Aerospace Corporation, Northrop Grumman Corporation • The Boeing Company, Softstar Systems, Raytheon • United Launch Alliance, Massachusetts Institute of Technology, University of Southern California, and • United States Army • Unites States Navy

  34. Exploratory Survey Participant Background System Application Domain • 22 respondents • 23 years average industry experience • Primarily from a Military/Defense and Space Systems Background • Experienced on Systems with a fairly balanced H/W and S/W work content System H/W –S/W breakdown

  35. Expected Level of Volatility Across Lifecycle Phases Most respondents expect >20% volatility during the conceptualize phase of the project, decreasing to <5% in the transition to operation phase Source: LAI Knowledge Exchange / CSSE ARR Survey (2010)

  36. Operational Test & Evaluation Lifecycle Phase Transition to Operation Lifecycle Phase Impact of Hardware/Software Project Breakdown on Expected Volatility Respondents from S/W intensive projects tend to expect more volatility later in the lifecycle Source: LAI Knowledge Exchange / CSSE ARR Survey (2010)

  37. In general, results of the survey support observations from the literature and causal model Most respondents stated that requirements volatility will cause a moderate to large increase in the number of system requirements and rework Impacts of Volatility % of respondents Moderate decrease Moderate Increase No impact Large Increase

  38. Ranked Causes of Requirements Volatility (Exploratory Survey) 6 7 8 9 Contextual / Environmental Changes Changes in Org / structure/ Process Changes in COTS Products Experienced Staff 2 1 3 4 5 Poor Understanding of the System & Customer Needs Requirements Process Maturity Changes in Co-dependent Systems Technology Maturity Customer-driven Scope Change Requirements Volatility Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)

  39. Workshop # 3: PSM Conference • Survey Exercise administered during the 2010 Practical Software and Systems Measurement Conference • Attended by 9 participants from diverse engineering organizations and academic institutions • Organizations Represented: • Distributed Management, Northrop Grumman • Lockheed Martin, Ericsson España • Samsung SDS, U.S. Navy • Massachusetts Institute of Technology (MIT)

  40. Workshop # 3: Survey Exercise Participants were asked to: • Draw a requirements volatility profile across the lifecycle phases covered by COSYSMO • Draw an “ease of change” profile across the same life cycle phases to determine the volatility weighting factor • Discuss variation in 1 and 2 above for: • Large and small projects • Hardware and software projects • Development and recurring projects

  41. Sample Volatility Profiles Source: Practical Software & Systems Measurement Conference Survey (2010)

  42. Ease of Change Profile Cost Penalty defined as 1 / ease of change Average Ease of Change Factor (Estimated) 0.8 0.4 0.1 0.05 Average Cost Penalty (Estimated) 1.25 2.5 10 20

  43. Workshop #4: COCOMO Forum • Attended by 15 participants from 10 different engineering organizations and academic institutions • The participants were asked to estimate: • Expected level of volatility • The cumulative volatility (%) over the four life cycle phases covered by COSYSMO • How those changes in requirements would be partitioned per life cycle phase • The breakdown of volatility per type of change (added, deleted, modified) • Cost Penalty • Provided a change cost penalty as a function of lifecycle phase • Estimated the cost penalty per type of change

  44. Expected Volatility per Lifecycle Phase Expected Cumulative Volatility = 28%, STD = 14% Requirements Volatility (% changes)

  45. Weighting Factor per Lifecycle Phase 1 = no penalty

  46. Breakdown per type of Change Expected Breakdown of Volatility per type of change: 34% added, 51% modified, 15% deleted Source: COCOMO Forum Survey Exercise (2010)

  47. Proposed Requirements Volatility Extension

  48. Requirements Volatility Size Driver Adjustment Factor REVL is defined as the % of the baseline set of requirements that changes throughout the lifecycle A weighting factor is added to account for the effort penalty due to rework This relationship is expressed through the following equation: Where, Rx,r = Baseline number of requirements (easy, nominal, difficult) Reff = Effective number of requirements at the end of the project wv = Requirements volatility weighting factor Source: Boehm (2000)

  49. Aggregate Weighting Factor As previously discussed, the effort penalty due to requirements volatility is likely to increase across lifecycle phases In order to capture this potential lifecycle phase dependency in the model, the following relationship is proposed: Where wv = Requirements volatility weighting factor wx,l = Weighting factor for added, deleted, or modified Θx,l = % of total changes that were added, deleted or modified l = lifecycle phases

More Related