220 likes | 232 Views
This research paper discusses the importance of understanding requirements volatility and its implications on systems engineering effort. It presents a proposed model and conducts a preliminary validation of the model. The paper also highlights the need for effective methods and tools to manage requirements volatility.
E N D
Mathematical Formulation and Validation of the Impact of Requirements Volatility on Systems Engineering Effort November 2nd, 2011 Mauricio E. Peña Dr. Ricardo Valerdi
Outline • Motivation and introduction • Implications to COSYSMO • Research methods • Observations from exploratory research • Proposed model of the impact of requirements volatility on Systems Engineering effort • Preliminary model validation • Conclusions and next steps
Importance of Understanding Requirements Volatility • Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects1 • Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans2 • The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure3 System developers often lack effective methods and tools to account for and manage requirements volatility Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393
Requirements Volatility is Expected • Changes to requirements are a part of our increasingly complex systems & dynamic business environment • Stakeholders needs evolve rapidly • The customer may not be able to fully specify the system requirements up front • New requirements may emerge as knowledge of the system evolves • Requirements often change during the early phases of the project as a result of trades and negotiations Requirements volatility must be anticipated and managed Sources: Kotonya and Sommerville (1995); Reifer (2000)
Requirements Volatility Definitions • Requirements volatility is typically defined as the change in requirements (added, deleted, and modified) over a given time interval • Also known as: • Requirements creep: An increase in scope and number of system requirements • Requirements churn: Instability in the requirements set – requirements are modified or re-worked without necessarily resulting in an increase in the total number of requirements Source: MIL-STD-498 (1994)
CSSE Parametric Cost Models • The Constructive Systems Engineering Cost Model (COSYSMO) was developed by the USC Center for Software and Systems Engineering (CSSE) in collaboration with INCOSE and Industry affiliates • COSYSMO is the first generally-available parametric cost model designed to estimate Systems Engineering effort • Built on experience from COCOMO 1981, COCOMO II • During the development of COSYSMO, volatility was identified as a relevant adjustment factor to the model’s size drivers Source: 7th Annual Practical Software and Systems Measurement Conference. COSYSMO Workshop, Boehm
COSYSMO 2.0 Operational Concept Proposed Volatility Factor Source: Fortune (2009)
Life Cycle Phase Definition • Conceptualize phase focuses on identifying stakeholder needs, exploring different solution concepts, and proposing candidate solutions. • The Development phase involves refining the system requirements, creating a solution description, and building a system. • The Operational Test & Evaluation Phase involves verifying/validating the system and performing the appropriate inspections before it is delivered to the user • The Transition to Operation Phase involves the transition to utilization of the system to satisfy the users’ needs. Source: Valerdi (2008); ISO/IEC 15288 (2002)
Research Methods In Progress Literature Review & 6 Workshops completed to date Source: Boehm et al (2000)
Observations from the Literature and Exploratory Research • Requirements volatility is caused by an identifiable set of project and organizational factors. • The level of requirements volatility is a function of the system life cycle phase. • Requirements volatility leads to an increase in project size and cost • The cost and effort impact of a requirements change increases the later the change occurs in the system life cycle • The impact of requirements volatility varies depending on the type of change: added, deleted, or modified
Life Cycle Effort Penalty due to Volatility (n = 27) Systems Engineering Effort Penalty Due to Volatility Operational Test & Evaluation Transition to Operation Development Conceptualize Data collected from two workshops: 25th Annual USC CSSE COCOMO and the 2011 USC-CSSE Annual Research Review
Estimated Life Cycle Effort Penalty per Change Category (n = 9) Estimated SE Effort Penalty due to Volatility Operational Test & Evaluation Transition to Operation Development Conceptualize Results from a two-round Delphi Survey held at the 2011 Practical Software and Systems Measurement Conference
Mathematical Formulation of Requirements Volatility ( 1 of 2) Where, R0 = Baseline number of requirements Req = Equivalent number of requirements wv= Requirements Volatility weighting factor • The effective increase in the number of requirements would result in an associated increase in systems engineering effort Sources: Boehm, B., et al (2000)
Mathematical Formulation of Requirements Volatility ( 2 of 2) Where wv = Requirements volatility weighting factor wx,l = Weighting factor for added, deleted, or modified requirements Θx,l = % of total requirements changes that were added, deleted or modified l = lifecycle phases Pena (2010)
Preliminary Model Validation • Data were collected from nine projects from a space systems application domain • Number of requirements, interfaces, algorithms operational scenarios (estimated systems engineering size) • Added, modified, and deleted requirements over time • The cost estimation accuracy of COSYSMO was compared to the accuracy of the model with the requirements volatility size driver adjustment factor • The models were tested using the coefficient of determination (R2) and predictive accuracy levels • The relationship between requirements volatility and systems engineering size was also examined
Baseline Coefficient of Determination (n = 9) * Due to proprietary reasons only the analysis of the model accuracy is shown, not the data itself
Prediction Accuracy Level The prediction accuracy at a particular level (l) is defined as Where, n = set of projects k = number of projects in the set whose Magnitude of Relative Error is ≤ l Source: Conte, Dunsmore, and Shen (1986)
Relationship Between Requirements Volatility and Sys. Engr. Size
Relationship Between the Volatility Adjustment Factor and Sys. Engr. Size
Conclusions – Next Steps • Observations from the literature and workshops were used to develop a mathematical framework for quantifying the impact of requirements volatility on systems engineering effort • A preliminary validation of the model was performed by comparing its prediction accuracy against COSYSMO • The preliminary results indicate an improvement in systems engineering effort prediction accuracy when a requirements volatility factor is added to the model • The correlation between the requirements volatility factor and the estimated systems engineering size was examined – the results were not conclusive • Additional project data will be collected to complete the model validation
References • B. Blanchard and W. Fabrycky, Systems engineering & analysis, Prentice Hall, New York, NY, 1998. • B. Boehm, Software risk management: Principles and practices, IEEE Software 1 (1991), 32-41. • B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E., Horowitz, R. Madachy, D.J. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, New York, NY, 2000 • S. Conte, H. Dunsmore, and V. Shen. Software Engineering Metrics and Models. Benjamin/Cummings Publishing Company, 1986. • Department of Defense, “Instruction 5000.02. Operation of the Defense Acquisition System,” 2008. • S. Ferreira, J. Collofello, D. Shunk and G. Mackulak, Understanding the Effects of Requirements Volatility in Software Engineering by Using Analytical Modeling and Software Process Simulation, Journal of Systems and Software 82 (2009) 1568-1577. • General Accounting Office, "Stronger management practices are needed to improve DoD’s software-intensive weapon acquisitions (GAO-04-393)," 2004. • D. Houston, "A software project simulation model for risk management," Ph.D. Dissertation, Arizona State University, 2000. • ISO/IEC, "ISO/IEC 15288:2002 (e) systems engineering - system life cycle processes," 2002. • C. Jones, Assessment and Control of Software Risks, Prentice Hall, Inc. Englewood Cliffs, New Jersey, 1994. • C. Jones, Strategies for managing requirements creep, IEEE Computer, Vol 20 (1996), pp. 92-94 • G. Kotonya and I. Sommerville, Requirements engineering: Processes and techniques, John Wiley & Sons, New York, NY, 1998. • G.P. Kulk, and C. Verhoef, Quantifying Requirements Volatility Effects, Science of Computer Programming, 72, 136-175, 2008. • Y. Malaiya, and J. Denton, Requirements Volatility and Defect Density, Proceedings of the International Symposium on Software Reliability Engineering, 1999 • MIL-STD-498, "Software development and documentation," 1994. • S. Nidumolu, Standardization, Requirements Uncertainty and Software Project Performance, Information and Management. Vol 31 (No. 3) (1996), pp 135-150 • M. Pena, Characterizing the Impacts of Requirements Volatiliy, Proceedings of the 25th Annual COCOMO Forum, 2010 • D. Pfahl, and K. Lebsanft, Using Simulation to Analyze the Impact of Software Requirements Volatility on Project Performance, Information and Software Technology, Vol 42 (No. 14) (2000), pp 1001-1008. • D. Reifer, Requirements management: The search for nirvana, IEEE Software 17(3) (2000), 45-47 • D. Rhodes, R. Valerdi and G. Roedler, Systems engineering leading indicators for assessing program and technical effectiveness, Systems Engineering 12(1) (2009), 21-35. • R. Valerdi, The Constructive Systems Engineering Cost Model (COSYSMO): Quantifying the Costs of Systems Engineering Effort in Complex Systems, Saarbrücken, Germany, VDM Verlag, 2008. • D. Zowghi and N. Nurmuliani, A study of the impact of requirements volatility on software project performance, Proceedings of the Ninth Asia-Pacific Software Engineering Conference, 2002.