1 / 41

Requirements Volatility

Requirements Volatility. Mauricio E. Peña February 28, 2011. Agenda. Introduction Definitions Requirements development and management Requirements metrics Causes of requirements volatility Impacts of requirements volatility Mitigation approaches Summary.

astra
Download Presentation

Requirements Volatility

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Requirements Volatility Mauricio E. Peña February 28, 2011

  2. Agenda • Introduction • Definitions • Requirements development and management • Requirements metrics • Causes of requirements volatility • Impacts of requirements volatility • Mitigation approaches • Summary

  3. Importance of Understanding Requirements Volatility • Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects1 • Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans2 • The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure3 System developers often lack effective methods and tools to account for and manage requirements volatility Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393

  4. Requirements Volatility is Expected • Changes to requirements are a part of our increasingly complex systems & dynamic business environment • Stakeholders needs evolve rapidly • The customer may not be able to fully specify the system requirements up front • New requirements may emerge as knowledge of the system evolves • Requirements often change during the early phases of the project as a result of trades and negotiations • Requirements volatility must be anticipated and managed Sources; Kotonya and Sommerville (1995); Reifer (2000)

  5. Requirements Definitions “Requirements are intended to change vague desires into explicit and unambiguous statements of what the customers want” 1 “A requirement is a capability that a system must supply or a quality that a system must possess in order to solve a problem or achieve an objective within the system’s conceptual domain” 2 “A requirement is: (1) A condition or capability needed by a user to solve a problem or achieve an objective (2) A condition or capability that must be met or possessed by a system or system component to satisfy a contract, standard, specification, or other formally imposed documents ” 3 Source: 1- Weinberg, G. (1983), 2- Costello and Liu (1995); 3- IEEE (1990)

  6. Requirements Categories • Customer or operational requirements define the expectations of the system in terms of operational scenarios and environments, mission objectives, and measures of effectiveness • Functional requirements specify what has to be done in terms of tasks or activities • Performance requirements define how well or to what extent the activities must be accomplished • Design requirements dictate how to build products and what manufacturing processes to follow • Derived requirements are spawned from higher-level requirements; and • Allocated requirements: high level-requirements are partitioned into multiple lower-level requirements Source: DoD Systems Management College (2001)

  7. Requirements Volatility Definitions • Requirements volatility is typically defined as the change in requirements (added, deleted, and modified) over a given time interval • Also known as: • Requirements creep: An increase in scope and number of system requirements • Requirements churn: Instability in the requirements set – requirements are modified or re-worked without necessarily resulting in an increase in the total number of requirements Source: MIL-STD-498

  8. Requirements Management • The Systems Engineering Capability Model (EIA, 2002) defines five activities that influence the generation and evolution of requirements1 • Problem refinement • Requirements analysis • Requirements quality • Requirements evolution and maintenance • Feedback and verification • Requirements definition and analysis are sometimes collectively referred to as requirements engineering: • The use of systematic and repeatable techniques to identify, document, develop and maintain a complete and consistent requirement set2 Source: 1- EIA (2002); 2- Kotonya and Sommerville (1998)

  9. Context Diagram – Requirements Analysis Process Source: Systems Engineering Handbook. INCOSE Version 3.2

  10. Requirements Development The customer’s needs and objectives are developed into operational concepts to satisfy them System functional and performance requirements are developed to define what the system must do and under what environments and constraints it must perform Once the baseline system requirements and system architecture are established, they are decomposed through successively lower levels. Source: DoD Instruction 5000.02 (2008)

  11. Requirements Baseline and Change Control • A Requirements baseline is established when the functional requirements and characteristics of the system are formally documented and placed under configuration control • After the System Requirements Review or equivalent milestone • Modifications to the baseline are requested through Engineering Change Proposals (ECP) • Typically, a formal management approval structure is established to prioritize the ECPs, evaluate their impact, and make decisions regarding their approval and implementation • Other change documents used to control the requirements baseline are Requests for Deviation and Requests for Waivers • RDWs propose a departure from the baseline by allowing the acceptance of products that are non-conformant or do not meet the requirements as stated Source: DoD Instruction 5000.02 (2008)

  12. Systems Engineering Reviews and Acquisition Lifecycle Phases Source: DoD Instruction 5000.02 (2008)

  13. Requirements and Evolutionary Acquisition Process Source: DoD Instruction 5000.02 (2008)

  14. Requirements Base Measures Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010

  15. Requirements Derived Measures % Requirements Growth = ((# of requirements in current baseline - # of requirements in previous baseline) ( # of requirements in previous baseline) X 100 % Requirements Modified = (# of requirements modified / total # of requirements) Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010

  16. Requirements Volatility Metrics (1 of 2) Volatility expressed as a % of the total number of requirements Systems Engineering Leading Indicators Guide, Version 2.0, 2010

  17. Requirements Volatility Metrics (2 of 2) • Volatility metrics track the number and type of changes over time Source: Hammer, T., Huffman, L., and Rosenberg, L. (1998).

  18. Periodic and Cumulative Measures

  19. Expected Level of Volatility Across Lifecycle Phases (Survey) Most respondents expect >20% volatility during the conceptualize phase of the project, decreasing to <5% in the transition to operation phase Source: LAI Knowledge Exchange / CSSE ARR Survey (2010)

  20. Operational Test & Evaluation Lifecycle Phase Transition to Operation Lifecycle Phase Impact of Hardware/Software Project Breakdown on Expected Volatility Respondents from S/W intensive projects tend to expect more volatility later in the lifecycle Source: LAI Knowledge Exchange / CSSE ARR Survey (2010)

  21. Sample Volatility Profiles Source: Practical Software & Systems Measurement Conference Survey (2010)

  22. Requirements Trends as a Leading Indicator of Project Performance Evaluates trends in the growth, change, completeness and correctness of the system requirements. It helps to determine the stability and completeness of the system requirements which could potentially impact project performance Requirements Growth Trends Corrective Action Taken Number of Requirements Planned # of Requirements Actual # of Requirements Projected # of Requirements SRR PDR CDR Time Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010

  23. Causes of Requirements Volatility • External factors • Shifting customer priorities and needs • Changes in the political and business environment • Addition or change of stakeholders • Development of new technologies • Changes in co-dependent systems • Internal factors • Deficient requirements development processes • Lack of experienced systems engineering resources applied to requirements analysis • Poor initial understanding or interpretation of customer needs by the development team • Changes in organizational structure and policies Sources: Kotonya and Sommerville (1995); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)

  24. Systems Dynamics View of the Causes of Requirements Volatility 6 7 8 9 Contextual / Environmental Changes Changes in Org / structure/ Process Changes in COTS Products Experienced Staff 2 1 3 4 5 Poor Understanding of the System & Customer Needs Requirements Process Maturity Changes in Co-dependent Systems Technology Maturity Customer-driven Scope Change Requirements Volatility Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)

  25. Impacts of Requirements Volatility • Several research studies have found that requirements volatility is positively correlated with increase in • The functional size of the project (due to discarded lines of code and additional requirements) • Engineering effort • Cost (often >20%) and schedule duration • Changes in requirements result in additional rework and increased defect density • The impact of changing requirements is much greater the later the change occurs in the project lifecycle • Removing a requirement may not necessarily result in a net decrease in engineering effort and cost Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)

  26. Systems Dynamics View of the Impact of Requirements Volatility +/- +/- +/- +/- +/- +/- Quality Rework / Defects Project Cost Requirements Volatility Customer Satisfaction Project Effort Number of System Requirements Project Schedule Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)

  27. Cost Commitment on Projects Changes to the System are more difficult to implement the later they occur in the lifecycle Source: Blanchard and Fabrycky (1998)

  28. Expected Cost Penalty due to Requirements Volatility (Survey Results) Source: COCOMO Forum Survey Exercise (2010)

  29. Requirements Volatility: A Sizing and Cost Estimation Challenge • Our dynamic competitive environment leads to rapid changes in objectives, constraints and priorities • Requirements are often emergent instead of pre-specifiable • These trends have lead to the increasing use of incremental and evolutionary development strategies • Counting the number of source lines of code (SLOC) delivered may underestimate the size of the system • Portions of the code may have been modified or deleted due to changing requirements • These modified or discarded lines would not result in an increase in the final SLOC count Source: AFCAA Software Cost Estimation Metrics Manual Version 0.82 (draft)

  30. Options for Handling Cost and Size Estimation Challenges • Improve ability to estimate requirements volatility • Data collection and analysis: use of code counters able to determine numbers of SLOC added, modified, and deleted during development • If data are not available, estimate expected ranges of requirements volatility • For incremental and evolutionary development: • Treat the earlier increments as reused software and apply reuse factors • Include an Incremental Development Productivity Decline (IDPD) factor • A decrease in producibility is observed in later increments • Previous-increment breakage, usage feedback, and increased integration and test effort lead to higher unit cost Source: AFCAA Software Cost Estimation Metrics Manual Version 0.82 (draft); Nguyen (2010);

  31. Requirements Volatility and Parametric Cost Estimation • The Constructive Cost Model (COCOMO II) incorporated the impact of requirements volatility on the effective size of software systems using an adjustment factor called REVL (Requirements Evolution and Volatility) • REVL is also known as the “breakage parameter,” defined as the % of code discarded due to changes in requirements. Where, SizeD is the initial equivalent size of the software product adjusted for reuse Source: Boehm et al. (2000).

  32. Requirements Volatility Expected Ranges Source: AFCAA Software Cost Estimation Metrics Manual Version 0.82 (draft)

  33. Incremental Development Productivity Decline (IDPD) Example 4M SLOC 1M Source: AFCAA Software Cost Estimation Metrics Manual Version 0.82 (draft)

  34. Actions to Mitigate Requirements Volatility • Improve communication between stakeholders and development team • Improve the capability and maturity of the requirements development process • Additional training / more experienced personnel • Increase the use of prototypes and models to validate requirements • Improve technology insertion processes • Increase management oversight (i.e. metrics, reviews) • Establish a more strict change control process

  35. Mitigation Factor Effectiveness: Survey Results Source: CSSE ARR Survey (2010)

  36. Characteristics of Good Requirements (1 of 3) • Necessary • Avoid specification of design details & redundant requirements • Implementation Independent • Specify “what” is to be done, not “how” to do it • Clear and Concise • Don’t use ambiguous terms; avoid compound requirements with multiple “shall statements” and conjunctions such as “and”, “or” • Example – is the following a good requirement? • “The car shall accelerate from 0 to 60 MPH in 5 seconds and break to a standstill in 10 seconds under extreme conditions” Source: Systems Engineering Handbook. INCOSE Version 3.2

  37. Characteristics of Good Requirements (2 of 3) • Complete • Provide all the information necessary for the developer to understand what you mean • Do you have any TBXs (to be determined, to be confirmed)? • Consistent • Is the requirement self-consistent? • Do you have conflicting statements? • Achievable • Engage the developer/designer in the requirements definition to ensure they concur that the requirement can be met Source: Systems Engineering Handbook. INCOSE Version 3.2

  38. Characteristics of Good Requirements (3 of 3) • Traceable • Are there any orphan requirements (i.e. they don’t trace to a higher level requirement)? • Verifiable • Avoid subjective terms such as “provide support,” and “user friendly” • Look out for conditions that are not clearly achieved such as “minimize,” “to the degree possible,” etc. • Example: • The aircraft should be as light as possible • The wet weight of the aircraft shall not exceed 2000 lbs. Source: Systems Engineering Handbook. INCOSE Version 3.2

  39. + - - + + + Requirements Monitoring & Control Org Improvement Actions +/- SE Process Maturity Staff Experience Volatility Metrics / Thresholds Requirements Volatility +/- +/- Number of Requirements Rework +/- +/- SE Project Effort +/- +/- Project Schedule SE Cost Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)

  40. Summary • Requirements changes are unavoidable in today’s rapidly changing environment • Some level of volatility is healthy and is part of the system development process • However, excessive volatility and late requirements changes can have a very detrimental effect on the project – this is a risk that must be addressed and mitigated • Requirements volatility should be anticipated, managed, and accounted for in cost and size estimates

  41. References • Air Foce Cost Analysis Agency Software Cost Estimation Metrics Manual, Draft Version 0.82 (2011) • Boehm, B. (1991). “Software Risk Management: Principles and Practices.” IEEE Software. Vol 8 (No.1).pp 32-41 • Boehm, B., Abts, C., Brown, A.W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D.J., and Steece, B. (2000). Software Cost Estimation with COCOMO II. Prentice Hall • EIA (2002). EIA-731.1 - Systems Engineering Capability Model • Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements volatility in software engineering by using analytical modeling and software process simulation.” The Journal of Systems and Software. Vol. 82, pp 1568-1577. • General Accounting Office (2004). Stronger Management Practices are Needed to Improve DOD’s Software-intensive Weapon Acquisitions (GAO-04-393). Defense Acquisitions. • Hammer, T., Huffman, L., and Rosenberg, L. (1998). “Doing requirements right the first time.” Crosstalk, the Journal of Defense Software Engineering. Pp 20-25. • Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State University • IEEE (1990). IEEE Standard Glossary of Software Engineering Terminology (ANSI), [1-55937-067-X] [SH13748-NYF], IEEE • INCOSE (2010). Systems Engineering Handbook. INCOSE Version 3.2 • Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and Sons, Ltd. • MIL-STD-498. 1994. Software Development and Documentation. U.S. Department of Defense. • Nguyen, V. and Boehm, B. (2010). A COCOMO Extension for Software Maintenance. 25th International Forum on COCOMO and Systems/Software Cost Modeling • Reifer, Donald J. 2000. “Requirements Management: The Search for Nirvana.” IEEE Software. Vol 17 (No. 3), pp 45-47 • Roedler, G. and Rhodes, D. (2007). Systems engineering leading indicators guide. Version 1. Massachusetts Institute of Technology, INCOSE, and PSM • Weinberg, G. (1993). Quality Software Management: Volume 2 First-Order Measurement. Dorset House. • Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project Performance. Proceedings of the Ninth Asia-Pacific Software Engineering Conference

More Related