Stefan Biffl Inst. of Software Technology and Interactive Systems TU Wien, Austria

1. Value-Based SE & Empirical SEValue-Based Project Monitoring an the contribution of QA activitiesExperiences, Opportunities, ChallengesISERN Meeting, Noosa Head, Australia, November 15 2005 Stefan Biffl Inst. of Software Technology and Interactive Systems TU Wien, Austria Stefan.Biffl@tuwien.ac.at

2. Experiences with Empirical QA/Inspection Studies Our experiences with empirical studies in the QA/inspection area • Reading techniques (perspective-based, usage-based); inspector capability • Estimation techniques (defects, reliability) • Recently: value-based requirements tracing (65% less effort with similar benefits) Focus on process-oriented models of methods, cost, and technical measures • Often: effectiveness, efficiency; some value orientation with focus on “critical defects”. • Interest in experimental findings that hold in a more general context. However, cost-benefit models to justify QA/inspection in a project need to take project and company context into account: • Cost drivers can typically be modeled well with data from project monitoring • However, the value contribution of QA and the impact of QA improvement is much harder to argue and quantify (especially for a general case). New book on “Value-Based Software Engineering” with many ISERN authors • Theory on valuation and decision support; measurement. • Applications: Architecture, RE selection for release planning, and QA/test management. • -> Benefits Results Chain helps model QA/inspection contributions to project monitoring.

3. VBSE Key Element: Benefits Results Chain SE is only part of development for software-intensive systems, e.g., aircraft, cars. • While software is an important success factor for added value, the actual benefits come from the application in the systems context. • Need for communication of value between stakeholders with different backgrounds and perceptions of "the project“ and key requirements • Stakeholders from SE and other disciplines need to sufficiently understand • The impact of IT requirements on SE project and product, e.g., simplifiers and complicators. • The impact of SE decisions on the “external” system and its evolution options. • Value-based SE key element: the “Benefits Results Chain”, allows linking SE project and product capabilities to business outcomes(focus on success factors, agility, bottlenecks, value breakers). • VBSE allows cost-effective “internal” trade-off analysis of SE activities according to value translated from the requirements (most valuable first). • Weigh conflicting quality attributes • Find balanced solutions for resource allocation in projects (how much investment in QA activities is enough?)

4. Opportunity: Value-Based Project Monitoring Value-based IT project reports on the actual value contributions over time allow: • Better alignment of stakeholders, project management, and engineers. • Release planning that allows linking project value to release results. • A negotiation process between stakeholders, management, and key engineers on the value contribution of • release products, • the strategic options for engineering solutions, and • their impact on the project and system lifetime cost and benefits; • Project reporting that links the results of release planning to project progress and its stakeholder value; allows to more flexibly react to real-world changes during a release. Potential contributions of QA activities to value-based project monitoring: • Plan-driven development: QA can avoid late rework and provide visibility on intermediate results. • QA provides opportunities for stakeholders to assess the value of deliverables. • QA provides tracing between concurrently evolving artifacts.

5. VBSE/EMSE Research Issues and Discussion Empirically study “best practice” for important SE planning and decision situations • Benefits Results Chain allows analysis to ensure sufficient strength for all major contributions and to identify weak links for improvement. • Elicitation of project and company context factors that influence the value contribution of a SE approach. • Identify context factors that are main value drivers. • Stakeholders utility functions may vary strongly -> need for negotiation support. • Investigate measures and models of SE value contribution in industry on project and business unit level • E.g., results chains of typical projects and IT initiatives -> frame for GQM. • Identify best practices for reporting IT project value contributions and the contribution of SE and QA practices. • Use benefits results chain to • Set expected levels of SE productivity and quality in a project context ;e.g., determine sufficient documentation in software lifecycle tailored to project needs as basis for QA planning. • Help focus improvement initiatives (QA, SPPI) on best value contributions.

6. Value-based SE: References • Boehm B.W.: Value-Based Software Engineering. Software Eng. Notes 28(2), 2003, 1-12 • Boehm, B. W. and Huang, L.G.: Value-Based Software Engineering: A Case Study, IEEE Computer, March 2003 pp 33–41 • Biffl, S., Aurum, A., Boehm, B., Erdogmus H., Grünbacher P.: Value-based Software Engineering. Springer Verlag, Heidelberg; September 2005 • Chps. 1, 2: Initial VBSE Theory • Chp. 3: Valuation of Software Initiatives Under Uncertainty • Chp. 4: Preference-Based Decision Support • Chp. 6: VBSE Key Elements • Chp. 7: Stakeholder Value Proposition Elicitation and Reconciliation • Chp. 9: Industrial Empirical Study on Requirements Selection • Chp. 11: Value-based Test Management • Chp. 12: Value-based Release Planning • Chp. 16: Quantifying the Value of New Technologies

7. Backup Slides Backup Slides

8. Challenges of Value-Based Project Monitoring Structure a project to support value delivery • Early assessment of options for product structure (architecture) that links to most important win conditions (success factors, bottlenecks, value breakers; stakeholder incentive alignment) • Trace requirements value (priorities) to project management, SE, and QA. Measurement Challenges: • If individual and group preferences are not known, inefficient requirements priorities, SE and QA investments may come out. • Stakeholders utility functions may vary strongly and even conflict. • It is not always meaningful to aggregate measures from different dimensions to one utility measure. • Multi-criteria models allow stakeholders a more direct assessment of options. However, the more complex data needs better analysis and visualization support to understand trade-offs between SE, QA, and other activities.

9. Value of Project Deliverables Project contribution to IT change impact • Analysis of requirements for their value impact. • Identify key win conditions that frame the project. • Communicate value priorities to project participants (project manager, developers). • Assess need for QA to complement SE activities. • Results chain: allows analysis to ensure sufficient strength for all major contributions and to identify weak links for improvement (bottlenecks; value breakers)