Defect Density Estimation through Verification and Validation

1. Defect Density Estimation through Verification and Validation Mark Sherriff and Laurie Williams North Carolina State University HCSS ’06 April 19, 2006

2. Agenda • Background • Motivation and Hypothesis • Software Certificates and Software Certificate Management Systems (SCMS) • DevCOP (Defect Estimation with V&V Certificates on Programming) • Research Goal and Methodology • DevCOP Certificates • The DevCOP SCMS Eclipse plug-in • Limitations • Current Status • Questions 2

3. Motivation • Software Reliability • Often not estimated until development is complete • Actual reliability not known until system is shipped to customers • Corrective action is more expensive later in the process • If defect density could be estimated in-process… • Steps could be taken to address issues early • More economical, could improve development effort 3

4. Hypothesis • Defect density estimation can be based upon the history of verification and validation techniques that have been performed on the project. Questions that need to be answered: • What is the best way to record V&V techniques? • How do I build a model that can predict defect density with V&V information? 4

5. Recording V&V Techniques • Software Certificates • A record of a verification and validation (V&V) practice employed by developers and used to support traceability between code and evidence of the V&V technique used • Software certificates could be any type of record: • Logs from test case runs • Reports from code inspections • Details on pair programming assignments • However, these are all different forms of V&V information and maintaining this information could be expensive. 5

6. Recording V&V Techniques • Software Certificate Management Systems • Provides an interface and infrastructure to automatically create, maintain, and analyze software certificates • Benefits: • Software maintenance • Analysis of V&V technique effectiveness • Reference in future projects • Current research: • OGI/PSU: Programatica, a SCMS for Haskell 6

7. Parametric Modeling • Method by which dependant variables are related to one or more independent variables with regard to previous data • In Software Engineering… • Purpose is to provide an estimated answer to a software development question earlier in the development lifecycle • Famous SE parametric models: COCOMO 81 and COCOMO II 7

8. Parametric Modeling • Software Testing Reliability Early Warning • Java and Haskell versions • Uses a suite of metrics gathered on static code to provide a reliability estimate • The model is calibrated to a particular organization using a regression equation • STREW is a good option because using operational profiles for a similar prediction can be expensive to create and maintain • If a reliability estimate can be created from testing and static metrics, could it be improved if we added other verification and validation information to the model? 8

9. DevCOP • Research hypothesis: Defect density estimation can be based upon the history of V&V techniques that have been performed on the project. • Methodology: Build a parametric model using software certificate information and previous project defect data to create a prediction model for future projects. 9

10. DevCOP Certificates • Records: • identifying information for the function it is associated with including its name, signature, class, and file location; • identifying information for the developer that created it; • the type of V&V technique used; • a hash of the function’s abstract syntax tree (AST); and • a significance weight. 10

11. DevCOP Certificates • Types of Certificates • Manual • Includes all manual techniques, such as pair programming and code inspections • Automated Static Analysis • Includes all techniques that can be run automatically on uncompiled code • Dynamic • Includes all techiques performed automatically at run time, such as black box testing • Formal • Includes all formal methods, such as lambda calculus and proofs 11

12. DevCOP SCMS Eclipse Plug-in • Currently supports manual V&V techniques and jcoverage certificates • Provides different methods for examining and managing certificate data • Demo 12

13. Building the Model 13

14. Limitations • Granularity of Certificates • Method level, not class or line of code • Composition of Certificates • Not much is known about how one V&V technique adds to another • Defect Severity • All defects are treated equally 14

15. Potential Side Effects • Retrospective Causal Analysis • Once certificates are recorded on a project and bugs are reported, developers can use certificate and defect information to evaluate the efficacy of their V&V practices. • Building certificate information in with compiled code base • If certificate information could travel with compiled code, it could be referenced at runtime so that other systems could evaluate whether it was to work with that system. 15

16. Thank you! • Questions? Queries? Quandries? Contact Information: Mark Sherriff mark.sherriff@ncsu.edu DevCOP Project: http://agile.csc.ncsu.edu/devcop/ 16