ISERN Survey & Benchmark 10 th anniversary meta-experiment project

1. ISERN Survey & Benchmark10th anniversary meta-experiment project Session Chair, Stefan Biffl Marcus Ciolkowski, Forrest Shull, and Dieter Rombach Strategy Organization Survey Design Benchmark Design Next Steps contact/info: ciolkows@informatik.uni-kl.de

2. Meta-experiment Goal • ISERN develops empirical methodology, i.e. how to use experiments (case studies, controlled experiments). • 1992: What is an experiment? • 2000: Many experiments conducted • 2002: How to conduct a meta-experiment; Data Collection from the distributed individual surveys and benchmarks; Presentation of initial results, Meta-analyis with historic experiments

3. Research Problem: History • ISERN has successful history on experiments with inspections, but mostly in academic environments • Variations in replications not systematically recorded: Make best use of existing information • Influence factors for inspection from an industry point of view should be elicited • Methodic organization • How to acquire knowledge • Demonstrate added value

4. ISERN Inspection Survey & Benchmark • Common empirical study in ISERN • One large survey study • Several options for inspection benchmarks • All interested ISERN-organizations collaborate • Run a series of small but related studies • Each will provide interesting results for the host site • In the aggregate, a chance to compare the impact of various environmental variables on the results. • Also include historic experiments.

5. Benefit • Common ISERN experiment • Pragmatic 1st round – low risk context; reasonable return on investment (ROI) • Low risk/cost for software inspection due to reuse • Experimental framework (baseline document, baseline systematic inspection technique) • Package with baseline inspection materials • Reasonable ROI due to scalable investment • Allows organizations to commit to various levels of effort, based on their constraints

6. Opportunities through ISERN Survey and Benchmark • Work with industry as well as universities • Benefit for industry: • Learn: under which constraints are results (positive trends) transferable? • Study: Demonstrate benefit and gain knowledge not by chance, but based on testable hypotheses (study design). • QIP for experiments: Maximize benefits from experiments. • Industry can introduce own questions and own factors • External experience often basis for introduction of techniques • Measurement data important for regular feedback on and reinforcement of successful techniques in practice.

7. Organization: Roles in the Experiment

8. Organization: People for roles • Steering Committee: contributors to experiment • Experiment Preparation Committee: authors of experiment documentation • Experiment Coordinator: knows experiment status • Librarian: administers experiment documentation • QA for Data Repository: checks incoming data • Meta-analysis Committee: plans data analysis

9. Working Group Inspection Survey • Goal: Overview on state-of-the-practice in inspections in industry • Characterize documents and defects in practice • Measure: How well does reading technique fit to document/defect profile/inspector skills? • So far: studies/questionnaires from Lund, Sweden • Working group chair: Ross Jeffery

10. Working Group Experiment Design • Goal: Create experimental design and materials for distributed experiment • Provide standard inspection techniques and baseline documents for inspection • So far: three levels of commitment • Working group chair: Forrest Shull, FC MD

11. ISERN Experiment Support • Preparation • Baseline document and inspection material • Training material • Conduct • Guidance through local researcher • Data Analysis and Presentation • Data collection (guaranty of anonymity) • Evaluation and customized feedback to organization

12. Next Steps • Negotiation and Agreement on Procedure • Commitment to conduct studies • Milestones for individual study • Contact to organization • Negotiate options • Conduct study • Data collection in ISERN repository • Data meta-analysis (including historical data)

13. Questions & Open Issues • Clear, can be discussed now: • Unclear, should be discussed off-line: • Working groups to resolve unclear issues into a set of clear options for participants.

14. Sign-up for participation • Local researcher: ____________________ • Contact: ___________________________ • Max. number of partners ...: • ... for survey:___ • ... for study of standard process:___ • ... for semi-controlled experiment:___ • Opportunities for collaboration (shadow experiments): _____________________