Seminar Software Quality and Safety

1. Seminar Software Quality and Safety Topic: “Comparing design models resultant from two processes: with and without Architecture – an empirical study“ Author: Lukas Banach Tutors: Nazim Madhavji Remo Ferrari Thursday 17.07.2008 Motivation Task Context Variables Instrument Results

2. Overview Motivation Task description Study context Static / dynamic attributes Instrument to measure attributes Preliminary results Future Work Motivation Task Context Variables Instrument Results

3. Motivation Someone wants to create lower-level models Does architecture help? Do we produce higher quality (faster / more precise, ...) with architecture? What are the costs and the benefits? There is a huge gap between research and the industry Useful to say: use architecture because of ... Motivation Task Context Variables Instrument Results

4. Motivation Answer research question: “How do lower-level design models constructed with a software architecture compare to lower-level design models that are not based on an architecture?“ Motivation Task Context Variables Instrument Results

5. My task description My focus is to establish the methology of study to answer this question Dependent variables, instument, statistical analysis Prepare an empirical study No study – beyond the scope of this seminar Motivation Task Context Variables Instrument Results

6. Basic idea of an empirical study Dependent variable (292) Dependent variable (38) Dependent variable (35) Dependent variable (436) Dependent variable (007) Dependent variable (112) Dependent variable (82) Dependent variable (2) Dependent variable (93) Dependent variable (4) influence Independent variable (1) Independent variable (0) Binary variable (0/1) Motivation Task ContextVariables Instrument Results

7. Where are the models from? Participants: ungraduate students Randomly assigned by the staff Their task: model the behaviour of a garage door Two types of groups With and without architecture Motivation Task ContextVariables Instrument Results

8. Where are the models from? 6 groups ouf of 19 XML data (Models) Task description All created documents Motivation Task ContextVariables Instrument Results

9. Uppaal design models Templates States Transitions Xml – Files Simulator Verifier Motivation Task ContextVariables Instrument Results

10. Study variables Independent variable (binary) Architecture or no architecture Dependent variables Static attributes Dynamic attributes Motivation Task Context VariablesInstrument Results

11. Static attributes Number of templates Number of states Number of transitions Size of the xml file Number of available features Detail level of the available features Complexity Level of the templates Motivation Task Context VariablesInstrument Results

12. Dynamic attributes Percentage of accessible states Percentage of correct scenarios Deadlocks Infinite loops Motivation Task Context VariablesInstrument Results

13. Instrument to measure attributes Xml parser Counting states, transitions, templates & complexity level Creates queries to check accessible states (E <>) Checklist available features and detail level Motivation Task Context VariablesInstrument Results

14. Instrument to measure attributes - Xml parser Motivation Task Context VariablesInstrument Results

15. Instrument to measure attributes Verifier Accessible states (E <>) Deadlocks (E<> deadlock) Scenarios Available features Simulator Motivation Task Context VariablesInstrument Results

16. Instrument to measure attributes - verifier Motivation Task Context VariablesInstrument Results

17. Instrument to measure attributes - simulator Motivation Task Context VariablesInstrument Results

18. Collected data Motivation Task Context VariablesInstrument Results

19. Collected data - features 1a: Opening the garage door by user pressing the button. 1e: All operations should involve displaying messages on the control unit which would be located in the garage. Motivation Task Context VariablesInstrument Results

20. Collected data - scenarios 1: User presses the garage door button to open the garage. While the door is opening, the user presses the button. The user presses the button again; however, there is an obstacle in the way of the door going down. Motivation Task Context VariablesInstrument Results

21. Statistical analysis - total Motivation Task Context VariablesInstrument Results

22. Statistical analysis - percentage Motivation Task Context VariablesInstrument Results

23. Statistical analysis - complexity Motivation Task Context VariablesInstrument Results

24. Statistical analysis - correlation Motivation Task Context VariablesInstrument Results

25. Statistical analysis - significance • χ2-Test (Chi-Square-Test) • H1: The correlation between the attribute “Architecture“ and the number of correct Scenarios is statistically significant. • H0: The attribute “Architecture“ and the number of correct Scenarios are statistically not correlated • χ2 = 0.1893 Motivation Task Context VariablesInstrument Results

26. Summary Attributes Created instrument Collect preliminary data Statistical analysis Motivation Task Context VariablesInstrument Results

27. Future work Improve scenarios (yes/no -> stopped at ) Create more queries with the xml parser For the study: Use professional people instead of students Motivation Task Context VariablesInstrument Results

28. The end Questions ?

29. χ2 χ2 = (3.5 – 4)² / 4 + (3.5 – 3)² / 3 + (11.5 – 11)² / 11 + (11.5 – 12)² / 12 χ2 = 0.1893 χ2 df = 1:5% = 3.84 χ2 df = 1:1% = 6.68 χ2 df = 1:0.1% = 10.8