1 / 26

GQM and data analysis Example from a Norwegian company

This example showcases how a Norwegian company used GQM and data analysis to determine the characteristics of SDL modules that caused errors. They identified metrics and performed ANOVA analyses to find significant contributors to module complexity and errors.

lacroixd
Download Presentation

GQM and data analysis Example from a Norwegian company

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GQM and data analysisExample from a Norwegian company Tor Stålhane IDI / NTNU

  2. The Problem The company in question develops hardware and software. • two software groups • each with circa 15 developers. Part of the system is developed in SDL. In order to focus their V&V work better they needed to know which SDL module characteristics that caused errors. Possible candidates were number of module states, number of input signals etc.

  3. What to measure? In order to get a consistent and efficient data collection and analysis, we started with a GQM process. • A half-day basic course in GQM. • A half-day workshop where they identified • the questions that needed answers. • which metrics they needed in order to answer these questions.

  4. The Metrics Some of the metrics collected for eachSDL module: • Number of errors found. • Subjective - perceived - complexity. • Number of pages of SDL description. • Number of SDL states • Number of signals in • Number of signals out

  5. Qc – block complexity When we defined the Qc question, we decided to use a Kiviat diagram to display the metrics included in this question. We show the following data: • Non-modified metrics. Just to show that it is not a good idea • Normalized metrics values • Mean metrics values for each complexity class

  6. Non-modified metrics The large max value of M6b reduces everything else to “noise”

  7. Normalized metrics values – 1

  8. Normalized metrics values – 2 Low complexity High complexity Beware: different scales on the axis in the two diagrams

  9. Mean metrics values – 1 Beware differentscales on the axisin the three diagrams

  10. Mean metrics values – 2

  11. What makes it complex - 1 We went through all the hypothesis put forward by the developers and will look at three of them: • Number of states - M5 • Number of signals out - M9 • Number of pages in the SDL description - M10

  12. What makes it complex - 2 The data for the three metrics M5, M9 and M10 were sorted according to the complexity scores (High, Medium and Low). An ANOVA analyses was then performed for eachdata set. We decided to require a p-value better than10%

  13. All data points

  14. Data points split according to H, M or L H H M H M M L L

  15. ANOVA results - 1 Number of states – M5: Source of Variation SS df MS P-value Between Group 1190,04 2 595,02 0,25 Within Groups 1631,83 5 326,37 Number of states does not contributesignificantly to the complexity asperceived by the developers.

  16. ANOVA results - 2 Number of signals out – M9: Source of Variation SS dfMS P-value Between Group 2779,0 2 1389,521 0,098 Within Groups 1813,83 5 362,77 Number of signals outcontribute significantlyto the complexity as perceived by thedevelopers

  17. ANOVA results - 3 Number of pages in the SDL description – M10: Source of Variation SS df MS P-value Between Groups 5586,04 2 2793,02 0,04 Within Groups 2133,83 5 426,77 Number of pages in the SDL descriptioncontribute significantly to the complexity asperceived by the developers

  18. Summary - 1 SDL module complexity as perceived by the developers depends on two factors: • Number of signals out • Number of pages in the SDL description The other suspected factors identified during the GQM process did not give a significant contribution.

  19. What about Errors We now have some ideas on what makes amodule look complex to the developers. Is there anyconnectionbetween module complexity and the number oferrors in the modules. The ANOVA can give us an answer.

  20. Complexity and Errors - 1 Errors and complexity Source of VariationSS df MS P-value Between Groups 1646,832823,42 0,06 Within Groups 770,67 5154,13 It is reasonable to assume that complexmodules have more errors.

  21. Complexity and Errors - 2 If we look at the ANOVA summary table, we see that the differences are quite large: Groups Count Sum Average Variance Column 1 3 102 34 343 Column 2 3 22 7,33 41,33 Column 3 2 2 1 2 Due to few observations for each complexity level, the variances are large. Thus, we should be a bit careful in our conclusions.

  22. Conclusions With all the necessary caveats in mind the company decided as follows: In order to reduce the number of errors we need to focus our testing effort on modules with : • Large descriptions - more than 35 pages of SDL description. • Many signals out - more than 30. The limiting values are the average values from the ANOVA summary tables.

More Related