1 / 60

Effects of Using Examples on Structural Model Comprehension: A Controlled Experiment

Effects of Using Examples on Structural Model Comprehension: A Controlled Experiment. Dina Zayan , Michał Antkiewicz , Krzysztof Czarnecki 2014.06.06. http://gsd.uwaterloo.ca. http ://necsis.ca. Many problems in SE are related to domain knowledge transfer. For example….

bbaldwin
Download Presentation

Effects of Using Examples on Structural Model Comprehension: A Controlled Experiment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Effects of Using Examples on Structural Model Comprehension: A Controlled Experiment Dina Zayan, Michał Antkiewicz, Krzysztof Czarnecki 2014.06.06 http://gsd.uwaterloo.ca http://necsis.ca

  2. Many problems in SE are related to domain knowledge transfer

  3. For example… • Business analysts communicating to other stakeholders who are novices • Industry standard which contains domain models applied by novices • Specifications written by experts given to suppliers

  4. Why do we have these problems? Difficult to articulate Complex in nature Difficult to communicate Domain Knowledge

  5. Let’s run a mini experiment

  6. In a “second degree price discrimination” strategy, price varies according to the quantity demanded. Larger quantities are available at a lower unit price. Additionally, sellers applying this strategy are not able to differentiate between different types of consumers. Thus, the suppliers will provide incentives for the consumers to differentiate themselves according to preference. This also applies to product quality. This is an Abstraction This is an Example Airlines often offer multiple classes of seats on flights, such as first class and economy class.

  7. Results in Cognitive Psychology and Software Engineering Examples + Abstractions Effective Knowledge Transfer

  8. Modeling Example-Driven Model = Abstractions + Examples

  9. Of Course Examples Help! Capture intuition Relate to existing knowledge Show the intended usage of an abstraction

  10. However… • When exactly do examples help? • How much can they help? • What types of examples help? • Which stakeholders benefit the most from examples? • Where is a PROOF that they help?

  11. Provide the Missing Empirical Evidence and Guidelines Objective

  12. Controlled Experiment “Analyze the effects of using explicit examples on structural model comprehension in the context of knowledge transfer to model receptors who are novices to the application domain.”

  13. Two Groups of Participants Model Abstractions Model Abstractions Examples! Experimental Procedures & Requirements List Experimental Procedures & Requirements List EDM Control (13) (12)

  14. Instrumentation & Actors Map SME Model Creator Model Abstractions Experimental Procedures & Requirements List Examples!

  15. Treatments and Metrics Sobeys Object Diagram EDM Control

  16. 23 classes, 34 associations, and 8 constraints

  17. (EDM Only) 4 positive and 2 negative examples

  18. Treatments and Metrics 1) Diagram Completeness Sobeys Object Diagram EDM Control

  19. Diagram Completeness • Mean (Control) = 49.2 % • Mean (EDM) = 88.1 % • p-value < 0.0002 • Maximum score for Control is lower than the minimum score for the EDM + 37% increase

  20. Treatments and Metrics 1) Diagram Completeness 2) Diagram Mistakes Sobeys Object Diagram EDM Control

  21. Diagram Mistakes • Mean (Control) = 15.1 mistakes • Mean (EDM) = 3.5 mistakes • p-value < 0.0053 • Maximum number of mistakes for EDM is equal to the minimum number of mistakesfor Control - 80% decrease

  22. Treatments and Metrics 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence Sobeys Object Diagram EDM Control

  23. Participants Rated Themselves Difficulty Confidence 1 (unsure) to 5 (very sure) Control: 2.33 EDM: 4.08 + 35% increase 1 (very easy) to 5 (very difficult) • Control: 3.83 • EDM: 2.46 -27% decrease

  24. Treatments and Metrics 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence Sobeys Object Diagram EDM Study Questions Control

  25. Treatments and Metrics 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence Sobeys Object Diagram 1) Task 2 Completeness EDM Study Questions Control

  26. Task 2 Completeness Max 24 • Mean (Control) = 13.6 • Mean (EDM) = 21.2 • p-value < 0.0012 + 31% increase

  27. Treatments and Metrics 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence Sobeys Object Diagram 1) Task 2 Completeness 2) Efficiency EDM Study Questions Control

  28. Task 2 Efficiency • Mean (Control) = 0.7 • Mean (EDM) = 1.2 • p-value = 0.00082 + 71% increase

  29. Treatments and Metrics 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence Sobeys Object Diagram 1) Task 2 Completeness 2) Efficiency 3) Perceived Difficulty 4) Perceived Confidence EDM Study Questions Control

  30. Participants Rated Themselves Difficulty Confidence 1 (unsure) to 5 (very sure) Control: 3.00 EDM: 4.15 + 23% increase 1 (very easy) to 5 (very difficult) • Control: 3.25 • EDM: 2.08 -23.5% decrease

  31. Treatments and Metrics 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence Sobeys Object Diagram 1) Task 2 Completeness 2) Efficiency 3) Perceived Difficulty 4) Perceived Confidence EDM Study Questions 1) Number of domain-related questions Control Participants Questions

  32. Number of Domain-Related Questions • Control: 10questions/session • EDM: 1 question/session - 90% reduction

  33. Treatments and Metrics 1) Diagram Completeness 2) Diagram Mistakes 3) Perceived Difficulty 4) Perceived Confidence Sobeys Object Diagram 1) Task 2 Completeness 2) Efficiency 3) Perceived Difficulty 4) Perceived Confidence EDM Study Questions 1) Number of Domain-related Questions Control Participants Questions UML Class Diagram Examples Feedback

  34. Understanding Without Examples? EDM: “The class diagram was huge, I didn’t know where to start reading to understand, so I looked at the examples one by one while looking at the model abstractions simultaneously.”

  35. Classes Most Difficult To Comprehend

  36. How Were the Examples Helpful? • Example 2: • Regular product vs. product with an offer associated • Application of the BonusMechanic class • Example 4: • Propagation of points across classes and the sequence for accumulating points • Account’s “pointsBalance” updated with Bill’s “totalPoints” attribute

  37. How Were the Examples Helpful? • Example 5: • Redemption process • Example 6: • Violation of the constraint prohibiting accumulation and redemption for the same bill instance

  38. Are Both Positive and Negative Examples Needed? • “Yes” 12/13 EDM participants • No comments about the number and content of examples

  39. Discussion

  40. Examples are mostly needed by Novices • FixedPriceOff vs. FixedPercentOff • familiar • RedemptionMechanic, RedemptionChannel, andBonusMechanic • foreign

  41. Having a Variety of Examples Useful when • Having more than one association linked to a single class

  42. Having a Variety of Examples Useful when • Having more than one association linked to a single class • Complex constraints

  43. Having a Variety of Examples Useful when • Having more than one association linked to a single class • Complex constraints • Propagation of attributes through classes

More Related