CBR in Software Engineering

1. CBR in Software Engineering Brought to you by Chris Creswell (and Klaus-Dieter Althoff)

2. Outline • Chapter 9 of Case Based Reasoning Technology • “CBR for Experimental Software Engineering” • By Klaus Dieter-Althoff, Andreas Birk, Christianne Gresse von Wangenheim, and Carsten Tautz • Another paper on the subject: • “Effective Experience Repositories for Software Engineering” • By Kurt Schneider and Jan-Peter von Hunnius

3. CBR for SWE – Goals • Present infrastructure for learning in the software domain using CBR • Use CBR to build an experience base • Outline additional uses of CBR for ESE (that's experimental software engineering)

4. Definitions • Software Engineering -- concerned with the definition, refinement, and evaluation of principles, methods, techniques, and tools to support: • Individual aspects of software development and maintenance • Planning of a software development project • Performing development, project management, and quality assurance • Assessing the performance of these methods, techniques, and tools

5. Motivation • Software development needs to improve • Software quality • Development time • Development cost • Software projects often fail to meet the projected goals for each of these • Why? • Software complexity has increased dramatically • Processes haven’t changed much since the old days

6. How to improve • Better software comes from a better development process • Two ways to improve the process • Top-down Methods • Evaluate software development organizations and their processes • Capability Maturity Model (CMM) • Five levels of maturity • Evaluate by comparing to a reference best-practice model • Bottom-up Methods • Use continuous learning and reuse of experiences • Quality Improvement Paradigm/Experience Factory (QIP/EF) • We'll study an application of CBR to implement this

7. ESE – What is it • Experimental Software Engineering • A branch of SWE that does reasearch to improve software development by experiment • Built on the QIP/EF approach • The QIP is a 6 step procedure with 3 phases • Phase 1 – Planning – where the process could benefit from CBR • QIP1) Characterize the initial situation • QIP2) Identify goals • QIP3) Develop a plan • Phase 2 – Execution • QIP4) Execute the plan • Phase 3 – Evaluation • QIP5) Analyze the performed actions • QIP6) Package the lessons learned into reusable artifacts

8. The Experience Factory – What is that • “The Experience Factory (EF) is a logical and/or physical organization that supports project development by analyzing and synthesizing all kinds of experience, acting as a repository for such experience, and supplying that experience to various projects on demand.” • Conducts steps 5 and 6 (phase 3) of the QIP

9. Current uses of CBR in SWE • Current ways CBR is used in SWE: • to introduce more flexibility in software reuse, in combination with object-oriented techniques • capturing and formalizing best practices • effort prediction • requirements acquisition • It could do much more for ESE

10. Extending CBR for use in an EF for ESE • CBR works well for fine grained decisions • It has many goals in common with the EF • Research overlaps in the subjects of reuse and similarity based retrieval • To be applied to large scale organizational aspects, CBR must be extended

11. A few more terms • Technology application domain – describes the situations in which a technology can be applied successfully • Here technology means SWE technologies like lifecycle models, design methods, configuration management systems • Technology application – a combination of an overall task, the specific goal of the application, what technology was used, and the context/domain of the application

12. Case representation for ESE • We'll represent cases as technology domain models (TDMs) • TDMs consist of intuitive descriptions of context characteristics each containing a context factor • e. g. characteristic: experience of developers, factor value: high

13. Example of a TDM • Characteristic: Value • Technology: Reading by stepwise abstraction • Task: Code verification • Goal: High reliability of software product • Programming language: C • Amount of reuse: less than 40% • Experience of developers: average • Size of project: small (on-site) • What problems might there be with this representation?

14. TDMs continued • So as we have seen, TDMs are a qualitative experience representation • Get them by talking to software professionals using “knowledge acquisition techniques” • Assure the validity of the knowledge by tracing it back to concrete experiences

15. The Decision Process • Use a case base if there is one available • It should contain domain models for multiple technologies that share the same task and goal • This allows decision makers to pick a technology for the problem most similar to theirs • Decision process steps: • 1) Determine the task and goal for the project • 2) Characterize the project in terms of domain factors • 3) Perform similarity-based retrieval to find technologies whose TDMs are sufficiently similar to the present project

16. So what do we need • A tool that performs similarity based retrieval on TDMs • The tool should allow decision makers to • Explore a filtered, prioritized set of cases to allow them to make informed decisions during planning • Since TDMs are so qualitative the probability of exact matches in similarity is very low • So the tool uses an interactive decision method

17. Maintaining case bases • This maintenance can be described using the CBR task decomposition by Aamodt and Plaza • That's the big tree that starts with retrieve, reuse, revise, and retain • We tailor it to the principles of ESE • The QIP described previously

18. The Result: MIRACLE • MIRACLE – Model Integrating Reuse And Case-based reasoning for Lots of software engineering Experiences • It’s a combination of the CBR task decomposition and the QIP model • Sounds like a big deal, but there’s only one real difference between this and the CBR decomposition

19. MIRACLE

20. MIRACLE – elaboration • Specify – specify the project • Search – search for possibly relevant planning information • Choose – choose planning info from projects that are similar to the forthcoming one • Select – select suitable goals and models for them • Copy – copy the most suitable models or create new ones

21. MIRACLE – elaboration • Adapt – modify models • Apply – perform project and collect data • Assess – evaluate success of applying models using the collected data • Improve – detect weaknesses and find out how to improve the models • Extract – identify information to store (lessons learned) • Index – set up suitable characterization schemes for and relationships between the items to be stored • Integrate – Update the experience base

22. MIRACLE – analysis • The only real difference from the CBR decomposition is the inclusion of the apply task, and a few names have been changed • Let’s look at how MIRACLE maps to the CBR task decomposition and the QIP

23. Specification 1. Retrieve 5. Retain New Slides • Repository of Presentations: • 5/9/00: ONR review • 8/20/00: EWCBR talk • 4/25/01: DARPA review Slides of Talks w/ Similar Content 4. Review New Case Talk@ cse395 Revised talk 3. Revise First draft 2. Reuse Reminder – here’s what the CBR cycle looks like Example: Slide Creation - 9/12/03: talk@ cse395

24. Mapping MIRACLE to CBR Retain Retrieve Revise Reuse

25. Mapping MIRACLE to QIP QIP 6: Package QIP 1: Characterize QIP 5: Analyze QIP 2: Set goals QIP 3: Choose models QIP 4: Perform

26. Walking through the MIRACLE process • Retrieve objects with characterization similar to the one specified • Must be able to cope with incomplete information • CBR can do this as long as we • Specify the needed objects in as much detail as possible • Retrieval corresponds to steps specify, search, choose, and select of MIRACLE

27. Similarity • The set of candidate cases must be reduced after searching – this is step choose • This is done using a global similarity measure and taking into account all specified features (whereas search only uses one feature) • A global similarity metric is something like the Hamming Distance we saw in class last week

28. Copying and Adapting • Selection should calculate costs for • Modifying an object • Creating a new one • Using these estimates, we decide if it is worthwhile to reuse an object • If a new object is to be created, a template should be used and the result modified • So either way, something is copied and adapted

29. Copying and adapting continued • Two ways to adapt • Transformational reuse – modify the existing object itself to suit the new problem • Derivational reuse – don’t reuse the existing object itself, instead reuse the process that created it • Let the process be a guide to creating the new object, and modify it as necessary along the way • We will study case adaptation in more detail later in the course • After adaptation, the object must be validated

30. Validating the object • This can be done in several ways • Apply the object and assess its performance after the fact • Review by experts • Test by model – use a formal model to determine the objects suitability

31. Improving the revised object • If it worked perfectly, then just characterize and store the new case • If the object failed, we can learn from it • Reasons for failure include • Wrong specification – specification was incorrect, the other steps were fine • Wrong application – the reused object was not applied as planned • Wrong adaptation – all the other steps were correct but the adaptation was faulty • After we know what went wrong (if anything), improve the object accordingly

32. Retaining the object • Store the new insights for reuse • Experience base should be updated regardless of whether the reused object worked successfully • When storing info • The object can be stored as a new object or a modification of an existing one • Objects other than the one reused can be stored • Information related to the reused object can be stored (e. g. its characterization, assessment results, etc.) • This is called the experience package • That’s it for MIRACLE

33. Current status • The authors are “constructing an experience base focusing on case representation for reusable objects, definition of similarity measures, and the representation of general knowledge” • Translation – it doesn’t work yet, nobody has implemented MIRACLE completely • But, that’s not all …

34. Current status • The authors are also focusing on creating a metaCBR tool • They have selected 50 features to characterize CBR systems and allow: • CBR practitioners to describe their system quickly • Developers to choose a system, then give feedback, thus creating more experience • The cases were developed with a commercial tool (CBR-Works from TecInno) • They are available online • http://www.iese.fhg.de/Competences/QPE/QE/metaCBR.html

35. Conclusion • We’ve seen MIRACLE, a model that merges the CBR cycle and the principles of the QIP • Case representation with TDMs • “We believe that the experience factory concept offers a kind of infrastructure that is helpful for CBR applications not only in ESE, but in any kind of applications fielded in industrial or other business environments. Thus, an experience factory operationalizes a CBR system in industrial/business environments, while CBR offers computer based support on a very broad level.” • Questions about this chapter before we move on?

36. “Effective Experience Repositories for Software Engineering”by Kurt Schnieder and Jan-Peter von Hunnius • The product of several attempted experience repositories at Daimler-Chrysler • They have developed a set of success factors for experience repositories • Motivation: the difficulty of producing an effective tool is often underestimated, some guidelines are needed to develop a successful repository (in one try, these guys failed several times)

37. Key success factors for experience repositories • 1 – User guidance • 2 – Usability • 3 – Process conformance • 4 – Feedback mechanism • 5 – Maintainability

38. Key success factors for experience repositories – elaboration • User guidance – the system should provide direct support for the user’s work • Usability – the user should not be required to learn much to use the system • A whole lot of work has been done on these first two • Process conformance – the system should conform to the task that it is designed to support. • “By making an improved process the center piece and backbone of an experience repository, navigation, orientation, and search are highly improved.”

39. Key success factors for experience repositories – elaboration • Feedback mechanism – the repository should enable and encourage feedback through several channels • Experience calls for continuous update and rework • Maintainability – new feedback needs to be analyzed and integrated into the existing contents

40. Lessons that support these key qualities • Be specific – don’t be vague in structure of repository • One process at a time – a repository should concentrate on a single process at a time • Usability counts – if users have a hard time with the system, they won’t bother with it • One-way communication fades away – the system cannot be hardcoded, it has to be able to improve

41. Fast repository assessment • The authors developed a checklist to evaluate repositories quickly • A few example questions from the list • User Guidance • Is a new user properly introduced into the area of interest? • Can the community of practice discuss using the repository? • Process Conformance • Is the defined process the “Best Known Practice”? • Does feedback have an impact on the process?

42. Final point • “Repositories are important, but they are not sufficient to make a software organization effectively learn from experiences. Without a learning attitude and some appreciation for continuous process improvement, even the best repository will not make the experiences ‘fly’.” • Questions about this paper?