The Software Matrix An Architecture for Software Salvage

1. The Software MatrixAn Architecture for Software Salvage RiddhimanGhosh Advisor: Dr. James Fawcett Master of Science Thesis Dept. of Electrical Engineering and Computer Science, Syracuse University December 15, 2004

2. Software Salvage • Salvage: lifting of a significant block of existing systems and inserting them into a newly developed system.

3. Overview • Our work centers on significantly managing the problem of software salvage. • We propose an architecture for system construction to simplify salvage and use message passing and mediator structures. • Message passing has been used before to communicate between processes and machines. • Our contribution is to show that Mediated Queued Message Passing at the module level is an effective way to support software salvage • Adding the mediator structure makes assembling a new application with salvaged parts much easier • Our work shows we gain a lot of flexibility at an acceptable performance cost.

4. Introduction • Approximately 100 billion lines of source code at work in the world today. • Large fractions of code in systems are functionally equivalent. • However systems are often constructed without leveraging existing code bases – less than 15% of new code serves an original purpose. • “Reinvention of the wheel” in the software industry.

5. Introduction • Software construction is needlessly error-prone and expensive. • We’re maintaining multiple copies of essentially the same software. • Time and cost of developing, testing and documenting a piece of software is multiplied by the number of equivalent copies in existence.

6. Software Reuse • Study of reuse – recycling of software assets – has been an important branch in the software engineering discipline. • Effective reuse promises • Reduced development and maintenance costs • Gains in development schedule and quicker time-to-market • Increased robustness and quality • Not a new idea.

7. Software Reuse • Systematic Reuse: “an institutionalized organizational approach to product development in which software assets are intentionally created or acquired to be reusable” • However, few organizations practice systematic reuse, in spite of a recognition of its potential benefits. • Reuse is hard!

8. Gap between “theory” and “practice” in Reuse “ Not all the “theoretical oriented” and “sophisticated” solutions presented by researcher[s] have ...thrilled the practitioners... – Zand, M. ...the reuse community has worked on complex technologies and methods with high ceremony, yet most of the software community seems to be looking for simpler solutions... High ceremony methods require an organization with high process maturity to achieve success. – Griss, M. ”

9. Software Salvage • What we seen in the industry is not necessarily software reuse, but rather “software salvage” • Salvage: lifting of a significant block of existing systems and inserting them into a newly developed system. • Radar Systems Department, General Electric Company (Syracuse, NY) routinely attempted salvage in the building of a new radar.

10. Reuse vs. Salvage • We make a distinction between the terms software reuse and software salvage. • Reuse connotes immutability • the individual pieces meant for reuse cannot be modified; they’re designed and implemented to be adaptable but with no intent to change even a single character of source code. • Salvage makes no guarantee of immutability • very often source code of individual pieces being salvaged is modified.

11. Problem • Effective salvage is difficult to accomplish • The large pieces we want often have many dependencies on parts we don’t want • Requires expensive changes • Think of salvage as analogous to transplanting the heart from one living organism to another! Similar problems, when we pull out a part from a software system due to the connectedness inherent in typical software.

12. Problem It is difficult to salvage existing parts of systems and build new systems from them, with ease.

13. Goal • Enable leveraging of major parts of systems • Eliminate or reduce changes required by salvage so that salvage moves closer to the reuse model (immutability), without requiring high-process organization. • Simplify software salvage to make it a useful paradigm.

14. Prior Approaches • Two major (and fairly recent) approaches towards encouraging reuse have been • object-oriented reuse • component-oriented reuse.

15. Object-Oriented Reuse • Reuse has been one of the classic motivations of object-orientation • Object oriented theory • mandated discipline in writing code through best-practice guidelines such as separating interface from implementation • Encouraged reuse through inheritance, composition, parameterization

16. Object-Oriented Reuse • However OO technologies didn’t quite engender the reuse revolution that was hoped for. • Only OO libraries widely used are user-interface frameworks (such as MFC) and libraries for data structures. • OO techniques have made compiler libraries an effective means of reuse but business organizations have had a harder time getting leverage from their pre-existing software assets only through the creation and consumption of OO libraries.

17. Object-Oriented Reuse • The definition of an object is purely “technical” • Defined as an encapsulation of state and behavior • No direct mapping to the physical unit that is actually deployed, versioned and potentially reused. • Objects don’t fit the “salvage” model well • In terms of granularity, they are of an inappropriate size to be mixed and matched. • They are the size of grains of sand, while what salvagers are typically looking for in building systems, are bricks.

18. Component-Oriented Systems • The logical next step was the notion of components • a coherent package of software implementation that can be independently deployed and composed with other components. • Defining feature: system can be made better by only updating components • of coarser granularity than objects • independent and deployable implies executable or loadable code

19. Component-Oriented Systems • There are several component technology offerings • COM, CORBA, JavaBeans/EJB, .NET Components • Adopting a component-oriented approach (rather than only concentrating on programming language as in OO theory) is beneficial. • But the existence of these component technologies has not solved the problems that plague software salvage, discussed earlier • Packaging techniques alone are not enough

20. Our Approach • We are of the view that components are useful only if there is a framework that actively supports and promotes their reuse. • .NET and J2EE have framework support, however it is focused on generic industry problems (network communication, web publishing, application security, etc.) • We seek to support vertical, perhaps proprietary, applications, where building a huge framework is impractical from a return on investment point of view.

21. Our Approach • Make salvage easier by • viewing applications as compositions of different pieces • having a framework of collaborating pieces from which applications can be composed dynamically • Try to achieve benefits of the “Software-IC” model— a plug-and-play approach to software construction.

22. Our Approach • We are limiting the hard problem of general reuse to a smaller domain—of salvage within an organization, to be used in the construction of modest-sized systems (about 200,000 lines of code). • To address this domain would be to provide solutions of value to the small and medium-sized software shops that have been resistant to adopt systematic reuse.

23. Software Matrix • The Software Matrix is a framework that actively supports and promotes the salvage of components. • We focus on the reuse of major blocks of code rather than low-level functionality • By employing message-passing and mediator structures and by supporting the discovery of needed types, we’ve built a pluggable architecture that can gracefully adapt to salvage operations.

24. Software Matrix • In particular, the Matrix is a runtime infrastructure that acts as a substrate into which individual pieces of an application – different blocks of code – can plug.

25. Software Matrix From these “plugged-in” individual pieces the Matrix dynamically composes applications.

26. Software Matrix • This infrastructure was named the Software Matrix in order to connote a very structured pattern of building software. • System elements are embedded in this substrate – a matrix /collection – through which applications are dynamically composed. • The image of endless banks of incubators of humans in a matrix, as from “The Matrix” motion picture, isn’t entirely unintentional, if somewhat flippant.

27. Cells • The individual blocks of code that plug-in to the Matrix are called “Cells” and are the building blocks out of which applications are built.

28. Cells • Cells represent the unit of composition and reuse in our system.

29. Cells • An application is built through the collaboration of Cells • Cells communicate with each other strictly through messages (we use XML encoded messages)

30. Cells • One of the problems of salvaging: • extracting parts of monolithic (or very tightly-coupled) applications. • We are insisting on loose coupling • Enforce a separation of concern between Cells by inserting a message bus between them

31. Message Passing • Advantages of Message Passing Interface (MPI) over Procedure Call Interface (PCI) • MPI is a universal interface. All entities using MPI use something equivalent to GetMessage() and PostMessage(msg). So any Cell providing MPI is guaranteed to be plug-compatible with another Cell accepting MPI • Messages are self describing command and state carriers. If a Cell does not understand all elements sent to it, its free to ignore what it does not understand. PCI has no such support. • Messages can be easily intercepted and inspected. Therefore MP systems are easier to debug • Message Passing makes it easier to use the Mediator pattern, since the Mediator does not need to know a lot of interfaces, just the small MPI.

32. Mediator Pattern • The use of the Mediator pattern is significant in the Software Matrix • About the Mediator pattern Gamma et al. say: • Thus, proliferating interconnections between different partitions of a system reduces the potential for salvage. The Matrix acts as a mediator – the different Cells only know about the mediator. “Mediator promotes loose coupling by keeping objects from referring to each other explicitly, and it lets you vary their interaction independently. …lots of interconnections make it less likely that an object can work without the support of others”.

33. A closer look at Cells… • Every Cell contains: • a message queue: holds request and response messages during collaboration with other cells.

34. A closer look at Cells… • Every Cell contains: • a capability list: used to advertise the capabilities of a cell to other cells, via the Matrix. (E.g. SU.Math.Convolution) The capability list is used by the Matrix in order to discover the right cells for system construction.

35. A closer look at Cells… • Every Cell contains: • a globally unique identifier (GUID): Cells can be uniquely identified using a GUID. This is used by the Matrix for several operations such as discovery and registration.

36. A closer look at Cells… • Every Cell contains: • functionality: Cells also contain the functionality that allows them to be considered as software assets with potential for reuse.

37. ICell A closer look at Cells… • All cells subscribe to a common protocol (ICell) that specifies how to: • register and un-register with the Matrix • advertise capabilities • send and accept messages • collaborate with other cells (could be synchronous, asynchronous or one-way) MyCell

38. A closer look at Cells… • Every cell also has an entry-point (start), and is given a chance to execute once it is plugged-in. • This entry-point (empty/non-empty) decides whether a cell will be only a passive ‘server’, or itself actively seek collaboration from other cells.

39. Example • Sample application: needs to read data samples from an input file, perform a signal processing operation on the data (e.g., filtering), plot the results of the operations on the system display, and log the results to a file. • The major pieces of the application would be responsible for: • file operations • signal processing • graphical plotting

40. Example • These would be written as cells and plugged-in to the Matrix. • The Matrix would then assume the responsibility of constructing the application from these individual cells. • In order to perform its task a cell may need services of another cell. But cells do not explicitly bind to other cells – they only specify what message type they need handled, and the Matrix discovers cells capable of handling that message. • If no suitable cell found, a “not supported” message is generated.

41. Example

42. Example Sequence of events in the construction of the sample application

43. Example • We see here: Dynamic composition – we are building a system from pieces that exist on the Matrix at runtime. Matrix takes care of compositional aspects (as opposed to only computational aspects) of software. Matrix automatically connects the right pieces at run-time without having to bind to anything explicitly at compile-time. Software is now amenable to salvage operations. The very same Cells could be used to build other applications.

44. How do cells “plug-in”? • Cells are implemented as “plug-in” modules, and are realized using .NET components. • The Matrix uses the ‘reflection’ and ‘late-binding’ (Fusion) features of the .NET framework to discover and register plug-ins. • Given an .NET assembly, the Matrix will reflect over the contained types to determine which of them implement the ICell interface in order to recognize valid plug-ins.

45. How do cells “plug-in”? Throughout its lifetime, the Matrix monitors the plug-in directory, in order to discover new cells Valid cells are ‘registered’ and available for system construction.

46.  In order to enable salvage, pieces of an application (at the time of writing it, or existing pieces) are wrapped in a Cell. Create a wrapper that inherits from ICell (e.g. we wish to create FileManager Cell responsible for common file operations ) FileManager : ICell { ... } Steps to take advantage of the Matrix

47.  The Capability List of this FileManager Cell is then populated to indicate the types of messages it is capable of handling. Steps to take advantage of the Matrix CapabilityList.Add(“SU.FileManager.Files.Read”); CapabilityList.Add(“SU.FileManager.Files.Write”); CapabilityList.Add(“SU.FileManager.Files.Search”); CapabilityList.Add(“SU.FileManager.Files.Compression”);

48.  Override the “process” method to add appropriate message processing Basically you specify what is to be done in response to a particular message type – delegate calls to your implementation. Steps to take advantage of the Matrix

49.  Compile, and copy the resulting binary into the “plug-in” directory of the Software Matrix. The Matrix will automatically detect and register the cell, and it will be available for composition. Steps to take advantage of the Matrix

50. If a cell wishes to use other cells (or is a program executive for instance), it will probably say: ... Result = syncSend( "SU.FileManager.Files.Search", Params); ... Here it is trying to locate a cell that can handle the named message type. Steps to take advantage of the Matrix