Activity 9: Decide on a strategy for persistent data storage and management

1. Activity 9: Decide on a strategy for persistent data storage and management Here we need to decide on how we are going to store persistent data. Persistent data is data that has to be retained between distinct executions of a system. In other words, data whose life-time is longer than a single execution. Data storage strategies depend on the nature of data. Choices are: Flat files Indexed files Relational databases, and Object-oriented databases Each has its own distinct advantages and disadvantages that have to be weighed against one-another.

2. Flat files are simple abstractions provided by virtually all programming languages through the operating system. The data format is usually very simple (often only a sequence of bytes) and data manipulation must be taken care of within the application. This is both an advantage and a disadvantage. The simple structure of the flat file allows for various tailoring of data storage and retrieval to suit a particular application. This could be advantageous as it can yield high levels of optimization. This is also a disadvantages as given the simple structure of the file abstraction, much of the work of data storage, retrieval, formatting and presentation must be explicitly done in the application; often in multiple places.

3. Flat files are therefore suitable in situations where: • There is a stable, specialized and simple data storage requirement By stable requirements we mean that it would be unlikely that the content, format, volume, association or type of the data storage or retrieval is altered. For example a file containing a look-up temperature conversion table. By specialized we mean specific to one particular operation or application. For example historical sensor readings of a particular sensor, or a core dump file. Simple means that the information content of each unit of information stored is small. In other words the data types are not complex. For example a simple number or a simple ordered-pair.

4. The information unit granularity is coarse • Information density is low • There is low requirement for complex manipulation This means that there are a large number of bytes required to store one unit of information. For example a map, an image or a CAD model. This means that the information value of each unit is binary or near binary. For example an access log. This means that the information is retrieved and used in the same way or nearly the same way as stored. For example if a unit (a record) is written in one operation (write_record(a_record)) it is also read in the same way through a complementary operation (read_record(a_record)). There is no need to mix or group records or parts of records. The retrieval is at unit level rather than the set of units level.

5. The simple format of a file abstraction often allows for certain optimizations. The most popular of these is indexing where special operations and structures are added to the file abstraction to provide an indexed file abstraction. Indices make retrievals made using the indexed variable much faster. An example might be a telephone directory where the file might be indexed on last name of the owner. Use an indexed file when there is one or a very small number of preferred search variables.

6. Relational databases are at a much higher level of abstraction than the file. With this comes certain advantages and disadvantages. The disadvantages are that direct and specialized manipulation is no longer easily done as there is a further level of abstraction (called the schema) between the user and the storage mechanism. This might constrain certain specialized, simple data storage requirements such as simple reading and writing of simple records. Also, there is a general speed and cost penalty that comes with adding layers of abstraction such as a database management system. The advantage is that schema introduces a set-at-a-time data manipulation capability. The database system now takes over much of the effort in writing, retrieving, grouping and presenting records.

7. Use a relational database when: • There is need for complex often set-level manipulation of data units. • There is need to accommodate and relate large tables of similar data • More than one application, sub-system or system needs to utilize the data, often concurrently. A relational database takes care of concurrency issues.

8. The relational model breaks encapsulation as it segregates data from operations on data. Object-oriented databases are at a higher level of abstraction from the relational model as they store objects and their associations together. The advantage here is that each element can now be treated as an object and manipulated as such. In addition relationships and facilities that were not possible to model or were extremely difficult to do so in the relational model now become possible. For example we can mention very complex user defined types, inheritance, abstract types, and aggregate types.

9. Use an object-oriented database when: • The design model is object-oriented or object-based. • Tabulation is not possible, permitted or is at a significant loss of information, particularly abstract associations such as sub-typing. • There is complex relationships (associations) between objects in the model • The object-sets are not very large

10. Activity 10: Decide on choice and use of middleware and distribution software This decision may have already been taken in that there may be a corporate standard that all application development must follow.

11. Activity 11: Decide on access control mechanisms and policies Access control is deciding who can use or manipulate what part of the system and under what conditions. The first step in access control is to create a mechanism to identify each user or class of users so that: • Only those allowed to interact with the system can do so • Those users who are allowed to interact with the system are known to the system. • Each user or group is protected against accidental or malicious access to the environments under their control by others This is called authentication

12. User access authentication is traditionally achieved via a “security pass” mechanism. This is a combination of a user-name that identifies the user to the system, and an item of information or pattern that is to be uniquely known or available to the user, something such as a password, a retinal pattern or a finger or voice print. The individualized access codes are usually encrypted and retained in an access repository, for example the unix password file. The access and authentication mechanism lends itself to a repository based design style where the access data is retained in a repository and the authentication, encryption and management modules utilize it.

13. Once users are made known to the system we need to now determine which user should have access to various parts of the system. Again, this can be done through a central mechanism based on the repository style of design where a global access table acts as a repository. Every user and every operation of every class registers with this repository. Looking up this table will allow us to assess whether a particular user has a particular access capability or not. Alternatively, we can attach an access control list to each class. Where access is allowed, the list shall contain an entry for each user with respect to each operation of that class. Similarly it is possible to have a list of all classes and operations therein attached to each user.

14. These access control mechanisms work well when access knowledge is a priori known. In situations when access rights are to be dynamically figured (at run time), we need a different mechanism. We need to model the access as an independent association between the user and the target service or object. This is the same design problem as with the case of sub-system separation discussed earlier in the case of transport data isolation. The design pattern proxy which was of help there can also be used here again.

15. Activity 12: Decide on an appropriate control flow mechanism The choices are as presented before. These are basically procedure driven or event driven. For concurrent systems multi-threaded control flow must be considered. Particularly in the case of event driven systems, it is prudent to segregate and localize the control flow mechanism from the rest of the application. This assists with flexibility and maintainability of the system. In conjunction with the MVC style of architecture, the command design patterns can be gainfully utilized to achieve this segregation in an orderly and flexible fashion. We shall discuss this later.

16. Activity 13: In view of all the above, and particularly item 4, identify and “earmark” reuse libraries In a mature development environment reuse is well practiced. Capability and schedules are planned in full view of the potential for reuse of existing intellectual property. By this we mean more than just reusable code. It also includes analysis patterns, application frameworks, design modules, reusable sub-system level components, as well as reuse class and other code libraries. Whilst it is prudent to consider reuse throughout the process, at this stage, we must consider specific reuse libraries that are to be used. Issues of compatibility and availability become important.

17. Similarly, it should be noted that reuse libraries are to be maintained and enhanced, consideration as to how this particular project may contribute to the reuse library should be given here. It is sometimes even prudent to re-work the design into a more general and less optimum version if doing so yields greater future reuse opportunities. This is of course a judgment call, and a difficult one at that, and must be made on a case by case basis. Use of abstract types, genericity, and use of established architectural styles and design patterns goes a long way to assist with future reuse.

18. Activity 14: Evaluate and revise the design based on quality criteria and design goals We now must evaluate the design as it stands. This does not mean that we refrain from evaluation throughout the other activities. It just means that an explicit evaluation is called for at this stage. Design is by nature an iterative activity. It is micro-grained. It requires methodical technical knowledge and discipline and artistic, inspired, opportunistic ingenuity to work hand in hand. This means that many aspects (if not every aspect) of design influence many other (if not every other) aspect, explicitly or otherwise. The designer must constantly ask:

19. “How is THIS decision going to help make this product (this overall design) a better product?”; “What are the consequences of making THIS decision?, what, if anything, must change?”, “Will the balance of these changes lead to a better situation, one that takes us closer to our objectives?” “How can I show that this is in fact the case?” These questions will help lead the designer through the design process. They are part of the personal design evaluation process. However, overall evaluations at logical points in the process will also be helpful. These explicit evaluations are called design reviews.

20. Design evaluation must be against specific criteria. These must directly relate to both the quality criteria expected of the product and the specifically stated or implied design goals (which themselves should be a specific interpretation of the product quality expectations). It is best that these design quality criteria are measurable and a-priori stated including a statement of the conditions under which such evaluation would take place. From a client view point, a good design document should be adequately: Correct Complete Consistent From a practical view point, a design also should be: Cost-effective Comprehensible

21. There are several techniques of design evaluation based on the nature of the design itself. These include: Formal correctness proof: This is when there is a design is formally derived from a formal specification. Under these circumstances, we can prove the correctness of that derivation against the specification. This activity will demonstrate the correctness, consistency and to a degree completeness (as much as the specification is complete) of the design but does not go far in demonstrating implementability, cost-effectiveness and understandability.

22. Design review: This is a process where the design is put through a peer review process . Design review processes (often simply called “Design Reviews”) are specific software process techniques that are conducted in accordance to a pre-agreed defined process that have been demonstrated to be effective. Process descriptions and templates for conducting such reviews exist in the literature and are often excellent examples of good process specification. During a design review the reviewers might only evaluate the design product, or the design process or both. Usually both are considered. Design evaluations may be considered both a preventive as well as a corrective defect management technique.

23. Architectural (system) Design Documentation As design architecture decisions are made, they should be documented in the form of diagrams, models, notes, mathematical expressions, etc. Once an overall design architecture is obtained and stabilized it is must be coherently documented by bringing all the interim documentation together and ensuring that they represent the current situation. As mentioned before, this document must be correct, consistent, complete. It should also be concise and comprehensible. There are many templates available in the literature. One such format is as provided below:

24. System design template: • Table of contents and list of figures and tables • Preambles, caveats conditions • Description and definition of special notation used, values of constants, etc. • Executive Summary • Synopsis of System • Current situation and environment (which the system is to improve) • Operating environment and conditions (e.g. specialized hardware, security, support ) • Outline of process to be followed • Current high level architecture and system description • Proposed high level architecture and system description • Expanded system architecture:

25. 11.1 Overview • 11.2 Identifying functional and non-functional design goals • 11.3 Physical and logical topology and major sub-systems • 11.4 Sub-systems descriptions and sub-system architectural styles • 11.5 Storage mechanisms and data transport • 11.6 Distribution and middleware • 11.7 Access control • 11.8 Flow control • 11.9 Reuse issues

26. Post-amble or concluding remarks • Appendices: • 13.1 Listing of important use-cases • 13.2 All other diagrams and supporting materials • 13.3 Glossary of terms and definitions • 13.4 Traceability to the Requirements Specification Document • 13.5 Index • 13.6 References

27. Implementation The design obtained thus far is in terms of communicating modules. In other words, the components of the design are composed as encapsulated elements which request services of each other. If followed correctly, the outcome of this design process should yield an object-oriented design. Also, given the assertion that a design should be a complete or nearly complete and implementable solution to the problem at hand, it is implied that implementation is not a constructive but a translational step. As a translation activity, implementation (including programming) should therefore be a trivial step.

28. Implementation In fact, assuming the availability of a complete design model, implies that the translation into an executable form could be even automatic as all the information required to implement the solution is already available. It is, however, not necessarily so. An object oriented design may be implemented in an object-oriented programming language as an object-oriented implementation (set of executable programs). However it is possible to do otherwise. For example, it is possible to implement: An object oriented design, in an object oriented language, as a non- object oriented implementation (yes it is possible!).

29. Implementation An object oriented design, in a non OO language, as an-OO implementation An object oriented design, in a non OO language, as a non-OO implementation A non-OO design, in an OO language, as a non-OO implementation A non-OO design, in a non-OO language, as a non-OO implementation. Other combinations although logically enumerable are not feasible. In other words in order to have an OO implementation, the design must be object oriented, the language need not be, but it helps.

30. Implementation Implementing in an OO language: It is very difficult to define an object oriented language. However it might be said that an object oriented language is one which has facilities for direct implementation of object oriented concepts. The most important of these are: Encapsulation in the form of a class: name, type, features (attributes and operations), instances, visibility. Inheritance: one class can be declared as a sub-class of another Association: communication through messages in order to establish a client server relationship.

31. Object-Oriented Languages Only a few, if any of object oriented languages would have all of these features. Most have nearly all. The least prevalent of all of these features is direct support for associations. As mentioned previously, associations are implemented as direct or indirect pointers or collection and/or tables. But this is fine as our design is already in a form that does not require direct language support for associations. Most viable object oriented languages have support for the class concept, allow instantiation to obtain an object, have some visibility rules, allow inheritance and are typed. Java, C++ and Eiffel are examples of such languages.

32. Object-Oriented Languages Java has all these features and the additional interesting feature that all constructs (e.g. functions and variables) must be encapsulated in a class. This makes non-OO “free wheeling” more difficult. Java currently does not have support for genericity or executable assertions. C++ has the constructs mentioned above but contrary to the design of Java, does allow programming outside the CLASS construct. It also has other language features (e.g. the friend function) that would allow breakage of encapsulation and non-OO programming. Eiffel is a pure OO language, it has all the features of Java mentioned, plus genericity, assertions and explicit exception handling but has none of the encapsulation breaking features of C++ .

33. Support for Persistent Objects • There are generally two models to employ to support persistence: • Flat files • Database Management Systems. • If opting for a DBMS, then the real choices are: • A Relational Database Management System , or • An Object-oriented Database Management System (an • Object-base) We have already seen the design criteria that allows us to select when to use each technology. The table here depicts the relative

34. Support for Persistent Objects advantages and disadvantages of selecting different combinations of DBMS technology and programming languages for application development (Goldberg and Rubin, 1995).

35. Rules of good programming Program for functionality: Stick to the design model. Do not deviate, improvise or try to improve or correct the design in your implementation. If you feel the design is not good enough, send it back with your comments and recommendations for review and correction, do not just implement what you think is right.

36. Rules of good programming Program for maintainability It can not be emphasized more strongly that a significant portion of cost of ownership of software is attributable to maintenance. Our software development process, particularly the design activities, have so far been in keeping with the principle to increase maintainability as much as possible. As such it is important to ensure our code is also written in such a way that continues to assist maintainability. Here are some guidelines that might help.Aim for: Understandability – classes, methods, naming, etc. Small methods, Coding standards/templates

37. Rules of good programming Understandability: Name each class clearly. Choose a meaningful name that captures the semantics of the objects being created through its instantiation. Use the same class name for implementation as used in the design. Keep the names short but descriptive Avoid using verbs or verb-phrases to name classes. In fact if you feel you have to do this, your design may need a review. Avoid using role names to name classes (e.g. Driver, Teacher). If you have many of these, review your design.

38. Rules of good programming Choose clear names for the elements (features) of a class. Use the same feature name for implementation as used in the design. Keep the names short but descriptive. Keep the names distinct and describe them precisely. For example distinguish between clone ( ), copy( ), create( ) and new( ), or clone( ) and deep_clone( ). Use the same convention to name operations. For example, name all routines that obtain a value as get-X( ), all those that alter a value as set-X( ) and possibly all operational procedures as do-Y( ), or simply Y( ). Where X is an attribute and Y is an action or operation.

39. Rules of good programming Separate as much as possible all your features into logical groups. For example, all the attributes in a logical group, followed by all the routines. Within this we can then have the attributes enumerated based on visibility (public ones on top). Routines can be divided into three groups, creators and constructors, all procedures and all functions. Within this, we can list the public ones on top, followed by those with more restricted visibility, progressively.

40. Rules of good programming Aim for Small methods: Rarely should there be a method with more than 10 to 12 lines of code. Methods should be small and cohesive. They must do as much as possible only one thing at each level of granularity. In fact methods should be only of four categories: Those that get a value or object (get( ) methods) Those that set a value or invoke an object (set( ) methods) Those that manipulate data to accomplish a task. Those that create and/or initialize an object (constructors)

41. Rules of good programming As a rule of thumb, of these, those belonging to the first two and the last categories should be very small, two to five lines of code only. Those belonging to the third category can be slightly larger but not very much so. If a routine is getting too large, it is either: Doing too much, is not cohesive and should be broken up into two or several smaller, cohesive routines, or Is cohesive, but has too much exposure at one level, or is doing too much at one level. Parts of it can be carved up as smaller routines that will then be called by a now smaller than original parent routine to accomplish the same task (in other words, break it up to smaller parts.)

42. Rules of good programming Use polymorphism instead of decision trees and case statements: Polymorphism not only improves the design reusability, it also assist with reducing the size of code, and its readability. It may at times however, detract from understandability of the code, particularly in cases where the reader is not familiar with polymorphism and its uses.

43. Rules of good programming • Keep the client-server model intact. • Do not break encapsulation. Avoid use of : • Global entities • Friend functions • Inheritance, particularly implementation inheritance • Do not pass the data-structure, just pass the element • Ensure methods only traverse from a client to their server and NOT BEYOND. Strictly avoid “link-hopping”.

44. Rules of good programming Adopt a programming/coding standard Choose or develop a coding standard that reflects your conventions, approach and decisions about uniformity, understandability, readability, and presentation of code. Ensure that every implementer has been adequately trained in its use and make its use mandatory. A programming standard not only would assist with understandability and maintainability of your code, but also would help obtaining of reliable and comparable code size information and metrics.

45. Rules of good programming • Program for reliability: • Validate input • Implement assertions when possible, • Ensure minimum public visibility, • Ensure exception handling is implemented and does not fall in an infinite loop • Avoid setting arbitrary limits

46. Rules of good programming • Program for performance • further optimize after the program runs Test your unit, module and application to identify bottlenecks, performance traps, memory black-holes, etc. If the defect is a design defect, send it back for design review. If it is an implementation issue, optimize by taking a different approach. Beware: Optimization helps usability and performance but is usually at the expense of maintainability and reusability.

47. Unit (class) Testing Yes, this is part of implementation! Programmers usually do a degree of testing of their creations. This is excellent practice but should not be relied upon as the sole means of verification and validation or defect management of software. The programmer should therefore: Write the program with testability in mind (e.g. scaffold the program) Test the module for integrity and validity of inputs and outputs. Ensure that all exception handling mechanisms for the module are robust and valid. Ensure that the module works well with its clients and servers.

48. Berson, A.; “Client-server Architecture”; McGraw-Hill; 1996 Bruegge, B. and Dutoit, A.; “Object-oriented Software Engineering: Conquering complex and changing systems”; Prentice-Hall; 2000 Rumbaugh, J.; et al.; “Object-oriented Modeling and Design”; Prentice-Hall; 1991. Serain, D.; Craig, I. (trans.); “Middleware”; Springer-Verlag; 1999 Shaw, M. and Garlan, D.; “Software Architecture: Perspectives on an Emerging Discipline”; Prentice-Hall; 1996. Stallings, W.; “Local and Metropolitan Area Networks”; Prentice-Hall; 1999