Generalizing Transactions: Building Dependable Systems from Arbitrary Components

1. Generalizing Transactions ...[... Into a Compositional Means for Building Dependable Systems From Arbitrary Components]Andreas ReuterHPTS, September 26, 2005

2. Setting the Stage - I • At first I hoped that such a technically unsound project would collapse, but I soon realized it was doomed to success. Almost anything in software can be implemented, sold, and even used given enough determination. There is nothing a mere scientist can say that will stand against the flood of a hundred million dollars. But there is one quality that cannot be purchased in this way---and that is reliability. The price of reliability is the pursuit of the utmost simplicity. It is a price which the very rich find most hard to pay. (C.A.R. Hoare, "The Emperor's Old Clothes", Turing Award Lecture) • I have yet to see any problem, however complicated, which, when you looked at it in the right way, did not become still more complicated. (Poul Anderson) • Reality is that which, when you stop believing in it, doesn't go away. (Philip K. Dick) • The most likely way for the world to be destroyed, most experts agree, is by accident. That's where we come in; we're computer professionals. We cause accidents. (Nathaniel S. Borenstein)

3. An Observation From the Sideline • For a number of years, transaction stuff seemed to have fallen out of grace with the research community. • It obviously was considered a solved problem, SMOP, and thus left to the developers. • And, sure enough, much progress was made and many interesting implementations came about. • But more recently, transactions and/or concepts closely related to them attracted some new attention; in some cases from the „old“ communities, but also with groups that never before were considering those issues. • This talk is a snapshot of that situation. It was intended to provide some kind of a unifying perspective, but for reasons that (hopefully) will become clear as I go along, it is rather a description of a conceptual mess. • Depending on how one judges the reasons for the renewed interest in transaction-based solutions, one can (ab)use this analysis as an argument for the need of more orchestrated work – or as an illustration of how ill-advised all these attempts are.

4. From the „Atomic Manifesto: a Story in Four Quarks“ „The interpretations and roles of the atomicity concept(s) vary substantially across these communities [DB/TA; FT; formal methods; HW; PL]. For example, the emphasis in database systems is on algorithms and implementation techniques for atomic transactions, whereas in dependable systems and formal methods atomicity is viewed as an intentionally imposed (or sometimes postulated) property of system components to simplify designs and increase dependability. On the other hand, all communities agree on the importance of gaining a deeper understanding of composite and relaxed notions of atomicity. Moreover, the hope is that it will eventually be possible to unify the different scientific viewpoints into more coherent foundations, system development principles, design methodologies, and usage guidelines.“ Source: http://drops.dagstuhl.de/volltexte/2004/9

5. Let‘s Get the Terminology Straight - I From a paper presented at the Dagstuhl workshop: „Transactions require extensive run-time support, and slow down execution significantly. If persistence and tolerance to crash failures is not needed, then a simple monitor-based design can provide the same behavior, with considerably better performance.“ Its summary says: „Atomicity is considered during initial system operation specification in the Operation Model to abstract away the complexity of concurrency. As the development process goes on, the Operation Model is refined – the system operations are broken up into smaller pieces – to slowly introduce concurrency back into the system. Finally, at the design stage, low-level concepts that provide atomicity, such as transactions or monitors, are used in the Interaction Model to ensure consistent concurrent updating of the application state.“

6. Let‘s Get the Terminology Straight - II This is quoted from Luca Cardelli‘s ICSE 2005 Keynote: • Wish: Wouldn‘t it be nice to hide concurrency from programmers? • SQL does it well • UI packes do it fine (mostly single-threaded!) • RPC does it OK • But we are moving towards more asynchrony, i.e. towards more visible concurrency (e-commerce scripts and languages, web-services, etc.). You can hide all the concurrency some of the time, you can hide some concurrency all the time, but you cannot hide all the concurrency all the time. • Asynchronous message-based concurreny does not fit easily with more traditional shared-memory synchronous concurrency control. • Goal: Make concurrent flows available and checkable at the language level.

7. i.e. spanning in-memory and external data, and expressive enough to replace rather than augment mutexes and condition variables in common uses Let‘s Get the Terminology Straight - III This is a quote from a talk by Tim Harris: • Suppose we have efficient support for atomic multi-word updates • Including contention-management & fairness issues etc • Whether using a pure hardware implementation, a hybrid hardware/software one, or a pure software approach • What problems remain in using this to implement pervasive atomic blocks? • What expressiveness problems remain with using pervasive atomic blocks as a programming abstraction?

8. So There Are Many Words: • Transaction • Atomicity • Consistency • Isolation • Durability • Concurrency • Synchronization • Monitor • Serializability • Observability • Linearizability • Repeatability • Nesting • Compensation • Undo/Redo • ............

9. And I Have Not Mentioned ... • transactional messaging; • transactional queues; • workflow; • web service composition; • and, unfortunately, your favorite topic.

10. Units of Execution • Low-level routine 10-6 sec • I/O routine 10-3 sec • Simple interactive TA 1 sec • Simple query 101 sec • Complex query 102 sec • Routine workflow 103 sec • Simulation / data mining app. 104 sec • Medium-sized workflow 105 sec • ... • WF repr. large construction project 108 sec

11. Let‘s Get the Terminology Straight – One More Time The paper „Investigating Atomicity and Observability“ by Jon Burton and Cliff Jones starts by saying: „Using the fiction of atomicity as a design abstraction and then refining atomicity as we develop an implementation is widely used in areas of concurrent computing such as database systems and transaction processing.“ It then goes on to make the following clarification: „In this paper, by the term ‚atomic‘ or by the property of ‚atomicity‘, we mean the isolation property described by the I of ACID in the database literature [...]. We do not mean the all-or-nothing property which is implied by the A of ACID.“

12. An Algebraic Analogy Up to the 16th century mathematicians treated versions of the cubic equation such as ax3 + bx = c ax3 = bx + c ax3 = bx2 + c as separate problems, for which individual solutions had to be found – actually some of the solutions only worked for special values of a, b, or c. Recognizing that these are all special cases of the general cubic equation ax3 + bx2 + cx + d = 0 and finding a solution for it required a major breakthrough and is generally regarded as the beginning of modern algebra.

13. Attempts That Have Not Really Worked • About 15 years ago, there was a considerable amount of work in what was called „extended transaction models“. • „Extended“ generally referred to the fact that at least one of the ACID properties was given up in order to achieve more flexibility in some area. • However, it was never really questioned whether the implementational „schema“ of a transaction and the guarantees provided to the programming environment necessarily had to be the same. • As an example consider a very simple transaction that does an update, provided some entry condition holds. In case it does, the application wants the update with all the glory of ACID. In case the condition does not hold, the user has to be informed about the abort – but the message has to be delivered exactly once. So from the application‘s perspective we have an abort, but the reliable delivery of the abort news requires the commit of some transaction. • This is where the notion of a guarantee comes into the picture: Applications want (differents types of) guarantees regarding the executions of their services. Those guarantees have various flavors of atomicity. Transactions, on the other hand, are the means for implementing guarantees, but there is certainly no isomorphism between the two sets on concepts.

14. Three Interesting Aspects • Define a framework for specifying guarantees about executions – which can then be supported by transactional mechanisms.

15. DB space Invoc. space Msg space Lock space Notification of success C holds D -> f(D) Notification of abort (not C) C does not hold D -> D Program calls abort for internal reasons Notification of abort (retcode) D -> D Notification of abort (retcode) System signals failure D -> D Transactional Guarantees - I Assume a simple transaction that performs function f on some set of data D if condition c holds; otherwise it aborts.

16. DB space Invoc. space Msg space Lock space Notification of success C holds D -> f(D) Notification of abort (not C) C does not hold D -> D Program calls abort for internal reasons Notification of abort (retcode) D -> D Notification of abort (retcode) System signals failure D -> D Transactional Guarantees - II Assume a simple transaction that performs function f on some set of data D and asynchonously invokes service s if condition c holds; otherwise it aborts. Invoke s Insert L(D) Install & activ. rec. for s Install & activ. rec. for s Install & activ. rec. for s If in addition the application needs to protect its updates for compensation after commit, it will store the identifiers in lock space.

17. Three Interesting Aspects • Define a framework for specifying guarantees about executions – which can then be supported by transactional mechanisms. • Give objects control over who is allowed to see them when – as opposed to transactions deciding unilaterally what they want to lock. That will help with fault containment, compensation, etc.

18. An Example: Order Entry Validate Customer Credit Check Create Order Create Pick-List Add Order Line Create Bill Create P.-Order Allocate Inventory Check av. to Promise Create M.-Order

19. Service Composition Create P.-Order Validate Customer Credit Check Create Order Create M.-Order Create Pick-List Add Order Line Create Bill • In principle, the necessary structure • can be defined by • The data and control flow dependencies • The execution guarantees required • The visibility rules for the data altered. Check av. to Promise Allocate Inventory

20. Three Interesting Aspects • Define a framework for specifying guarantees about executions – which can then be supported by transactional mechanisms. • Give objects control over who is allowed to see them when – as opposed to transactions deciding unilaterally what they want to lock. That will help with fault containment, compensation, etc. • Use the log for fault containment, debugging, and related purposes.

21. d4 D d2 d6 A E d3 d1 B C d5 t Spheres of Control I

22. d4 D d2 d6 A E d3 d1 B C d5 t Spheres of Control II

23. d4 D d2 d6 A E d3 d1 B C d5 t Spheres of Control III

24. Protective Wrappers Andersen, Randell und Romanovsky are suggesting „protective wrappers“ as a means for building highly reliable systems from standard (OTS-) components. Such wrappers would ideally monitor all input and output to/from the standard components and match them with the explicitly specified correctness-, reliability- and security constraints of the overall system.

25. One Final Invariant And they looked upon the software, and saw that it was good. But they just had to add this one other feature … (G.F. McCormick)