Managing Process Integrity (Chapter 8)

1. Managing Process Integrity(Chapter 8) B. Ramamurthy

2. Topics for discussion • Review of last lectures: • Services layers diagram (Ch. 1-7) • Webservices (design and deployment) • Relate these two in the context of design, development, deployment cycle of an SOA. • Lets understand WSDL • http://www.w3.org/TR/wsdl#_wsdl • http://www.w3schools.com/wsdl/wsdl_documents.asp • http://webservices.amazon.com/AWSECommerceService/AWSECommerceService.wsdl? • Lets look at the components of WSDL and look at Amazon.com ECS as example • Review for midterm • Process Integrity

3. Anatomy of a WSDL document

4. Sample WSDL • http://www.w3.org/2001/04/wsws-proceedings/uche/wsdl.html • http://webservices.amazon.com/AWSECommerceService/AWSECommerceService.wsdl?

5. Significance of WSDL • WSDL for a web service (WS) when specified in an IDE (such as Netbeans) exposes the operations and the signature for the operations. • This can be used access the operations of the WS. • When designing business systems using BPEL, WSDL represents the requirements of a process in a BPEL. • Business BPEL WSLD  services (SOA) • Thus WSDL is the link that aligns a business process with an IT service.

6. Data integrity • Consistency • Accuracy • Correctness of data • Entity integrity: each row in relational table must be uniquely identified • Domain integrity: the values must fall within a specified value range • Referential integrity: specifies the validity of relationship between different data sets.

7. Process integrity • Business processes span multiple IT systems. • There are times when a process state may be inconsistent. For example, a cancelled order takes some time to register with order system. • Technical failures such as network failure: system-wide solution • Business exceptions must be handled at the process level: WSDL fault provisions • Special cases: overbooking of a flight (cannot be handled by domain integrity.) Must be handled at code level. • Process ownership is another problem. For example, consider a workflow application.

8. Process integrity solutions • Logging and tracing • ACID transactions • Atomicity, consistency, isolation, durability • Transaction monitors • 2PC: two-phase commit protocol is used to ensure ACID properties for transactions that span more than a single resource manager. • Phase 1: all participating resource managers acquire locks on shared resources • Depending on phase transaction coordinator informs participants whether to commit or rollback • Phase 2: participants either commit or rollback to previous state depending on instruction from coordinator • Issues: performance, lack of support for long-lived transactions, integration of legacy systems, not suited for discontinuous network, use of multi-level and nested transactions

9. Process integrity solutions: transactional steps • Combination of transactional steps and persistent queue can improve reliability of complex processes. • Transactional steps are a set of closely related activities executed in the context of a single complex transaction. • Transactional steps facilitate the decomposition of complex, long-lived business processes into individual steps with short execution times and high transactional integrity. • Transactional steps dramatically increase the robustness and flexibility of distributed systems. • ~ provide a great way of ensuring the integrity of individual process steps.

10. Process integrity solutions transaction chain & compensation • ACID properties are too strong for complex transactions • Chained transactions with compensating transactions offer a viable means of dealing with process integrity. • Compensating transactions undo previously executed steps if a problem is encountered during the execution of a particular step in a transaction chain. • http://www.serviceoriented.org/ws-transaction.html

11. Sagas • Sagas are formal workflow models that build on the concept of chained transactions (steps). • A Saga describes a workflow wherein each step associated with a compensating transaction. • How do you compensate “out of funds” failure?

12. BPM and process integrity • BPM (Business Process Management) platforms provides features that enable business analysts and developers to design, execute and manage instances of complex business process. • Introduction of BPM will help the process integrity aspects of an enterprise application. • Explicit modeling separates technical integrity from process integrity • BPM engine provides mechanisms for monitoring processes at runtime • BPM supports compensating transactions which is key concept for managing the rollback of partially executed processes after a failure.

13. Related WS-Standards • See fig. 8-4 • A number of different industry alliances and standardization bodies are working on WS-based transaction protocols. • OASIS Business Transaction Processing (BTP) • WS Flow language • WS-BPEL • BPEL4WS • XLANG • WS-Composite Application Framework • WS-coordination • WS-Transaction • …

14. Process Integrity Spectrum 2PC/TPM in Heterogeneous env. 2PC/TPM in homogeneous env. Level of integrity Sagas with complex compensation BPM engine with Simple compensation Transactional Steps Custom-built log infrastructure DB log analyzer Eyes closed & Praying Implementation cost

15. Mid-term review • Date: 10/27 • Duration: 2 hour 30min • Topics: 1-7 Chapters • Closed book • Topics for questions: • Fundamental concepts: synchrony, loose coupling etc., interface vs. payload semantics • Service types • Design of an SOA for a application: layered architecture

16. Process Integrity in SOA Case Study: Travel Itinerary Management • Analyze the problem • Choose concurrency control: optimistic vs pessimistic • Use idempotent update operations • Avoid distributed 2PC • Build transactional steps • Design simple and flexible compensations • Combine SOA, MOA and BPM for flexibility

17. Problem Statement & Analysis (8.3.1 & Fig.8-5) • Typical sites enable you to book individual flights. • The new travel itinerary mgt. system should create complete itineraries for trips including car, hotel and flights. • Reuse existing IT subsystems esp. customer management and billing systems. • Two different front-ends: simple html front and a complex front-ends for call center (VB GUI) functionality • Three backend systems: customer management, billing, and other complex processes such as customer complaints etc.) • A number of partner systems need to be integrated; example: partner airlines, hotel and car reservation systems. • Key transactions are: confirm itinerary (typically irreversible) and create invoice; these transactions • cross organizational boundaries • cross several technical boundaries • involve different databases • involve different departments

18. IncidentManager Process state DB Architectural model Enterprise Reservation systems (cars, hotels, other airlines) Enterprise Reservation systems (cars, hotels, other airlines) Customer Customer & Itinerary DB Web front-end Billing Callcenter front-end Invoice DB

19. Concurrency Control (8.3.2, figs.8-6, 8-7) • This specifies how to deal with concurrent access to shared data in an SOA. • Optimistic and pessimistic • Pessimistic control gives exclusive access to data through acquisition of locks; locks typically are held for short duration. • Optimistic control: no locks are acquired during the transaction execution, only when data needs to be updated at the end of the transaction. • Optimistic control is the choice for long running transactions.

20. Implementing Optimistic Concurrency Control • In order to determine write conflicts, optimistic concurrency model must, at the end of each transaction, determine whether a competing transaction has changed the data after it was initially checked out. • Different ways: timestamps, version counts, state comparisons • For the first two an extra column is added in the relational table • Version number or timestamp is added to the primary key used in the WHERE clause of UPDATE operation • State comparison is implemented in Microsoft’s ADO.NET; advantage: does not require extra column in the relational table • Version numbers are associated with a service at root level • If a client is a trusted party then version number is preferable solution; else state comparison

21. Example • Customer profile • Customer updates using web || call center updates using telephone • Customer and call center read profile version #645 • Customer updates, saves, version #646 • Call center looses its updates, needs redo due version conflict • Alternatively, we could allow merge operation; possible with state comparison • If nothing works resort to pessimistic control, such as in an incident management service

22. Read customer profile update customer profile Version conflict yes no Commit changes Version inc. Optimistic concurrency control Merge possible Notify user yes yes merge changes

23. Pessimistic concurrency control Get a lock on Incident object Lock Avail? no Notify user yes yes Work on Incident Possibility for discard changes as well as supervisor notification no Commit changes Release lock

24. Make updates Idempotent • Effect of a N (N > 1) of the same requests is same as one request. • Use sequence numbers interface Itinerary { SQN getSeqeunceNumber(); void addBooking (in SQN s, in booking b); } • Can be used in error handling

25. Idempotent Operation in error handling while (retry limit not reached) try { // two idempotent operations in a block itineraryManager.confirmItinerary(); invoiceManager.createInvoice(); } catch (FatalError e) { //manage retry limit counter } if (not successful) { // handle error scenarios }

26. Avoid Distributed 2PC • First iteration: Client controlled transactions (fig. 8.9): • Has a transactions manager which implements the 2PC: begin commit & commit • Client should deal with distributed transactions • Lacks loose coupling • Second iteration: server controlled transactions (fig. 8.10): • Transactions spanning multiple databases combined on the server side • Tightly coupled, still 2PC, cost of sophisticated transaction monitor

27. Building Transactional Steps • Split transactions and introduce queues • confirm_itinerary() operation stores a message in a “pending confirmations” queue • A background thread uses this queue as input and executes operation and stores in “confirmed itineraries” queue • System picks from this queue and informs customer of success or failure • Problem: a transaction monitor is needed for 2PC among queues and DB • Solution: introduce services at various layers

28. Granularity Level BEING { pendingConfirmQ.get(); try { itineraryMgr.confirm_itinerary(); } catch (e1) { errorQ.put(e1); return; } COMMIT; try { invoiceMgr.create_invoice(r1); } catch (e2) { errorQ.put(e2); return;} COMMIT; confirmedItineraryQ.put(); } COMMIT;

29. Compensating Logic • Use simple compensating logic • Confirm itinerary  Cancel itinerary • Create invoice cancel invoice • Compensating logic may not work all the time. In such cases treat them as special cases at the application level (see 8-16)

30. SOA+MOA+BPM • Combine SOA (services) , MOA (messages and queues) and BPM (business processes, compensation and special handling) to increase flexibility • Read Ch.8 and review the figures explaining these concepts