Chapter 19 Process Integration

1. Chapter 19 Process Integration Lachlan Aldred

2. Overview • Introduction

3. Introduction • Process Integrations (PIs) need to couple (connect) differently depending on the purpose of integrating. • E.g. Feedback or Fire-and-forget • Terms such synchronous/ asynchronous, point-to-point publish subscribe refer to this coupling but have many meanings. • Industry solutions create adapters in earnest. One of the most compelling Process Integration products claims over 350 adapters. • No real support for aligning coupling with intent. • It is as if the what (the thing being integrated with) has drowned out the how (online/offline).

4. Introduction … • Languages for process integration are not taken seriously enough. • Important efforts such as BPEL and BPMN are not given the mindshare they deserve. • Neither is formally defined. • Neither can handle correlated event processing. • This lecture will not solve process integration. • Merely lay some conceptual groundwork. • Rediscover and redefine some old ideas. • Perhaps show some attributes we’ll see in future languages.

5. Process Integration as a Field of Endeavour • Workflow Reference Model (WRM) defined 5 interfaces • Interface 3 addresses invoking remote applications from a process. • Interface 4 addresses being integration with remote processes. • Large effort - requires programming to these interfaces. • BPM systems and ESBs superseded WRM. • moved/reduced the need to program to APIs. • Represented a major improvement for business users. • BPEL enabled SOA paradigm applied to Business Processes. • Interfaces 3 & 4 of WRM became invoking a Web services and exposing a process as a Web service – simple. • Sending incoming requests to the right process instance was addressed cleanly through Correlation-sets.

6. Why do we Care? • BPEL is a language for Process Integration so why improve it? • Expressive • Can’t step outside of the Web service paradigm. • Can’t handle correlated event processing. • Models cannot discriminate between tightly coupled and loosely coupled middleware (middleware is hidden in the WSDL binding). • Conceptualisation • No publish-subscribe construct. • Over the wire message format exposed to process layer.

7. Why do we Care …? • BPEL is… • Suitability • Difficult to model batch message processing. • Difficult to model message filtering. • Understandable • No graphical representation. • One sided: can’t model two integrated processes together. • Precise • Rules of BPEL written in English : ambiguity, interpretation.

8. Process Integration is not Trivial • Heterogeneity – of process engines, middleware, applications (most challenging) • Uncertainty/trust • 2 Generals paradox • Cannot trust that other general will co-attack based on messages. • There is no message protocol that can guarantee intent. • Reliability • Latency of messages (delays, stale data) • Lost messages, duplicates deliveries. • Events • Don’t wait for them to occur – just handle it if it occurs. • Composite events : Don’t do anything unless getX(Event_A) = getX(Event_B)

9. Process Integration is not Trivial … • Batch Messaging • handle/create large numbers of messages. • Avoid clumsy loops and lists • Conversations • Avoid clumsy correlation IDs. • Correlation ID variables • Technology agnostic • Nested conversations • Channel passing • Processes learn of new message source/sinks through message contents. • Technology agnostic

10. Conceptualizing PI • Conceptual modelling community have learned that conceptualising a domain has pitfalls. • Avoiding pitfalls - guiding principles: • 100 percent principle: • Every aspect of process integration should be describable. • Full expressiveness. No need to express aspects of process integration outside the language. • Conceptualization principle: • Only need to express relevant aspects. • Avoid needing to express every low-level detail (e.g. access credentials, message formats).

11. Conceptualizing PI … • Software development community have learned that abstracting a domain can fail: • e.g. Law of Leaky abstractions: • “all non-trivial abstractions, to some degree, are leaky“ (Spolsky). • Abstract APIs can save huge amounts of development time, but leaky ones cost just as much in learning time.

12. Why Patterns? • Patterns express what the key static and dynamic aspects of the domain are. • Patterns, collectively, express what needs to be modelled. • Blueprint for a process integration language

13. Pattern Categories • Patterns of Coupling • Dimension of Threading, Time, and Space • Batch Patterns • Multiple message sends/receives, filtering. • Bi-directional Interactions • Various patterns for receiving feedback – within the interaction. • Composed Interactions • Patterns achieving state-aware conversations, across many interactions. • Event-based Process Patterns • Patterns for handling unsolicited messages in-process

14. Pattern Categories … • Transformations • Patterns that transform messages and message data. • Process Discovery • Patterns that dynamically change during runtime to perform new actions.

15. Coupling Pattern: Non-blocking receive • Occurs when the receiving process is not waiting for the message to arrive. • Observable as foundation of most asynchronous architectures. • Enabler for Event-handling. • e.g. reallocate seats for airline cancellations.

16. Coupling Pattern: Time Decoupled • No indication back to sender about success of interaction. • ‘Fire and forget’ – send message and start doing something else while the interaction is running. • e.g. Reconciling inter-bank payments overnight.

17. Composed Interaction: Property-based Conversation • Modelled by expressing functions over messaging tasks • You only need to model the function logic and identify the tasks involved. • The process environment does the matching. • When a match is found the message is directed to the right process instance.

18. Conclusions • Patterns have been used in software engineering to express conceptual solutions to re-occurring problems. • Process Integration Patterns, on the other hand, are requirements patterns. • They express what process integration needs to b able to do. • They indicate what is relevant and what is less relevant (Conceptualisation Principle). • They indicate what a fully expressive PI language should be capable of (100 Percent principle).