Beyond the simple Pipeline - managing processes over time

1. Beyond the simple Pipeline - managing processes over time XML Prague , 17/06/2007Geert Bormans, independent consultantgeert@gbormans.telenet.be

2. Introduction • Who am I? • Background • 10 years team-lead and researcher • researchgroup document architectures Catholic University Leuven • 5 years VP professional services • British/Canadian XML software vendor • Independent Consultant • Mainly projects that surface the intrinsic value of (yet existing) content • Migrations, ‘legacy’ conversions, XML transformations • Content Management, content reuse • e-publishing

3. Introduction • What to expect? • A quick introduction of the problem space • Introduction of the project • Introduction of the solution • A word about pipelines • Challenges for building pipeline solutions • The solutions Architecture • Wider applicability and future plans • A quick demo

4. So we all love XML pipelines by now… Ever thought about managing them?

5. Managing Pipelines - Problem space

6. The project • The customer • A financial institution working for the goverment • Publishing yearly statements • New legislation requires archiving the statements • Current Architecture • Complex Monolithic Code • Growing with yearly changes

7. The project – Current Architecture

8. Example Statement

9. Next years Example Statement Publisher company rebrand ID added to statement

10. Original Request for Proposal

11. Managing XML instead of PDF ...

12. ... and version managing the pipeline Each asset in the CMS is linked (through metadata) to a specific version of the pipeline, hence assuring the possibility to reconstruct a statement as it was printed before

13. Pipeline benefits • Break up complexity • Simpler to develop • Easier to debug and maintain • Clear assignment of responsibilities • Mix technologies easily • Reuse components

14. Pipeline solutions - challenges • Execution • Fit components together and Compile Pipeline • Execute with high performance in a robust way • Avoiding intermediate (de)-serialisation • Allow for mixed technologies • Different technology per component possible • Easy creation of pipeline • Manageability

15. Solutions Architecture

16. Pipeline Component Managment System • Processing steps are URI-addressable resources • Steps pulled together at run time to create pipeline • Execution by Pipeline Execution Engine • Datamodel for simple pipeline

17. Pipeline Execution Engine • Thin layer on top of NetKernel • Resource oriented computing platform • Microkernel that schedules resource requests • Dynamic compilation of various scripting languages • Strong support for XML processing tasks • Pipeline language: DPML

18. DPML Example

19. Meeting the challenges • Pipeline execution by NetKernel • Unnecessary serialisation/deserialisation avoided by notion of aspects and transreptors • NetKernel supports pipeline components developed in various technologies • PCMS serves as abstraction layer between pipeline development and execution • PCMS takes care of pipeline version managment

20. Wider Applicability • Architecture provides version managment of pipeline components • Most important factor of complexity of XML processing is variation in time • Applications layered on this architecture will be better protected against damage by changes • A solution worthwhile considering for any complex XML processing application

21. Future Plans • Non lineair pipelines (forking, looping) • Dashboard for monitoring • Timings, breakpoints • Import filters for • XProc • Stylus Studio pipes • ...

22. Questions and Discussion Thanks for your attention