1 / 57

Wojciech Sliwinski Wojciech.Sliwinski@cern.ch Beams Department, Controls Group CERN

The role of messaging solutions in complex systems, based on experience in the accelerator controls at cern JINR/CERN/MEPHI Computing school DUBNA , 26 th October 2011 . Wojciech Sliwinski Wojciech.Sliwinski@cern.ch Beams Department, Controls Group CERN. Outline.

dudley
Download Presentation

Wojciech Sliwinski Wojciech.Sliwinski@cern.ch Beams Department, Controls Group CERN

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The role of messagingsolutionsincomplex systems, based on experienceintheacceleratorcontrolsatcernJINR/CERN/MEPHI ComputingschoolDUBNA, 26thOctober 2011 Wojciech Sliwinski Wojciech.Sliwinski@cern.ch Beams Department, Controls Group • CERN

  2. Outline • Defining Middleware • Types of Middleware • CERN Acc. Controls MessagingInfrastructure • Middleware Review for CERN Acc. Controls • Conclusions Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  3. Outline • Defining Middleware • Types of Middleware • CERN Acc. Controls MessagingInfrastructure • Middleware Review for CERN Acc. Controls • Conclusions Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  4. Why Middleware? • How do we build/connect applications over a network? • How do we facilitateDistributed Computing (2-tier, …, n-tier)? • How to support theheterogeneous environments? Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  5. What is Middleware? • Software, which allows an application to interoperate with other software, without requiring the user to understand and to code the low-level operations required to achieve interoperability • Software layer between OS and the applications • Hidescomplexity & heterogenity of distributed system • Handlesissuesrelated toOS, Network Protocols &hardware platforms Application Application Application Standard API Standard API Standard API MW MW MW Specific API Specific API Specific API Operating System1 Operating System2 Operating SystemN Communication system Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  6. Introduction to Middleware • Middleware provides support for: • Naming, Location, Service discovery, Replication • Protocol handling, Communication faults, QoS • Synchronisation, Concurrency, Failover, Scalability • Access control, Authentication • Middleware dimensions: • Request-Reply vs. Asynchronous Messaging • Language-specific vs. Language-independent • Proprietary vs. Standards-based • Small-scale vs. Large-scale • Tightly-coupled vs. Loosely-coupled components Middleware is everywhere. You might think you can leave it but itjust finds you again:-) Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  7. Outline • Defining Middleware • Types of Middleware • CERN Acc. Controls MessagingInfrastructure • Middleware Review for CERN Acc. Controls • Conclusions Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  8. Middleware – largedomain … which to choose? Network Embedded LAN WAN/Internet req/rep pub/sub direct p2p broker rpc/orb message Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  9. Types of Middleware • RemoteProcedureCall (RPC) • Object-Oriented Middleware (OOM) • Message-Oriented Middleware (MOM) • Event-Based Middleware a.k.aPublish-Subscribe Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  10. RemoteProcedureCall - RPC • Masks remote function calls as being local • Client-Server model withRequest-Reply paradigm • Synchronouscommunication • Marshalling of function parameters and return value • Provides Location & Platform transparency • Technologies: • DCE-RPC • XML-RPC Client App. ServerApp. IDL Stub IDL Skeleton Req/Reply Network Network Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  11. RPC - Disadvantages • Synchronous Request-Reply interaction • Tight coupling between client and server • Client may block for a long time if server loaded • Multi-threaded programming at client • Slow/failed clients may delay servers when replying • Multi-threadedprogramming at servers • RPC paradigm is not object-oriented • Invoke functions on servers as opposed to methods on objects Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  12. Object-OrientedMiddleware - OOM • Client-Server model withRequest-Reply paradigm • Synchronouscommunication • Marshalling of function parameters and return value • Provides Location & Platform transparency • Objectscan be localorremote • Remote objects accesible via proxy objects • Technologies: • OMG CORBA • MS DCOM • Java RMI • SOAP ServerApp. Client App. IDL Skeleton IDL Stub POA Adapter Client ORB Server ORB Req/Reply IIOP IIOP Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  13. Object-Oriented MW - Disadvantages • Synchronous Request-Reply interaction • Tight coupling between client and server • Client may block for a long time if server loaded • Multi-threaded programming at client • Slow/failed clients may delay servers when replying • Multi-threadedprogramming at servers • Distributed garbage collection • Releasing memory for unused remote objects • Error-prone and difficult to debug • Staticmappings and heavy-weight development • Difficult to extend an operational system Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  14. Message-OrientedMiddleware - MOM • Data exchange via Messages & MessageQueues • Asynchronous communication (Point-to-point One-to-One) • Client and Server arelooselycoupled • MessageServersmay do filtering, transforming, logging… • Technologies: IBMWebSphereMQ, TIBCO Rendez-vous, JMS, SOAP Client App. Server App. Message Servers message queues local message queues local message queues Network Network Network Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  15. Message-Oriented MW - Disadvantages • Poor programming abstraction (but has evolved) • Rather low-level • Request-Reply more difficult to achieve, but can be done • Message formats originally unknown to middleware • No type checking (solvedinJMS) • Queue abstraction only gives One-to-One communication • Limits scalability (solvedinJMSwithPublish-Subscribe) Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  16. What we lack, so far • General interaction patterns • We have Request-Reply and One-to-One • Publish-Subscribe? One-to-Many? Many-to-Many? • Dynamic joining and leaving? • Location transparency • Anonymity of communicating entities Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  17. Event-Based Middleware a.k.aPublish-Subscribe • Publishers (advertise and) publish Events (Messages) • Subscribers express interest in Events with Subscriptions • Event Service notifies interested Subscribers of published Events • Events can have arbitrary content (typed) • Technologies (Brokers): JMS (ActiveMQ, RabbitMQ), AMQP (…) subscribe Subscriber1 Publisher1 publish notify Event Service (Broker) subscribe publish Publisher2 Subscriber2 notify … … subscribe topics publish notify SubscriberN PublisherN Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  18. Properties of Publish-Subscribe • Asynchronous communication • Publishers and Subscribers are loosely coupled • Many-to-Many interaction between Pubs and Subs • Scalable scheme for large-scale systems • Publishers do not need to know Subscribers, and vice-versa • Dynamic join and leave of Pubs &Subs (thanks to Brokers) • Topic based Publish-Subscribe very expressive • Filtered information delivered only to interested parties Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  19. Outline • Defining Middleware • Types of Middleware • CERN Acc. Controls MessagingInfrastructure • CERN Acc. Controls Middleware Review • Conclusions Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  20. CERN Controls Middleware – inall SW tiers… CMW client (C++/Java) GUIs, LabView, RADE JMS client (Java) GUIs CMW server (C++/Java) Proxy, DIP, AlarmMon, AQ JMS client (Java) Servers: Logging, InCA, SIS CMW client (Java servers) Logging, LSA, InCA, SIS CMW server (C++) FESA, FGC, GM CMW server (C++) PVSS (Cryo, Vacuum) Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  21. JMS for Accelerator Controls • Java Messaging Service • Purpose • Reliable transport of data between Java Processes • Controls • Alarm System • Data Rendering Services • (Control) GUIs • Logging Services • Beam Security System(s) No JMS - No Beam ! Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  22. JMS for Accelerator Controls ~46K Subscriptions 280 Connections ~50 Processes ~100 Processes JAVA Process GUI Process JMS JAPC JAPC JMS JMS CMW CMW C/C++ Process CMW Front End CMW 4000 servers @1700 machines Devices / Hardware Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  23. Examplesof JMS ”aware” systems Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  24. History of JMS inAccelerator Controls • Usage of SonicMQ JMS for LHC Alarm System since 2001 • ActiveMQ introduced in 2005 • Other project started to use JMS service • Today vital part of Beam Operations Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  25. Technical Aspects • Broker Configuration • Non-persistent messages on topics • No subscription recovery policy • No durable subscribers • No exclusive consumers • No replyTo mechanism • Broker is persistent • Clients • Pure JAVA - Wireformat Protocol Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  26. General JMS Deployment Situation • SonicMQ : 12 Brokers • TIM, LASER, DIAMON • Always as two node cluster • ActiveMQ : 16 Brokers • One main default Broker Service • Otherwise project dedicated Brokers • All Brokers on Technical Network Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  27. Overall Broker Service Jan 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  28. JMS Service Data Handling • Central JMS Service (276 days in 2010) • 58 Million msg / day • 120 TB IN • 650 TB OUT • Additional handling of • Heavy Connection andSubscription handling • Bursts loads • Connected subscribers grewfrom 2K to 12K in 6 months => 440 GB /day YES, ActiveMQcan! => 2.5 TB /day Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  29. Data Handling during LHC startup 2010 Total Data Out 2.57 TB/day Increase in data demand 1TB/day LHC startup Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  30. JMS Service Availability 2010* • Total hours : 6624 • ActiveMQ: 99.99% • = 8 minutes downtime (due to kernel update) • SoniqMQ : 99.87 % • = 8 hours downtime * November 2010 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  31. Monitoring and Troubleshooting Tools • DIAMON & Lemon • JConsole / JVisualVM • Dump of broker information to SQLite DB • Traffic Monitoring Tool • Various Message Listeners PRIORITY Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  32. Easy Identification of Bad Producers Problematic Producer Fixed Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  33. JMS Service – LessonsLearnt • ActiveMQ was agood choice • Running reliably • Performs well • Provides many metrics for diagnostic and monitoring • Good integration in BE-CO monitoring tools via JMX • Projects are notalwaysaware of sending so much data • Service Level Agreements with projects required • More analyzing tools for end users required Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  34. JMS Service – LessonsLearnt • Monitoring is a vital part of operations • Message reception speed test • Memory Percent usage • JMX attributes (#topics, #consumers, #subscriptions) • Broker Configuration is not always obvious Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  35. JMS Service – LessonsLearnt • Clustering • Good for failover and load balancing • Additional complexity • Updating clients for new cluster members • Automatic rebalancing of clients • Memory is important • Tuning of Java GCoptions Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  36. Outline • Defining Middleware • Types of Middleware • CERN Acc. Controls MessagingInfrastructure • CERN Acc. Controls Middleware Review • Conclusions Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  37. CERN Controls Middleware – CMW GUI operator consoles Controls Middleware Front ends“virtual devices” Embedded Middleware Hardware Devices Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  38. CMW Mandate & Scope • Standard set of MW solutions & libraries • Centrally managed services • Track & optimize runtime parameters • Scope: CERN Accelerator Complex • Operational 24*7*365 • Must be Reliable & High Quality • ~80’000 HW devices, ~4’000 servers • ~1700 machines, ~2’000’000 IO points • Hugeinfrastructure! Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  39. CMW Data Flow Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  40. CERN Controls Middleware Linux/Windows C++/Java process Controls Middleware API CORBA • Problems: • - C++ and Java implementations differ  • Heavy in memory usage  • Complex error prone API  • No direct support for pub/sub  • Blocking issues (JacORB)  • Shrinking community  • Lack of new releases and bug fixes  TCP/IP Controls Middleware CORBA Controls Middleware API C++ process Long LHC shutdown at the end of 2012 A unique possibility! Instead of patching let’s provide a new one! RT Linux/LynxOS Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  41. CERN Middleware Requirements • Lightweight Desirable • Active community • Friendly API, documentation • Request/reply & pub/sub patterns • QoS • Asynchronous • Performance & Scalability Mandatory • Stability, Maturity & Longevity • Open source, redistributable license • C++/Java • Linux/Windows • Over TCP/IP LAN Fundamental Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  42. How did we evaluate –> our criteria • Appearance • Simple usage • Testing • Communication patterns • Performance • QoS • Exceptional situations • Creators • specification • documentation • Users • forums • bug reports • Internet • Download • licensing • Compile • Linux • LynxOS • Run examples CRITERIA resources, binary size, memory QoS Community,maturity API, look & feel, documentation Communications patterns performance Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  43. Surprising how many great new products! 2011 1991 1983 ARPANET IP v6 TCP UDP IP v4 TODAY 1969 1974 1980 1981 1998 2011 Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  44. Evaluated middleware products All opinions are based only on our knowledge and evaluation. Each of the products, depending on the requirements, may constitute a good solution. OpenAMQ CoreDX RTI DDS QPid ZeroMQ OpenSpliceDDS RabbitMQ YAMI Ice omniORB MQtt RSMB JacORB Thrift Mosquito Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  45. CORBA (omniORB, JacORB) Object-oriented communication platform standardized by OMG. It is over 20 years old, and the standard is well established but... • Complex, hard to learn and use  • Differences between implementations  • Big memory footprint  • Shrinking community  • Seldom updates and bug fixes  Dying out Shrinking community Complex Heavy Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  46. Ice Object-oriented middleware by ZeroC, people who left CORBA to fix its problems. Conceptually similar to CORBA but... • Better design and implementation  • Modern easier to use API and IDL mapping  • Concise C++ and Java implementation  • Support for versioning  • Active community  Active community Well designed Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  47. Thrift Middleware by Facebook (moved to Apache) • Lightweight, free of unnecessary dependencies  • Modern, clean API  • Active community  • Ongoing development, still incomplete  • No documentation, empty tutorial  Work stillin progress Poordocumentation Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  48. YAMI4 Message-oriented middleware developed by one of our former colleagues. Used by some CERN Controls services. • Lightweight, free of unnecessary dependencies  • Well designed, modern API  • Asynchronous req/rep and pub/sub patterns  • Dynamic type specification (paid in performance)  • Small community  Welldesigned Smallcommunity Lightweight Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  49. AMQP – OpenAMQ, Qpid, RabbitMQ Middleware where messages are distributed by a broker. A few independent implementations of the broker and clients. • Big community  • Designed to support pub/sub  • Protocol standard issues  • Broker – single point of failure  • Broker – additional hop, slower communication  • Broker – additional, non compliant monitoring tools  Brokerissues Active community Slower for req/rep Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

  50. DDS – RTI, CoreDX, OpenSplice Data-oriented middleware with p2p communication. Standard defined by OMG, with a few compatible implementations. • Big industrial and military community  • Possible compatibility with CORBA  • Designed to support pub/sub  • Req/rep possible but needs 2 channels  • Steep learning curve  • Complex API  Big industry presence Heavy Complex Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

More Related