Scalable, Fault-tolerant Management of Grid Services: Application to Messaging Middleware

1. Scalable, Fault-tolerant Management of Grid Services: Application to Messaging Middleware Harshawardhan Gadgil hgadgil@cs.indiana.edu Ph.D. Defense Exam (Practice Talk)Advisor: Prof. Geoffrey Fox

2. Talk Outline • Motivation • Architecture • Service-oriented Management • Performance Evaluation • Application: Managing Grid Messaging Middleware • Related Work • Conclusion

3. Motivation:Characteristics of today’s (GRID) Applications • Increasing Application Complexity • Applications distributed and composed of multiple resources • In future, much larger systems will be built • Components widely dispersed and disparate in nature and access • Span different administrative domains • Under differing network / security policies • Limited access to resources due to presence of firewalls, NATs etc… • Components in flux • Components (Nodes, network, processes) may fail • Services must meet • General QoS and Life-cycle features • (User defined) Application specific criteria • Need to “manage” services to provide these capabilities

4. Motivation:Key Challenges in Management of Resources • Scalability • With Growing Complexity of application, number of resources that require management increases • E.g. LHC Grid consists of a large number of CPUs, disks and mass storage servers • Web Service based applications such as Amazon’s EC2 dynamically resizes compute capacity, so number of resources is NOT ONLY large BUT ALSO in a constant state of flux. • Management framework MUST cope with large number of resources in terms of • Additional components Required

5. Motivation:Key Challenges in Management of Resources • Scalability • Performance • Performance Important in terms of • Initialization Cost • Recovery from failure • Responding to run-time events • Performance should not suffer with increasing resources and additional system components

6. Motivation:Key Challenges in Management of Resources • Scalability • Performance • Fault – tolerance • Failures are Normal • Resources may fail, but so also components of the management framework. • Framework MUST recover from failure • Recovery period must not increase drastically with increasing number of resources.

7. Motivation:Key Challenges in Management of Resources • Scalability • Performance • Fault – tolerance • Interoperability • Resources exist on different platforms • Written in different languages • Typically managed using system specific protocols and hence not INTEROPERABLE • Investigate the use of a service-oriented architecture for management

8. Motivation:Key Challenges in Management of Resources • Scalability • Performance • Fault – tolerance • Interoperability • Generality • Management framework must be a generic framework. • Should manage any type of resource (hardware / software)

9. Motivation:Key Challenges in Management of Resources • Scalability • Performance • Fault – tolerance • Interoperability • Generality • Usability • Simple to deploy. Built in terms of simple components (services) • Autonomous operation (as much as possible)

10. Summary:Research Issues • Building a Fault-tolerant Management Architecture • Making the architecture SCALABLE • Investigate the overhead in terms of • Additional Components Required • Typical response time • Recovery Time • Interoperable and Extensible Management framework • General and usable system

11. Architecture:Assumptions • For our discussion: RESOURCE = SERVICE • External state required by resources is small • Can be captured using a message-based interaction interface • Resource may maintain internal state and can bootstrap itself • E.g.: Shopping Cart (Internal State = Contents of cart, External state = Location of DB and access credentials where contents persist) • Assume a scalable, fault-tolerant database store that acts as a registry to maintain resource specific external state

12. Definition:The process of Management • E.g. Consider Printer as a resource that will be managed • Generates Events (E.g. Low Ink Level, Out of paper) • Resource Manager appropriately handles these events as defined by Resource specific policies • Job Queue Management is NOT the responsibility of our management architecture. • We imagine existence of a separate Job Management Process which itself can be managed by our framework (E.g. Make sure it is always up and running)

13. Management Architecture built in terms of • Hierarchical Bootstrap System – Robust itself by Replication • Resources in different domains can be managed with separate policies for each domain • Periodically spawns a System Health Check that ensures components are up and running • Registry for metadata (distributed database) – Robust by standard database techniques and our system itself for Service Interfaces • Stores resource specific information (User-defined configuration / policies, external state required to properly manage a resource) • Messaging Nodes form a scalable messaging substrate • Message delivery between managers and managees • Provides Secure delivery of messages • Managers – Active stateless agents that manage resources. • Resource specific management thread performs actual management • Multi-threaded to improve scalability • Managees – what you are managing (Resource / Service to manage) – Our system makes robust • There is NO assumption that Managed system uses Messaging nodes • Wrapped by a Service Adapter which provides a Web Service interface

14. … Architecture:Scalability: Hierarchical distribution ROOT Spawns if not present and ensure up and running • Passive Bootstrap Nodes • Only ensure that all child bootstrap nodes are always up and running EUROPE US … • Active Bootstrap Nodes • /ROOT/EUROPE/CARDIFF • Responsible for maintaining a working set of management components in the domain • Always the leaf nodes in the hierarchy CGL FSU CARDIFF

15. Always ensure up and running Always ensure up and running Architecture:Conceptual Idea (Internals) Periodically Spawn Manager processes periodically checks available resources to manage. Also Read/Write resource specific external state from/to registry Connect to Messaging Node for sending and receiving messages User writes system configuration to registry

16. Architecture:User Component • Characteristics are determined by the user. • Events generated by the Managees are handled by the manager • Event processing is determined by via WS-Policy constructs • E.g. Wait for user’s decision on handling specific conditions • The event handler has been specified, so execute default policy, etc… • Note Managers will set up services if registry indicates that is appropriate; so writing information to registry can be used to start up a set of services • Generic and Application specific policies are written to the registry where it will be picked up by a manager process.

17. Issues:Issues in the distributed system • Consistency – Management architecture must provide consistency of management process • Examples of inconsistent behaviour • Two or more managers managing the same resource • Old messages / requests reaching after new requests • Multiple copies of resources existing at the same time leading to inconsistent system state • Use a Registry generated monotonically increasing Unique Instance ID to distinguish between new and old instances • Security – Provide secure communication between communicating parties (e.g. Manager <-> Resource) • Leverage NaradaBrokering’s Topic Creation and discovery and Security framework to provide: • Provenance, Secure Discovery, Authorization & Authentication • Prevent unauthorized users from accessing the resource • Prevent malicious users from modifying message (Thus message interactions are secure when passing through insecure intermediaries)

18. Consistency • All new entities get a unique InstanceID (IID), generated thru registry. All interaction from that entity use this id as a prefix. We assume this to be a monotonically increasing number (E.g. an NTP timestamp) • Thus If a Manager Thread starts, it is assigned a unique ID by the registry. A newer instance has a higher id, thus OLD manager threads can be distinguished • The resource ALWAYS assumes the manager thread to be current with the highest known ID • Thus requests from manager thread A are considered obsolete IF IID(A) < IID(B) • This IID may also be used as a prefix for interactions • Message ID = [X:N] where X is the registry assigned IID and N is a monotonically increasing number generated by an instance with IID as X • Service Adapter stores the last known MessageID allowing it to differentiate between duplicates AND obsolete messages • Similar principle for auto-instantiated resources. IF a resource is considered DEAD (Unreachable) and a new resource is spawned this new resource has the same ResourceID (allows us to identify the type of resource) but a higher InstanceID. • Later if the old resource joins back in, it can be distinguished by checking its InstanceID and appropriately taking action • E.g. IF IID(ResourceInstance_1) <IID(ResourceInstance_2), then ResourceInstance_1 was previously deemed OBSOLETE hence ResourceInstance_2 exists SO instruct ResourceInstance_1 to silently shutdown

19. Interoperability:Service-Oriented Management • Existing systems • Platforms, languages • SNMP, JMX, WMI • Quite successful, but not interoperable • Move to Web Service based service-oriented architecture that uses • XML based interactions that facilitate implementation in different languages, running on different platforms and over multiple transports.

20. Interoperability:WS – Distributed Management vs. WS-Management • Both systems provide Web service model for building application management solutions • WSDM – MOWS (Mgmt. Of Web Services) & MUWS (Mgmt. Using Web Services) • MUWS: unifying layer on top of existing management specifications such as SNMP, OMI (Object Management Interface) • MOWS: Provide support for management framework such as deployment, auditing, metering, SLA management, life cycle management etc… • WS Management identifies core set of specification and usage requirements • E.g. CREATE, DELETE, GET / PUT, ENUMERATE + any number of resource specific management methods (if applicable) • Selected WS-Management primarily due to its simplicity and also to leverage WS-Eventing implementation recently added for Web Service support in NaradaBrokering

21. Implemented: • WS – Specifications • WS – Management (June 2005) parts (WS – Transfer [Sep 2004], WS – Enumeration [Sep 2004] and WS – Eventing) (could use WS-DM) • WS – Eventing (Leveraged from the WS – Eventing capability implemented in OMII) • WS – Addressing [Aug 2004] and SOAP v 1.2 used (needed for WS-Management) • Used XmlBeans 2.0.0 for manipulating XML in custom container. • Security Framework for NB • Provides secure end-to-end delivery of messages • Broker Discovery mechanism • May be leveraged to discover Messaging Nodes • Currently implemented using JDK 1.4.2 (expect better performance moving to JDK 1.5 or better)

22. Performance EvaluationMeasurement Model – Test Setup • Multithreaded manager process - Spawns a Resource specific management thread (A single manager can manage multiple different types of resources) • Limit on maximum resources that can be managed • Limited by Response time obtained • Limited by maximum threads per JVM possible (memory constraints)

23. Performance EvaluationResults • Response time increases with increasing number of resources • Response time is RESOURCE-DEPENDENT and the shown times are typical • MAY involve 1 or more Registry access which will increase overall response time • Increases rapidly as no. of resources > (150 – 200)

24. Performance EvaluationResults: Increasing Managers on Same machine

25. Performance EvaluationResults: Increasing Managers on different machines

26. Cluster of Messaging Nodes … How to scale locally ROOT US CGL Node-1 Node-2 … Node-N

27. Performance EvaluationResearch Question:How much infrastructure is required to manage N resources ? • N = Number of resources to manage • Z = Max. no. of entities connected to a single messaging node • D = Max. no of resources managed by a single manager process • K = min. no. of registry database instances required to provide fault-tolerance • Assume every leaf domain has 1 messaging node. Hence we have N/Z leaf domains. • Further, No. of managers required per leaf domain is Z/D • Thus total components at lowest level = Components Per domain * No. of Domains = (K + 1 Messaging Node + 1 Bootstrap Node + Z/D Managers) * N/Z = (2 + K + Z/D) * N/Z • Note: Other passive bootstrap nodes are not counted here since (No. of Passive Nodes) << N • E.g.: If it’s a shared registry, then the value of K = 1 for each domain which represents the service interface

28. Performance EvaluationResearch Question:How much infrastructure is required to manage N resources ? • Thus for N resources we require an additional (2 + K + Z/D) * N/Z resources • Thus percentage of additional infrastructure is = [(2 + K + Z/D)*N/Z] * 100% N + (2 + K + Z/D)*N/Z = [1 – 1/(1+2/Z+ K/Z + 1/D)] * 100 % • A Few Cases • Typical values of D and Z are 200 and 800 and assuming K = 4, then Additional Infrastructure = [1 – 1/(1 + 2/800 + 4/800 + 1/200)] * 100 % ≈ 1.23 % • When Registry is shared and there is one registry interface per domain, K = 1, then Additional Infrastructure = [1 – 1/(1 + 2/800 + 1/800 + 1/200)] * 100 % ≈ 0.87 % • If the resource manager can only manage 1 resource at any given instance, then D = 1, then Additional Infrastructure = [1 – 1/(1 + 2/800 + 4/800 + 1/1)] * 100 % ≈ 50%

29. Performance EvaluationXML Processing Overhead • XML Processing overhead is measured as the total marshalling and un-marshalling time required. • In case of Broker Management interactions, typical processing time (includes validation against schema) ≈ 5 ms • Broker Management operations invoked only during initialization and failure from recovery • Reading Broker State using a GET operation involves 5ms overhead and is invoked periodically (E.g. every 1 minute, depending on policy) • Further, for most operation dealing with changing broker state, actual operation processing time >> 5ms and hence the XML overhead of 5 ms is acceptable.

30. Prototype:Managing Grid Messaging Middleware • We illustrate the architecture by managing the distributed messaging middleware: NaradaBrokering • This example motivated by the presence of large number of dynamic peers (brokers) that need configuration and deployment in specific topologies • Runtime metrics provide dynamic hints on improving routing which leads to redeployment of messaging system (possibly) using a different configuration and topology • Can use (dynamically) optimized protocols (UDP v TCP v Parallel TCP) and go through firewalls but no good way to make choices dynamically • Broker Service Adapter • Note NB illustrates an electronic entity that didn’t start off with an administrative Service interface • So add wrapper over the basic NB BrokerNode object that provides WS – Management front-end • Allows CREATION, CONFIGURATION and MODIFICATION of broker topologies

31. A single participant sends audio / video … … … … 400 participants 400 participants 400 participants Prototype:Use Case • Use case I: Audio – Video Conferencing GlobalMMCS project, which uses NaradaBrokering as a event delivery substrate • Consider a scenario where there is a teacher and 10,000 students. One would typically form a TREE shaped hierarchy of brokers • One broker can support up to 400 simultaneous video clients and 1500 simultaneous audio clients with acceptable quality*. So one would need (10000 / 400 ≈ 25 broker nodes). • May also require additional links between brokers for fault-tolerance purposes • Use Case II: Sensor Network • Both use cases need high QoS streams of messages • Use Case III: Management System itself *“Scalable Service Oriented Architecture for Audio/Video Conferencing”, Ahmet Uyar, Ph.D. Thesis, May 2005

32. Failure HandlingWS - Policy • Policy defines resource failure handling • Implemented 2 policies (based on WS-Policy) • Require User Input: No action taken against failure. A user interaction is required to handle • Auto Instantiate: Tries auto instantiation of failed broker. • Location of a fork process is required.

33. Prototype:Costs (Individual Resources – Brokers)

34. Recovery time: • Assuming 5ms Read time from registry per resource object

35. Prototype:ObservedRecovery Cost per Resource • Time for Create Broker depends on the number & type of transports opened by the broker • E.g. SSL transport requires negotiation of keys and would require more time than simply opening a TCP connection • If brokers connect to other brokers, the destination broker MUST be ready to accept connections, else topology recovery takes more time.

36. Management Console:Creating Nodes and Setting Properties

37. Management Console:Creating Links

38. Management Console:Policies

39. Management Console:Creating Topologies

40. Related work • Fault-Tolerance Strategies • Replication • Provide transfer of control to a new or existing backup service instance on failure • Passive (primary / backup) OR Active • E.g. Distributed databases, agents-based systems

41. Related work • Fault-Tolerance Strategies • Replication • Check-pointing • Allow computation to continue from point of failure OR for process migration • E.g. MPI-based systems (Open MPI) • Can be done independently (easier to do but complicates recovery) OR co-ordinated (performance issue but recovery is easy)

42. Related work • Fault-Tolerance Strategies • Replication • Check-pointing • Request-Retry • Logging • Checksum

43. Related work • Fault-Tolerance Strategies • Failure Detection • Via periodic Heartbeats (E.g. Globus Heartbeat Monitor) • Scalability • Hierarchical organization (E.g. DNS) • Resource Management (Monitoring / Scheduling) • E.g. MonALISA, Globus GRAM

44. Conclusion • We have presented a scalable, fault-tolerant management framework that • Adds acceptable cost in terms of extra resources required (about 1%) • Provides a general framework for management of distributed resources • Is compatible with existing Web Service standard • We have applied our framework to manage resources that have modest external state • This assumption is important to improve scalability of management process

45. Summary Of Contributions • Designed and implemented a Resource Management Framework: • Tolerant to failures in management framework as well as resource failures by implementing resource specific policies • Scalable - In terms of number of additional resources required to provide fault-tolerance and performance • Implements Web Service Management to manage resources • Our implementation of global management by leveraging a scalable messaging substrate to traverse firewalls • Detailed evaluation of the system components to show that the proposed architecture has acceptable costs • The architecture adds (approx.) 1% extra resources • Implemented Prototype to illustrate management of a distributed messaging middleware system: NaradaBrokering

46. Future Work • Current work assumes SMALL runtime state that needs to be maintained. • Apply management framework and evaluate the system when this assumption does not hold true • More messages / Higher sized messages • XML processing overhead becomes significant • Apply the framework to broader domains

47. Publications • On the proposed work: • Scalable, Fault-tolerant Management in a Service Oriented Architecture Harshawardhan Gadgil, Geoffrey Fox, Shrideep Pallickara, Marlon PierceSubmitted to IPDPS 2007 • Managing Grid Messaging Middleware Harshawardhan Gadgil, Geoffrey Fox, Shrideep Pallickara, Marlon PierceIn Proceedings of “Challenges of Large Applications in Distributed Environments” (CLADE), pp. 83 - 91, June 19, 2006, Paris, France • Relevant to the proposed work: • A Scripting based Architecture for Management of Streams and Services in Real-time Grid Applications Harshawardhan Gadgil, Geoffrey Fox, Shrideep Pallickara, Marlon Pierce, Robert Granat In Proceedings of the IEEE/ACM Cluster Computing and Grid 2005 Conference, CCGrid 2005, Vol. 2, pp. 710-717, Cardiff, UK • On the Discovery of Brokers in Distributed Messaging Infrastructure Shrideep Pallickara, Harshawardhan Gadgil, Geoffrey FoxIn Proceedings of the IEEE Cluster 2005 Conference. Boston, MA • On the Discovery of Topics in Distributed Publish/Subscribe systems Shrideep Pallickara, Geoffrey Fox, Harshawardhan GadgilIn Proceedings of the 6th IEEE/ACM International Workshop on Grid Computing Grid 2005 Conference, pp. 25-32, Seattle, WA (Selected as one of six Best Papers) • A Framework for Secure End-to-End Delivery of Messages in Publish/Subscribe Systems Shrideep Pallickara, Marlon Pierce, Harshawardhan Gadgil, Geoffrey Fox, Yan Yan, Yi Huang (To Appear) In Proceedings of “The 7th IEEE/ACM International Conference on Grid Computing” (Grid 2006), Barcelona, September 28th-29th, 2006

48. Publications:Others • On the Secure Creation, Organization and Discovery of Topics in Distributed Publish/Subscribe systems Shrideep Pallickara, Geoffrey Fox, Harshawardhan Gadgil(To Appear) International Journal of High Performance Computing and Networking (IJHPCN), 2006. Special Issue of extended versions of the 6 best papers at the ACM/IEEE Grid 2005 Workshop • Building Messaging Substrates for Web and Grid ApplicationsGeoffrey Fox, Shrideep Pallickara, Marlon Pierce, Harshawardhan GadgilIn special Issue on Scientific Applications of Grid Computing in Philosophical Transactions of the Royal Society, London, Volume 363, Number 1833, pp 1757-1773, August 2005 • Management of Real-Time Streaming Data Grid Services Geoffrey Fox, Galip Aydin, Harshawardhan Gadgil, Shrideep Pallickara, Marlon Pierce, and Wenjun WuInvited talk at Fourth International Conference on Grid and Cooperative Computing (GCC2005), Beijing, China Nov 30-Dec 3, 2005, Lecture Notes in Computer Science, Volume 3795, Nov 2005, Pages 3 -12 • SERVOGrid Complexity Computational Environments(CCE) Integrated Performance AnalysisGalip Aydin, Mehmet S. Aktas, Geoffrey C. Fox, Harshawardhan Gadgil, Marlon Pierce, Ahmet SayarAs poster and In Proceedings of the 6th IEEE/ACM International Workshop on Grid Computing Grid2005 Conference, pp. 256 - 261, Seattle, WA, Nov 13 - 14, 2005