1 / 43

Presenter: Khalid Saleem

Enabling autonomic Grid Applications: Requirements, Models and Infrastructure Authors: M. Parashar, Z. Li, H. Liu, V. Matossian and C. Schmidt. Presenter: Khalid Saleem. Introduction. Investigate challenges and requirements of programming Grid Applications Self-Management Applications

hamlet
Download Presentation

Presenter: Khalid Saleem

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Enabling autonomic Grid Applications: Requirements, Models and InfrastructureAuthors:M. Parashar, Z. Li, H. Liu, V. Matossian and C. Schmidt Presenter: Khalid Saleem

  2. Introduction • Investigate challenges and requirements of programming Grid Applications • Self-Management Applications • Introduce Project Automate • Investigates Autonomic solutions to deal with • Complexity • Dynamism • Heterogeneity • Uncertainty

  3. Grid Computing – Challenges & Requirements (1) • Goal: • Applications utilizing intellectual and physical resources across different disciplines and organizations • Providing more effective and efficient solutions to important Scientific, Engineering, Business and Government problems • Built on Seamless and secure mechanisms for • Discovery • Access • Interactions

  4. Grid Computing – Challenges & Requirements (2) • Key characteristics of Grid Execution Environment and Applications are • Heterogeneity • Dynamism • Uncertainty • Security

  5. Grid Computing – Challenges & Requirements (3) • Goals attainment requires • Implementation Models • Conceptual Models • Implementation Models address • Virtualization of organizations (GRIDs) • Creation and Management of virtual organizations • Instantiation of a virtual machine as the execution environment of an application • Conceptual Models define • Abstract machines supporting programming models and systems enabling application development

  6. Grid Computing – Challenges & Requirements (4) • Requirements for Grid Programming Systems • Specify applications that can detect and dynamically respond during execution to the changes in • state of execution environment • state and requirements of the application • Grid Applications should • Constitute of discrete, self managing components • Provide separate specifications for functional, non-functional, interaction and coordination behaviors • Provide implementation independent interface definitions for above specifications

  7. Requirements for Self-managing Applications on the Grid • Self-managing applications on the Grid require • Static application requirements • System and Application behaviors should be relaxed • Sensitivity and Ability of elements and applications to adapt to the • Dynamic state of the system; and • Changing requirements of the application • Essential common knowledge be expressed semantically (i.e. Ontology) rather than names addresses and identifiers • Core Middleware services (e.g. Discovery) driven by semantic knowledge • AutoMate • An attempt to address the above challenges and requirements

  8. AutoMate • Investigates autonomic solutions to address challenges posed to Grid Computing • Based on Biological Systems • Development of Conceptual and Implementation models/architectures • Programming Models, Framework and Middleware Services • Definition of autonomic elements • Development of autonomic applications • Policy, Content and Context driven application definition, management and execution

  9. Components of AutoMate

  10. Accord: A Programming Framework for Autonomic Applications • Extends existing distributed programming systems • Realizes • Separation of computations from coordination and interactions • Separation of non-functional behaviors (e.g. Resource requirements, performance) from functional behaviors • Separation of policy and mechanism • Accord Programming Model key components • Autonomic Elements • Accord Rules • Autonomic Composition

  11. Autonomic Elements (1) • Autonomic Element • Extends programming elements • Objects, Components and Services • Defines self-contained Modular software unit with • Specified interfaces • Explicit context dependencies • Encapsulates • Rules • Constraints • Mechanisms • Can dynamically interact with other elements and the system

  12. Autonomic Elements(2) Autonomic Element

  13. Autonomic Elements (3) • Autonomic Element defined by 3 ports • Functional Port • Defines a set of functional behaviors provided and used by the element • Control Port • Defines a set of sensors/actuators and the corresponding access constraints • Sensors: Interfaces providing info about the elements • Actuators: Interfaces for modifying state of the elements • Operational Port • Defines the interfaces to formulate and manage rules that govern the runtime behavior of the elements and their interactions.

  14. Autonomic Elements (4) • Element Manager • Manages element execution • Monitors state of the element and its context • Controls execution rules • Cooperates with other element managers

  15. Accord Rules (1) • Rules in the form • IF condition THEN action • Condition: Logical combination of sensors, elements, function interfaces and events • Action: Sequence of invocations of element and/or system actuators and other interfaces • Priority based mechanism to resolve conflicts

  16. Accord Rules (2) • Rule Classes • Behavioral Rules • Controls runtime functional behaviors of an autonomic element • Executed by the element manager within a single element • Interaction Rules • Controls interactions and coordination of autonomic elements • Element managers collaborate to provide coordinated execution

  17. Autonomic Composition (1) • Enables relationship between elements to be established and modified at runtime • Dynamic composition consists of composition plan and execution • Plans created at runtime • Based on dynamically defined objectives, policies and the context and content of applications and systems • Plan execution involves • Discovering elements, configuring them and defining interaction relationships and mechanisms

  18. Autonomic Composition (2) • Composition Plans • Generated using Accord Composition Engine (ACE) • Expressed in XML • Element Discovery uses • Meteor content-based Middleware • Squid Discovery Service • Plan execution • P2P control network of element managers and agents within Rudder • Library of rule sets • Common control and communication relationships between elements • Runtime Negotiation Protocols • Address runtime conflicts and conflicting decisions

  19. Accord Implementation Issues • Accord allows interaction & coordination behaviors to be managed at runtime using rules • Deploying and executing rules • Impacts performance • Increases robustness of applications; and • Ability to manage dynamism • Runtime changes to interaction and coordination behaviors relatively infrequent and have small overheads • Time spent in establishing and modifying interactions is small as compared to computation times • References [35, 36]

  20. Rudder Coordination Framework • Scalable coordination middleware • Supports self managing applications in decentralized distributed environment • Consists of two key components • COMET • Enables flexible and scalable coordination among agents and autonomic elements • Agent Framework • Composed of software agents, agent interaction and negotiation protocols

  21. COMET (1) • Provides a global virtual shared coordination space • Accessible by all peer agents, independent of physical location of tuples or identifiers of the host • Build on an associative messaging substrate • Implements a distributed hash table • Index space generated from semantic information space (ontology) • Can be transient and dynamically constructed • Provides an associative communication abstraction • Maps virtual information space to a dynamic set of available peer nodes • Maintains content locality

  22. COMET (2) • Provides a coordination abstraction layer • Extends traditional data-driven model with event based reactivity to changes in • System state • Data access operations • Defines a Reactive tuple abstraction consisting of • Condition that associates Reaction to events • A Guard specifying how and when the Reaction will be executed • Provides Linda-like primitives http://en.wikipedia.org/wiki/Linda_(coordination_language)

  23. Agent Framework (1) • Composed of Dynamic Network of Software Agents existing at different levels • Agents are • Processing units • Perform actions based on dynamically defined rules • Agents • Monitor element states • Manage element behaviors and dependencies • Coordinate element interactions • Cooperate to manage overall system/application behavior • Agents use profiles to • Identify and describe elements • Interact with elements • Control elements

  24. Agent Framework (2) • Defines a set of protocols for • Agent coordination • Application/system management • Examples of protocols include • Discovery protocols • Control Protocols

  25. Meteor: A content-based Middleware • Scalable content-based Middleware infrastructure • Consists of • Self organizing content overlay • Content based routing engine and discovery service (Squid) • Associative Rendezvous Messaging Substrate (ARMS)

  26. Meteor Content Overlay • Composed of Rendezvous Peer nodes (RP) • RP nodes can join or leave the network at anytime • Provides a single operation • Lookup (identifier) • Locates the peer node where the content should be stored or fetched

  27. Squid • Content based routing engine and decentralized discovery service • Supports • Flexible content-based routing • Complex queries containing partial keywords, wildcards and ranges • Keywords can be common words or globally defined attributes based on ontologies or taxonomies • Uses Hilbert Space Filling Curve (SFC) • Maps multidimensional information space to peer identifier space

  28. Associative Rendezvous Messaging Substrate (ARMS) • Implements Associative Rendezvous (AR) interaction paradigm • Content-based decoupled interactions with programmable reactive behaviors • Allows for asynchronous interactions among senders and receivers • Extends conventional name/identifier based rendezvous by • Using flexible combination of keywords, wildcards and ranges from the semantic information space instead of identifiers • Enabling reactive behaviors at rendezvous points encapsulated within messages, thus increasing flexibility and enabling multiple interaction semantics (broadcast, multicast)

  29. Autonomic Grid Applications

  30. Autonomic Oil Reservoir Optimization (1) • Problems • Selection of appropriate optimization algorithms • Runtime configuration and invocation of these algorithms • Dynamic optimization of reservoir • Autonomic application based on AutoMate consists of • Sophisticated reservoir simulation components • Encapsulates complex mathematical models • Execution on Grid • Grid Services providing secure and coordinated access to resources • Distributed data archives • Stores historical, experimental and observed data • Sensors embedded in the instrumented oilfield • Provides real time data about the current state of oil field • External services providing data relevant to optimization of oil production or the economic profits • Optimization based on VFSA and SPSA algorithms • Actions of scientists, engineers in field, labs and management offices

  31. Autonomic Oil Reservoir Optimization (2) • Peers automatically • Detect sub-optimal production behaviors at runtime • Adjust interactions to correct sub-optimal production • Uses policies to • Discover, select, configure and invoke appropriate optimization services to determine optimal well location

  32. Autonomic Forest Fire Management Simulation (1) • Predicts the speed, direction and intensity of fire front • Uses dynamic environment and vegetation conditions • Consists of • Data Space Manager (DSM) • Partitions the forest represented by 2D data space into sub spaces based on information from CRM • Computational Resource Manager (CRM) • Provides system resources information to DSM • Rothermel • Simulates in parallel the fire spread in each sub space • Uses current wind direction and intensity info simulated by Wind Model • Wind Model • Provides wind direction and intensity simulations • GUI • Allows experts to interact with the above elements

  33. Autonomic Forest Fire Management Simulation (2) Implementation of Accord Port types

  34. Autonomic Forest Fire Management Simulation (3) Component Addition and Change in Interaction Relationship Example

  35. Conclusion • AutoMate • Implementation architecture and conceptual models that enable the development and execution of self-managing Grid applications

  36. Agnostic Question 1 • How does AutoMate deal with the heterogeneity among all the elements of a grid? How does AutoMate integrate so many heterogeneous autonomic components to work together? • By separating the policy from mechanism. Policies in the form of rules are used to orchestrate a repertoire of mechanisms to achieve context-aware adaptive runtime computational behaviors and coordination and interaction relationships based on functional, performance, and QoS requirements thus responding to heterogeneity and dynamics. Uses Dynamic composition to enable relationships between elements (via element managers) to be established and modified at runtime

  37. Agnostic Question 2 • Security and trust is one of the key issues in grid environments, and the maintenance of users and permissions may become a bottleneck. Does AutoMate provide a specific autonomic solution for a self-maintenance of users and permissions across administrative boundaries? • AutoMate does not explicitly provide a methodology for self maintenance of users and permissions across administrative boundaries. However, SESAME that is embedded into AutoMate allows for setting up security policies and permissions for avoiding fraud and intrusion.

  38. Agnostic Question 3 • The Accord framework incorporates practical human knowledge in the form of behavioral rules, but with the vastness of communication paradigms (e.g., RPC, RMI, publish/subscribe), how does Accord ensure the correctness of these rules? • “To ensure rule correctness, we design a rule template for each communication paradigm (e.g., RPC/RMI, publish/subscribe) and coordination pattern (e.g., conditional branch, loop, sequence, parallel execution). A rule template is instantiated via filling in parameters, such as the names of interaction parties and the exchanging data.” http://www.caip.rutgers.edu/~marialiu/Projects/Accord/index.htm

  39. Agnostic Question 4 • AutoMate utilizes a peer-to-peer middleware to support and enable the autonomic interactions among components. Can you think of a better architecture to achieve this goal? • In this scenario using peer-to-peer would be a better option as AutoMate is based upon autonomic agent based control networks. Element managers execute rules to establish control and communication relationships among these elements in a decentralized and parallel manner.

  40. Agnostic Question 5 • Does Accord target a specific programming language? • Nothing particular. However, the prototype implementation was done using C++ and MPI

  41. Agnostic Question 6 • Can you extend on how AutoMate lets Globus-enabled grids to achieve an autonomic management? What elements of the AutoMate architecture communicate with Globus elements? • AutoMate makes use of its Accord autonomic elements and rule set, the Rudder Agent Framework and COMET to provide dynamic flexible and scaleable coordination among the peer nodes along with the use of conflict resolution and negotiation protocols. No specific information mapping the components of AutoMate to that of Globus has been provided.

  42. Agnostic Question 7 • Are you aware of other projects with the same goals than AutoMate? • Grid MP by United Devices http://www.ud.com/products/gridmp_fabs.php • Optimal Grid http://www-128.ibm.com/developerworks/library/gr-opgrid/ • GridARM for Structured Adaptive Mesh Refinement (SAMR) application • OGSA to some extent

  43. Agnostic Question 8 • On a previous presentation of the same authors, we discussed that their so-called peer-to-peer architecture was in fact Client-Server. Do you think it is correct that they somewhat reuse this architecture for the Meteor and COMET components? • The previous presentation focused on PAWN where as the authors made use of Meteor in the current framework. Meteor is based on JXTA which is a general purpose P2P framework. Hence, it seems correct to use Meteor.

More Related