1 / 31

High Performance Event Service for CCA Framework: Design and Experiences

High Performance Event Service for CCA Framework: Design and Experiences. Khushbu Agarwal Manoj Krishnan Daniel Chavarria Ian Gorton. Outline . Event Service Overview Design and Implementation Design Challenges Preliminary Results Conclusion. CCA Event Service 101. Publish-subscribe

dylan-williamson
Download Presentation

High Performance Event Service for CCA Framework: Design and Experiences

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. High Performance Event Service for CCA Framework: Design and Experiences Khushbu Agarwal Manoj Krishnan Daniel Chavarria Ian Gorton

  2. Outline • Event Service Overview • Design and Implementation • Design Challenges • Preliminary Results • Conclusion 2

  3. CCA Event Service 101 • Publish-subscribe • 1-n, n-m, n-1 • Specification is similar to: • Many distributed event/messaging services 3

  4. Possible use cases • Event/message distribution between components in the same framework • Initial SciRun implementation • Event/message distribution across processes in a HPC application • Across address spaces • Needs to be fast • Handle a range of potential payload sizes • Event/messaging service schizophrenia!! • Other work exists … • ECho • Grid event service 4

  5. CCA Event Service • Started with Utah CCA/SciRun event service implementation • Created two standalone prototypes (no SIDL, no framework): (‘08) • Reliable: events transferred via files • Fast: events transferred over ARMCI on Cray XD1 • Single-sided memory transfers • Now: Event Service in CCA Framework: • Framework design based on specification provided by CCA forum 5

  6. Sidl Specification • Interfaces : • Event • EventListener • EventServiceException • Subscriber • Topic • Ports • interface PublisherEventService extends cca.Port {} • interface SubscriberEventService extends cca.Port { } 6

  7. Modifications from Sidl Specification : • Modifications to Interfaces • Topic /* The wildcard topic management is not implemented yet */ • Publisher • Needed by PublisherEventService • ESTypeMap --extends=gov.cca.TypeMap • Added Relocate() • Added Components • EventService --provides=PublisherEventService, SubscriberEventService • Driver --go=run --uses=SubscriberEventService ,PublisherEventService 7

  8. Implementation Experience • Used sidl file provided at the CCA forum website • Bocca create EventService --import-sidl = EventService.sidl Doesn't work LL1: The sidl file needs to be separated into multiple files (for each interface) • bocca create interface Event --import-sidl=Event.sidl • bocca create class CCAEvent --implements=Event .... • bocca create interface Subscriber --import-sidl=Subscriber.sidl • bocca create class CCASubscriber --implements=Subscriber .... 8

  9. Implementation Experiences (contd.) • Specifying dependencies during creation Example: bocca create interface A1 {func ();} bocca create interface A2 – requires=A1 { A1 obj_1; } bocca create interface A3 – requires =A2 { A2. obj1.func(); //Error, undefined A1 } 9

  10. Implementation Experience (contd.) • Solution: • Re-do create interface with explicit dependencies bocca create interface A3 – requires =A2,A1 and , include <A1> explicitly in A3’s implementation file Example: • bocca create class CCASubscriber --implements=Subscriber --requires=CCAESTypeMap --requires=CCAEvent LL2: Specify dependencies for all levels explicitly LL3: Cannot modify dependencies on fly 10

  11. Implementation Experiences (contd.) • Accessing data not defined in sidl file. E.g.: Sidl File: interface A { } A_Impl.cxx: CCA class A_impl{ ComplexType x; // Class A, member variable x A_func1(ComplexType y); // member function A_func1 } C_Impl.cxx : CCA class C_impl{ ComplexType p =A_obj.x; // Error: Class A does not define x A_obj.A_func1(); // Error: Class A does not define A_func1 } 11

  12. Implementation: Experiences(contd.) Solution: Access data through member methods only Define member method parameters using opaque A Sidl File: interface A { opaque getComplexType(); void setComplexType(in opaque); } A_Impl.cxx: CCA class A_impl{ ComplexType x; }; 12

  13. Implementation: Experiences(contd.) C_Impl.cxx: CCA class C_impl{ func() { ComplexType* y = (ComplexType*)A::getComplexType(); } }; LL4: Changes made to _Impl.xxx files are not visible to other classes unless specified in sidl file 13

  14. Implementation: Experiences(contd.) • Returning CCA objects as pointers C++ CCA Opaque func1(){ CCA_class_A *p = &(CCA_class_A::_create()); ……… return p; } class_A* func1(){ class_A *p = new class_A; …… return p; } • CCA implementation: func1() returns invalid pointer. Why? • CCA object is destroyed at the end of function. • Solution :Return objects, instead of pointers. • Edit class sidl file: • bocca edit CCAESTypeMap CCA_class_A func1(){ CCA_class_A p = CCA_class_A::_create(); ……… return p; } 14

  15. Design Challenges • Sidl file does not allow defining variables for non-primitive data types (e.g from libraries) • New constructors cannot be defined in sidl file, not accessible if defined in _Impl.xxx files) • Introduce init() function calls for all classes • Object creation • Use _ create() • New operator does not work • Explicit freeing memory for a Babel object, introduces segfaults. • auto de-allocation when object goes out of scope • Used dummy namespace to trigger destructor of EventService Object. • Directory process need to receive Quit eventservice message 15

  16. Mapping CCA objects to different address space • Challenges • CCA objects contain implicit pointers • CCA objects have dynamic allocation of some data (change size)? • Solution 1 • Sender unpacks and transfers elementary data. Receiver reconstructs required CCA object • Possible, but tricky in case of nested CCA objects • Example: Trying to transfer CCA Events between address spaces requires transferring ESTypeMaps which are represented using two FixedMaps, which in turn store primitive data 16

  17. Mapping CCA objects to different address space (contd.) – Potential Solutions • Solution 2 • Babel RMI -> Clean but potentially lower performance solution for HPC needs • Solution 3 • Babel support could be extended to provide RMI on MPI/ARMCI • Solution4 • Babel may provide an abstraction that returns a pointer to a flat C++ object given a CCA object 17

  18. Preliminary Result 4 processes • CCA Process 0: Total EventService Time 129.041195 (ms) Process 2: Total EventService Time 129.059076 (ms) Process 1: Total EventService Time 129.112005 (ms) Process 3: Total EventService Time 129.111052 (ms) • C++ Process 0: Total EventService Time 1.814842 (ms) Process 1: Total EventService Time 1.585007 (ms) Process 2: Total EventService Time 2.148151 (ms) Process 3: Total EventService Time 1.943827 (ms) 18

  19. Conclusion • The transfer of CCA objects into multiple address spaces efficiently, remains to be addressed. • The efficiency of event service design is somewhat limited by the CCA constraints. • The event service implementation is similar to the CCA provided specification. 19

  20. Questions ? 20

  21. More:: 21

  22. ARMCI Prototype • Goals: • maintain interface/semantics of the event service model • achieve high performance in a distributed memory HPC system • Used combination of MPI & ARMCI • MPI - Process 0 operates as a Topic Directory process • Maintains a Topic List with the locations of the publishers • Uses an MPI messaging protocol to serve topic creation requests and queries • ARMCI - Publishers create events locally in their own address space • Subscribers read remote events from the publishers using one-sided ARMCI_Get() operations • no need for coordination with the publisher 22

  23. ARMCI Prototype (cont.) • Used a combination of MPI & ARMCI to create the event service • Transfer C++ class instances directly over ARMCI without the need for type serialization • Events comprise two TypeMaps: header and body • Created a special heap manager for the ARMCI address space • objects can be allocated directly through standard new() and delete() operators • synchronous garbage collection by the publisher • For high performance, all objects in the ARMCI heap are flattened • no pointers or references to external objects • member variables embedded • fixed size 23

  24. Initial Performance Results • We measured event processing rates: • 66K events/second with one publisher/one subscriber (small event 4KB) • 950 events/second with one publisher/16 subscribers (large event 50KB) • Minimal overhead to reconstruct the object on the subscriber after the transfer 24

  25. Analysis • Performance drops as number of subscribers increases • Contention for events at publisher ARMCI memory • Alternatives implementations are possible: • Maintain topics for subscribers only in local ARMCI memory • Publishers write to subscriber memory directly for each event published 25

  26. Master topic list Topic 1 Sub1 Sub2 Topic 2 Sub1 Sub3 • ARMCI Subscriber Buffer Topic struct Topic1 mess mess next Topic2 Publisher Subscriber Alternative Design Maintain topic list in process 0 (using MPI) or ARMCI shared memory? Send() • Weaknesses? • Publish can fail if subscriber memory full • Some subscribers slower than others - events delivered unpredictably depending on consumption rate Strengths? Likely reduced contention Simplifies ‘publish semantics’ and event retention issues 26

  27. Polygraph Issues: Delivery Semantics • Basic pub-sub good for N-to-N event distribution • Need to keep events until all subscribers consume them • Optional ‘time-to-live’ in header can help • Workload distribution use cases require ‘load-balancing’ topics • Same programmatic interface • Each event consumed by only one subscriber • No complex event retention issues • Could define load-balancing policies for publishers • Declaratively? • A ‘one-to-one’ queue-like mechanism may also be useful? 27

  28. Issues: Topic Memory Management • Managing memory for a topic is tricky: • Need to know how many subscribers for each specific event • Events are variable size, hence allocating/reclaiming memory for events is complex • One possibility: typed topics • Associate an event type with a topic • Specify maximum size for any event • Simplifies memory management for each topic 28

  29. Issues - Miscellaneous • What are semantics when a new subscriber subscribes to a topic? • What exactly do they see? • All messages in topic queue at subscription time? • Only new ones? • In ARMCI implementation, memory for topic queues is finite • Should it be user-configurable? • What happens when topic memory full? • Standard publish error defined by Event Service? 29

  30. Issues - Miscellaneous • Event Service SIDL doesn’t clearly demarcate if there are: • Calls for publishers only? • Calls for subscribers only? • So … what happens if: • A publisher calls ReleaseTopic()? • A publisher calls ProcessEvents()? • How can CreateTopic() fail? • Two publishers call CreateTopic in a non-deterministic sequence. What happens? • Can a subscriber call CreateTopic()? • Why is argument to ReleaseTopic() a string? • Would a valid Topic reference be less error-prone/simpler? • Should events have a ‘standard’ header • Used by all event service implementations • Not settable programmatically • E.g. Time-to-live, timestamp, correlation-id, likely others … 30

  31. Next steps … • Implement alternative ‘subscriber side’ ARMCI implementation • Detailed performance analysis … • Use Event Service to implement PolyGraph use case 31

More Related