Charm++ Arrays, Parameter Marshalling, Load Balancing and what they have in common-- PUP

1. Charm++ Arrays, Parameter Marshalling, Load Balancingand what they have in common-- PUP 9/1/2001 Orion Sky Lawlor PPL Developer Bachelor's in Math and CS from Univ. of Alaska at Fairbanks MS in CS from UIUC

2. Arrays: the Basic Idea • So there's some parallel object • How do you talk to it? • MPI approach: send a message to the processor number • Make migration (almost) impossible • Makes load balancing (almost) impossible • Array approach: send message to array index (1D, 2D, etc.)

3. Array Syntax (.ci) module demo { array [1D] A { entry A(void); entry void foo(void); } }

4. Array Syntax (.h) class A:public ArrayElement1D { public: A(void); void foo(void); //For migration: A(CkMigrateMessage *) { } void pup(PUP::er &p); };

5. Array Creation ... in main chare ... int n=17; CProxy_A ap=CProxy_A:: ckNew(n);//Create array //Elements numbered {0..n-1} //Always dense //Always 1D

6. Sparse Array Creation ... in main chare ... int n=17; CProxy_A ap=CProxy_A:: ckNew();//Create array for (int i=0;i<n;i++) ap[7+2*i].insert(); ap.doneInserting(); //Don't forget the doneInserting!

7. Array Messaging //Just index the array proxy ap[7].foo(); //Or can make element proxy: CProxyElement_A aep=ap[9]; aep.foo(); //For higher dimensions: ap2(x,y).foo(); ap3(x,y,z).foo();

8. Array Broadcast //Leave off the index: ap.foo(); • Invokes foo on every existing array element • Works even with migrations • Bad syntax: easy to broadcast by mistake, causing disaster!

9. Array Reduction int myData[2]={...}; contribute(2*sizeof(int), (void *)&myData, CkReduction::sum_int); • Called from every array element • Collects result to PE 0 & calls fn: void myCli(void *userData, int redLen,void *redData); ap.setReductionClient(myCli,userData);

10. Arrays PUP

11. PUP-- Wazzup? • Before PUP, had to write 3 functions: • size-- compute message size, in bytes • pack-- write object into message • unpack-- read object out of message • PUP combines all three into one • Basic contract: here are my fields (types, sizes, and a pointer) • C++ version uses operator overloading for very spartan syntax-- just parens() and bar|

12. PUP Single Fields class simple { int x; float y; public: simple() { } ... other methods ... void pup(PUP::er &p) { p|x; p|y; } }; PUPmarshall(simple);

13. PUP Arrays class withArray { public: int n; double *d;//Holds n doubles simple() {n=0;d=NULL;} ~simple() { delete[] d;} ... other methods ... void pup(PUP::er &p) { p|n; if (p.isUnpacking()) d=new double[n]; p(d,n); } }; PUPmarshall(withArray);

14. PUP Subobject class sub { public: simple *s;//Holds a subobject sub() {s=NULL;} ~sub() { delete s;} ... other methods ... void pup(PUP::er &p) { if (p.isUnpacking()) s=new simple; s->pup(p); } }; PUPmarshall(sub);

15. PUP::able & auto-allocate class A:public PUP::able { public: ... other methods ... A(CkMigrateMessage *) { } virtual void pup(PUP::er &p); PUPable_decl(A); }; PUPable_def(A); (...in an initcall routine...) PUPable_reg(A); (...in some object's pup...) p|a;

16. PUP::able & trees class A:public PUP::able { A *left,*right; ... as before ... }; void A::pup(PUP::er &p) { p|left; p|right; } • Packs, allocates, and unpacks left and right children automatically • Works properly with NULL pointers • Doesn't work for object graphs with cycles! (Never terminates)

17. PUP::able & inheritance class B:public A { ... the usual ... PUPable_decl(B); }; PUPable_def(B); (...in an initcall routine...) PUPable_reg(B); • Now an A pointer that actually points to a B object will be restored properly-- B::pup will be called. • Combines well with virtual functions.

18. PUP Param. Marshall

19. Parameter Marshalling entry void foo(void); entry void bar(int a,int b); entry void baz(float f[7]); entry void far(int n,double d[n]); entry void boo(int n,int m, unsigned char area[n*m]); entry void oof(char s[strlen(s)+1]); entry void fub(something_t s); • Arrays passed as pointers in C++ • Array length evaluated on call side • Can declare everything “const", “&"

20. Param. Marshalling & PUP entry void foo(T a, V b); • Gets translated into: //Send side: CProxy_A::foo(T &a, V &b) { ...make PUP::er &p... p|a; p|b; ...use resulting p... } //Recv. side: ...in call-foo function... T a; V b; p|a; p|b; obj->foo(a,b); • Because there’s a default bar operator, T and V can be C structs; which are copied as bytes. • Because you can overload the bar operator using PUP_Marshall, if T and V have pup routines they’ll be called properly! • Makes marshalling flexible.

21. Param. Marshall Load Balancing

22. Load Balancing • All you need is a working pup • Load balancer will migrate constantly • Create the load balancer with: //In the .ci file: extern module RefineLB; //In the .C file: #include "RefineLB.h" //In the main chare: CreateRefineLB(); //Also: CommLB, GreedyRefLB, HeapCentLB, MetisLB, NeighborLB

23. AtSync Load Balancing • When you need more control over when balancing happens. • Create balancer as usual, then: //In the element constructor: usesAtSync=CmiTrue; //When ready to balance call: AtSync(); //When computation resumes, void ResumeFromSync(void); //gets called on each element.

24. Conclusion • Arrays support messaging, broadcasts, reductions, and migration • Migration is for load balance • Migration uses PUP • Parameter marshalling uses PUP • Can get lots of functionality (almost) for free with PUP