Teamster-G: A Shared-Memory Parallel Programming Environment on Grids

1. Teamster-G: A Shared-Memory Parallel Programming Environment on Grids 高雄應用科技大學 電機系 梁廷宇

2. 原高雄工專，創立於1963年，1990年改大。現擁有「電資原高雄工專，創立於1963年，1990年改大。現擁有「電資 學院」、「工學院」、「管理學院」、「人文社會學院」 等四個學院、四個博士班、十八個碩士班、十一個產業研發 碩士專班、十八個系、以及二個研究中心。

4. Outline • Background • Teamster-G • Current State of Teamster-G • Next Step of Teamster-G • Conclusions

5. Background • What is a Computational Grid ? • Ensemble of geographically-distributed resources • Analogy to Power Grids • deliver computational power and data in a cheap and consistent way • Grid Features • single sign-on • heterogeneous • dynamic and shared • autonomy

6. Background • Why we need computational Grids ?

7. Background • LCG : LHC Computing Grids

8. Background (cont.) • Providing a familiar programming toolkit is an important factor to make users change their working environments from PCs, workstation, servers, SMPs or clusters to computational grids • RPC • Java RMI • MPI • DSM • OpenMP

9. page fault return page page request Background (cont.) • DSM (IVY, Li Kai, 1986) • problem: false sharing mprotect() P0 P1 ………………… Pn M0 M1 Mn DSMLIB0 DSMLIB1 DSMLIBn Virtual Shared Memory

10. Background (cont.) • OpenMP: an application programming interface (API) that supports fork-join multithreading programming in C/C++ (1998) and Fortran (1997). • Compiler directives • Runtime library • Environment variables

11. Background (cont.) • OpenMP program example #include<omp.h> #define NUM_START 1 #define NUM_END 100 int main() { int i; double total=0; #pragma omp parallel for reduction(+:total) for(i=NUM_START;i<=NUM_END;i++) { total=total+i; } printf("\n 1+2+3+4+.....+%d = %f\n",NUM_END,total); }

12. Background (cont.) • OpenMP Compiler int _ompc_main(){ auto double sumall; auto double total; auto int i; (total)=(0); auto void *__ompc_argv[1]; (*(__ompc_argv))=(((void *)(&total))); _ompc_do_parallel(__ompc_func_3,__ompc_argv); # 20 "sum_omp.c" printf("\012 1+2+3+4+.....+%d = %f\012",100,total); }

13. Background (cont.) • OpenMP Compiler __ompc_func_3(void **__ompc_args){ auto double *_pp_total; (_pp_total)=(((double *)(*__ompc_args)));{ { auto double _p_total; (_p_total)=(0.0);(_p_i_0)=(1);(_p_i_1)=((100)+(1));(_p_i_2)=(1); _ompc_default_sched(&_p_i_0,&_p_i_1,&_p_i_2); for((_p_i)=(_p_i_0);(_p_i)<(_p_i_1);(_p_i)+=(_p_i_2)){ # 16 "sum_omp.c" (_p_total)=((_p_total)+(((double )(_p_i)))); } _ompc_reduction(&_p_total,_pp_total); _ompc_barrier(); }

14. Background (cont.) • Grid-enabled programming toolkit • RPC  GridRPC • Java RMI  Grid RMI • MPIMPICH-G2 • DSM  ? • OpenMP  ? • It is necessary to provide the shared memory parallel programming interfaces for grids in order for • Minimizing the programming complexity of grid computing • Allowing users to change their working environment from SMP workstations or clusters to computational grids • Enriching grid applications

15. Background (cont.) • Teamster-G project : Enabling Shared Memory Programming Interface for Grid Computing • Pthread, OpenMP • Transparent resource discover/allocation

16. Outline • Background • Teamster-G • Current State of Teamster-G • Next Step of Teamster-G • Conclusions

17. Teamster-G • Teamster-G is developed based on Globus toolkit and Teamster. (KUAS, NCKU and Leader, 2004) • Two work packages • Enabling Teamster for grid computing • Porting OpenMP API on Teamster • Two challenges • Minimizing the communication cost of data-consistency maintenance • Keeping source-compatible for OpenMP programs

18. Teamster-G (cont.) • Globus toolkit • Resource management service : GRAM (Grid Resource Allocation and Management) • Information service : MDS (Meta-computing Directory Services) • Security service : GSI (Grid Security Infrastructure) • Data movement and management: GASS (Global Access to Secondary Storage)

19. Teamster-G (cont.) • Teamster (T.B. Chang, NCKU) • a user-level DSM system supporting Linux-based SMP clusters • global shared address space • eager-released consistency protocol • support POSIX thread

20. Teamster-G (cont.) • Components • TGRUN • TGRB (Teamster-G Resource Broker) • TGCM (Teamster-G Cluster Manager)

21. Teamster-G (cont.) • Resource allocation

22. Teamster-G (cont.) • Job execution

23. Teamster-G (cont.) • Data consistency Maintenance : minimizing the number of messages transferred through WANs • Cluster cache • inter-copyset • intra-copyset • Two-level consistency protocol • inter-cluster update propagation • intra-cluster update propagation • Hybrid communication protocol • TCP used in WAN connection • UDP used in LAN connection

24. Teamster-G (cont.) Page owner B A A X1 Y1 Z1 Owner A0 X1 Y1 Z1 Owner A0 A2 A2 Page Fault B2 B1 Return the page Forward page request Return the page Send page request Send page request Return the page Send page request Return the page Page Fault Page Fault Owner B0 On request X1 Y1 Z1 X1 Y1 Z1 Owner Cluster A Owner B0 X1 Y1 Z1 On request

25. Teamster-G (cont.) barrier Page owner Inter twin Intra twin X1 Y1 Z1 X1 Y1 Z1 X2 Y1 Z2 X1 Y1 Z1 X1 Y2 Z1 X2 Y2 Z2 X1 Y2 Z1 X2 Y2 Z2 B1 arrive no diff to flush B2 arrive A2 arrive no diff to flush B0 arrive All nodes in this cluster are arrived A0 arrive Cluster A All node are arrived diff Y2 Y2 groupdiff ask for inter copyset and flush group diff ask for intra copyset and flush diff forward diff flush diff forward diff flush diff flush group diff groupdiff flush diff flush diff forward diff diff diff X2 Z2 Z2 X2 Inter twin Intra twin twin X2 Y1 Z2 X2 Y2 Z2 X2 Y1 Z1 X1 Y1 Z1 X2 Y1 Z2 X1 Y1 Z2 X2 Y2 Z2 X1 Y1 Z2 X1 Y1 Z1 X2 Y2 Z2 X1 Y1 Z2 X2 Y1 Z2 X1 Y1 Z1 X1 Y1 Z1

26. Teamster-G (cont.) • Performance Evaluation

27. Teamster-G (cont.) • Performance

28. Teamster-G (cont.) Ratios of LAN and WAN communication in the N-body application

29. Teamster-G (cont.) • Porting OpenMP on Teamster int var1; int var2 = 0; int main() { int *data=(int*) malloc(sizeof(int)*10); #pragma omp parallel { int id = omp_get_thread_num(); } } Global variables Memory allocation function OpenMP directives OpenMP library functions

30. Teamster-G (cont.) • Four components is implemented • OpenMP Translator (Omni OpenMP compiler, Tsukuba, Japan) • OpenMP Runtime Library (OMPLIB) • Distributed Portable Thread Library (DPTHLIB ) • Load Balance Mechanism (LBM )

31. Teamster-G (cont.) • Compiling OpenMP programs on Teamster-G LBM Translated Source code OpenMP Translator DPTHLIB OMPLIB Make gcc DSMLIB OpenMP sources Object file linking Executable files

32. Teamster-G (cont.) • Verification

33. Teamster-G (cont.) • Performance evaluation

34. Teamster-G (cont.) • Performance evaluation

35. Outline • Background • Teamster-G • Current State of Teamster-G • Next Step of Teamster-G • Conclusion

36. Current State of Teamster-G • Teamster-G2 based on GT4 and integrated with well known Grid toolkits such as Ganglia, NWS, Condor, PBS, SGE etc. • Long-term predictor of resource-availability and performance • Applications self-adaptive • Data-Racing Avoidance and Deterministic Replay

37. Current State of Teamster-G

38. Current State of Teamster-G • Long-term predictor of resource availability • Log resource load • Transfer load history into chaincodes • Discover frequent workload pattern (FWPs) • Predict resource load states by applying • Association rules • Calculate the availabilities of resources.

39. Current State of Teamster-G • Effectiveness of the proposed predictor

40. Current State of Teamster-G • Impact of the proposed predictor

41. Current State of Teamster-G • Progressive Multilayer Reconfiguration • virtual processor layer • node layer • inter-cluster layer

42. Current State of Teamster-G • Data Racing Avoidance and Deterministic Replay

43. Current State of Teamster-G • Verification of DRARS

44. Current State of Teamster-G • Cost of DRARS

45. Outline • Background • Teamster-G • Current Status of Teamster-G • Next Step of Teamster-G • Conclusion

46. Next Step of Teamster-G • IDE : Debugger tool • Quality of Service • Checkpoint and Recovery • Integration of Data/File Sharing (Data Grids) • Integration of Memory Sharing (RAM Grids) • Applications : GridERP

47. Outline • Background • Teamster-G • Current State of Teamster-G • Next Step of Teamster-G • Conclusion

48. Conclusion • Teamster-G has successfully provided the shared memory parallel programming interfaces for grid computing. As a result, It allows users to easily move their Pthread/OpenMP programs from SMPs or clusters to computational grids for execution. • However, we must continue to put effort into the development of Teamster-G.

49. The End Thank you for your listening & any questions ?