Array Operation Synthesis to Optimize Data Parallel Programs

1. Array Operation Synthesis to Optimize Data Parallel Programs Department of Computer Science, National Tsing-Hua University Student:Gwan-Hwan Hwang Advisor: Dr. Jenq Kuen Lee

2. Array Operation Synthesis to Optimize Data Parallel Programs 國立清華大學 資訊工程系 Student:黃冠寰 Advisor: 李政崑博士

3. Array Operation Synthesis on Distributed-memory Machines 國立清華大學 資訊工程學系 黃冠寰, Phd.

4. Research Interests • Key Issues • •Array Operation Synthesis for Intrinsic Array • Functions (JPDC, ACM PPoPP’95, ICPP’96) • Compiler Optimization for • Parallel Computations on • Distributed & Shared Memory • Machines • •Automatic Alignment for Data Parallel • Languages (LCPC’97) • •Communication Code for Block-Cyclic • Distribution of HPF(IPPS’98) • Concurrent Testing • •Reachability Testing of Concurrent Program • (IJSEKE’95, APSEC’93) • Parallel Object Program Model & • Heterogeneous Computing • •Java-Based Network Computing Environment • •Transparent Parallel Computing Environment • (Ongoing)

5. Outline of Presentation • Fortran 90 Intrinsic Array Operations • Array Operation Synthesis(AOS) • SYNTOOL • Apply AOS to Shared-Memory Machines • Apply AOS to Distributed-Memory Machines • Conclusion and Future Work

6. Outline of Presentation • Fortran 90 Intrinsic Array Operations • Array Operation Synthesis(AOS) • SYNTOOL • Apply AOS to Shared-Memory Machines • Apply AOS to Distributed-Memory Machines • Integrate AOS with Automatic Data Alignment • Conclusion and Future Work

7. Intrinsic Array Operations • Provided by Modern Program Languages. • E.g. Fortran 90, High Performance Fortran(HPF), HPF2, Fortran 97, APL, MATLAB, MATHEMATICA, NESL, C* • Engineering and Scientific Applications • Facilitate a Compilation Analysis for Optimization • Support Parallel Execution and Portability

8. Intrinsic Array Operations(Cont’d) CSHIFT, TRANSPOSE, MERGE, EOSHIFT, RESHAPE SPREAD, Section Move, Where Constructs, Reductions. • Array Operations Provided by Fortran 90, HPF. • Examples: B=CSHIFT(A,1,1) C=TRANSPOSE(B)

9. Consecutive Array Expressions • Array Expression • Consecutive Array Operations C=EOSHIFT(MERGE(RESHAPE(S,/N,N/),A+B,T),1,0,1) FXP=CSHIFT(F1,1,+1) FXM=CSHIFT(F1,1,-1) FXP=CSHIFT(F1,2,+1) FYM=CSHIFT(F1,2,-1) FDERIV=ZXP*(FXP-F1)+ZXM*(FXM-F1)+ ZYP*(FYP-F1)+ZYM*(FYM-F1)

10. Classification of Array Operations • Model Array Operations by Data Access Functions(DAF) Type 4 Type 2 Type 1 Type 3

11. Data Access Functions • Represent Array Operations by Mathematical Functions • Model Array Operations by Data Access Functions(DAF) • Single-Source, Multiple-Source • Single-Clause, multiple-Clause

12. Type 1: Single-source Single-clause Data Access Function • One Source Array • One Data Access Pattern B=TRANSPOSE(A) Data Access Function is B(I,J)=A(J,I)

13. Single-source Single-clause Data Access Function • One Source Array • One Data Access Pattern B=TRANSPOSE(A) Data Access Function is B(I,J)=A(J,I)

14. Type 2: Multiple-source Single-clause Data Access Function • Multiple Source Arrays • One Data Access Pattern R=MERGE(T,F,M) Array T Array F Array M Array R Data Access Function is where

15. Multiple-source Single-clause Data Access Function • Multiple Source Arrays • One Data Access Pattern R=MERGE(T,F,M) Array T Array F Array M Array R Data Access Function is where

16. Type 3: Single-source Multiple-clause Data Access Function • Single Source Array • Multiple Data Access Patterns B=CSHIFT(A,1,1) Array A Array B Data Access Function is : a segmentation descriptor

17. Single-source Multiple-clause Data Access Function • Single Source Array • Multiple Data Access Patterns B=CSHIFT(A,1,1) Array A Array B Data Access Function is : a segmentation descriptor

18. Type 4: Multiple-source Multiple-clause Data Access Function • Multiple Source Arrays • Multiple Data Access Patterns • No array operation of Fortran 90 belongs to type 4 • Synthesis of multiple array operations may derive a type 4 data access function.

19. Multiple-source Multiple-clause Data Access Function • Multiple Source Arrays • Multiple Data Access Patterns • No array operation of Fortran 90 belongs to this type • Synthesis of multiple array operations may derive a multiple-source multiple-clause data access function

20. Straightforward Compilation B=CSHIFT((TRANSPOSE(EOSHIFT(A,1,0,1),1,1) • Translate each operation into a parallel loop FORALL (I=1:N:1; J=1:N:1) IF (1<=I<=N-1) and (1<=J<=N) THEN T1(I,J)=A(I+1,J) ELSE T1(I,J)=0 ENDFORALL EOSHIFT FORALL (I=1:N:1; J=1:N:1) T2(I,J)=T1(J,I) ENDFORALL TRANSPOSE FORALL (I=1:N:1; J=1:N:1) IF (1<=I<=N-1) and (1<=J<=N) THEN B(I,J)=T2(I+1,J) ELSE B(I,J)=T2(I-N,J) ENDFORALL CSHIFT

21. Array Operation Synthesis B=CSHIFT((TRANSPOSE(EOSHIFT(A,1,0,1),1,1) • Construct the Parse Tree of Array Expression • Represent Array Operations by Mathematical Functions (DAF) EOSHIFT TRANSPOSE CSHIFT

22. Array Operation Synthesis (Cont’d) Synthesis of two functions CSHIFT TRANSPOSE COSHIFT+ TRANSPOSE EOSHIFT

23. Synthesis of two Data Access Functions • Substitution (Term Rewriting like method) • Having two Data Access Patterns: • The Synthesized Data Access Pattern is: where where

24. Synthesis of two DAFs (Cont’d) • For example, • By the substitution rule 

25. Synthesis of two DAFs (Cont’d) • For example,

26. Code Generation for Synthesized Data Access Function Code Generation FORALL (I=1:N:1; J=1:N:1) IF (/I,J/,/1:N-1,1:N/) (/J,I+1/,/1:N-1,1:N/) THEN B(I,J)=A(J+1,I+1) IF (/I,J/,/1:N-1,1:N/) (/J,I+1/,/N:N ,1:N/) THEN B(I,J)=0 IF (/I,J/,/N:N ,1:N/) (/J,I+1/,/1:N-1,1:N/) THEN B(I,J)=A(J+1,I-N+1) IF (/I,J/,/N:N ,1:N/) (/J,I+1/,/N:N ,1:N/) THEN B(I,J)=0 ENDFORALL

27. Code Generation for Synthesized Data Access Function After Optimization 1 N-1 N 1 N-1 N

28. Optimization • Simplifying the ranges at compilation time instead of runtime • Optimization process: • Normalize: • Intersection for each dimension: