Collective Chunk IO support inside HDF5

1. Collective Chunk IO support inside HDF5

2. Outline • Background • Previous work review • Unsolved issues in the previous work • Solutions • Discussions: APIs to be added • Future work

3. Software Stack for PHDF5 Application

4. Previous collective IO support inside HDF5 • Only supports collective IO for regular hyperslab selection with contiguous data storage

5. Current collective IO support inside HDF5 • Both chunking and contiguous • Both regular and non-regular selection • For chunking storage, collective IO works per chunk.

6. Unsolved issues from previous work • Mainly in chunking storage • Severe performance penalties with many small chunk IOs • Bad performance caused by improper use of collective IO

7. Collective VS. Independent • Misconception: Collective IO is always better than independent IO • Correct view: case by case. • Collective IO communication overheads • inside HDF5 – one collective call • inside MPI-IO – must also have some communication calls. • Collective IO requirement • All processes need to participate

8. P0 P0 P1 P1 P2 P2 P3 P3 P0 P0 P1 P1 P2 P2 P3 P3 Good Case for collective IO

9. Bad performance case 1 Problem • many small chunk IOs Solution • Building an MPI derived data type across all chunks • Using collective IO

10. Schematic for using MPI DDT to support collective chunk IO inside Parallel HDF5 chunk 1 chunk 3 chunk 4 chunk 2 P0 chunk 2 chunk 3 chunk 4 chunk 1 P1 chunk 2 chunk 3 chunk 4 chunk 1 Collective View

11. Bad performance case 2 Problem • Pattern not fit for collective IO Solution • Analyzing patterns for collective • Using independent for bad patterns for collective

12. Implementation strategy • Better to have one-big IO • Not all MPI-IO packages support collective IO using DDT well • Big chunk size • Two implementations are needed • One-linked chunk IO • Building a DDT across all chunks • Collective IO • multiple chunk IOs • Doing IO per chunk • Analyzing IO patterns that are set to do collective • Change collective to independent for this chunk

13. Why giving options to users ? • Different chunking storage may amplify the performance difference • Applications we don’t know • Optimizations invoke overheads • Prevent abandonment of HDF5 • This only affects chunking storage

14. Optional APIs • Provide an option for users to join in the decision-making process • Must be clear • Must be efficient

15. Flow chart of Collective Chunking IO implementation inside HDF5 Collective chunk mode Optional User Input Decision-making about One linked-chunk IO Optional User Input NO Multi-chunk IO Decision-making about Collective IO Yes NO Yes One Collective IO call for all chunks Collective IO per chunk Independent IO

16. Decision-making:one-linked or multiple-chunked • Name: H5Pset_dxpl_mpio_chunk_opt_num • Signature: • herr_t H5Pset_dxpl_mpio_chunk_opt_num (hid_t dxpl_id, • unsigned num_chunk_per_proc) • Purpose: • To set a threshold for doing one-linked chunk IO • Description: • The library will calculate the average number of chunk selected by each process. If the number is greater than the threshold set by the user, the library will do one-linked chunk IO; otherwise, IO will be done for every chunk.

17. Decision-making:collective or independent for multi-chunk IO • Name: H5Pset_dxpl_mpio_chunk_opt_ratio • Signature: • herr_t H5Pset_dxpl_mpio_chunk_opt_ratio (hid_t dxpl_id, unsigned percent_num_proc_per_chunk) • Purpose: • To set a threshold for doing collective IO in each chunk • Description: • The library will calculate the percentage of the number of process holding selections at each chunk. If that percentage of number of process in the individual chunk is greater than the threshold set by the user, the library will do collective chunk IO for this chunk; otherwise, independent IO will be done for this chunk.

18. Decision-making:one-linked or multiple-chunked • Name: H5Pset_dxpl_mpio_chunk_opt • Signature: • herr_t H5Pset_dxpl_mpio_chunk_opt_num (hid_t dxpl_id, • unsigned flag) • Purpose: • To set a flag to choose one-linked chunk IO or multi-chunk IO without involving decision-making inside HDF5