1 / 32

NetCDF4 Performance Benchmark

NetCDF4 Performance Benchmark. Part I. Will the performance in netCDF4 comparable with that in netCDF3?. Configurations. Dataset 40 MB: 6 files 1 MB: 6 files Storage Layout Contiguous Chunked (HDF5 default cache size: 1 MB) Chunked (HDF5 cache size: 64 MB) System Cache. System Cache.

chelseas
Download Presentation

NetCDF4 Performance Benchmark

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NetCDF4 Performance Benchmark

  2. Part I • Will the performance in netCDF4 comparable with that in netCDF3?

  3. Configurations • Dataset • 40 MB: 6 files • 1 MB: 6 files • Storage Layout • Contiguous • Chunked (HDF5 default cache size: 1 MB) • Chunked (HDF5 cache size: 64 MB) • System Cache

  4. System Cache • On • Use all caches and buffers provided by kernel • Drop • “drop_caches” to read data from disk • “fsync” to write data into disk

  5. 10 cases

  6. Default Hyperslab • One big hyperslab is selected

  7. 1. Contiguous layout with cache

  8. 2. Contiguous layout w/o cache

  9. 3. Chunked layout with cache

  10. 4. Chunked layout w/o cache

  11. 5. Chunked layout with cache

  12. H5Pset_alloc_time(EARLY) H5Pset_alloc_time(EARLY)

  13. 6. Chunked layout w/o cache

  14. 7. Contiguous layout with cache

  15. 8. Contiguous layout w/o cache

  16. 9. Chunked layout with cache

  17. 10. Chunked layout w/o cache

  18. Part II • Can I get better performance with netCDF4? If yes, under what circumstances can I get better performance?

  19. 256 256 16 16384 240 1 Non-contiguous Access • Logical layout for 2-dimensional arrays

  20. Chunk size [4096][1] Chunk size [8192][1] Chunk size [16384][1] Non-contiguous Access • Physical layout

  21. 11. Non-contiguous Access

  22. 12. Chunked layout with cache

  23. 13. Compression

  24. 13. Compression • Compression ratio

  25. Part III • Can netCDF4 performance be bad? How can I avoid the bad performance?

  26. 14. Chunk size • Too small chunk size is bad • Little bit smaller than é(number of elements) / Nù is bad

  27. 3162 3162 791 790 14. Chunk size dataset chunk

  28. 14. Chunk size

  29. 14. Chunk size (more) é3162/nù + 1 é3162/nù é3162/nù - 1

  30. 15. Many Hyperslab selections H5Pcreate() H5Dopen()

  31. 15. Many Hyperslab selections

  32. Conclusion • The performance in netCDF4 is comparable with that in netCDF3 • Improvement • Non-contiguous access pattern • Adjusted cache size • Compression • Pitfall • Small chunk size • Many small hyperslab selections

More Related