John Clyne & Alan Norton Scientific Computing Division National Center for Atmospheric Research

1. VAPORVisualization and Analysis Platform for Ocean, atmosphere, and solar ResearchSC06 Ultra-Scale Visualization Workshop John Clyne & Alan Norton Scientific Computing Division National Center for Atmospheric Research Boulder, CO USA This work is funded in part through a U.S. National Science Foundation, Information Technology Research program grant clyne@ncar.ucar.edu

2. Problem • Numerical simulations in the earth sciences have reached such extraordinary sizes that researchers can no longer effectively extract insight from their simulation outputs. • Result: loss of scientific productivity!!! [Numerical] models that can currently be run on typical supercomputing platforms produce data in amounts that make storage expensive, movement cumbersome, visualization difficult, and detailed analysis impossible.  The result is a significantly reduced scientific return from the nation's largest computational efforts. Mark Rast University of Colorado, LASP clyne@ncar.ucar.edu

3. Dichotomy of simulation and analysis needs and resources in today’s HPC environments clyne@ncar.ucar.edu

4. A sampling of various technology performance curves • Not all technologies advance at same rate • Impact of parallelization not shown clyne@ncar.ucar.edu

5. Communication limits for volume rendering assuming theoretical peak performance • Table shows limits expressed as frames per second imposed by communication alone • Assumes only 8-bit data quantities clyne@ncar.ucar.edu

6. Visual data browsing Refine Coarsen Quantitative analysis Data manipulation Visualization and Analysis Platform for Ocean, atmosphere, and solar Research (VAPOR) Key components • Domain specific application focus: simulated earth sciences fluid flow • Coupled Visualization and quantitative data interrogation and manipulation capabilities • Multiresolution enabled terascale data exploration on the desktop Combination of visualization with multiresolution data representation that provide sufficient data reduction to enable interactive work on terascale data from a desktop clyne@ncar.ucar.edu

7. Fluid flow in the geosciences • E.g. Numerically simulated turbulence • Cartesian grids (usually) • 5123 to 10243 • Up to 40963 “hero” calculations • 5 to 8 variables • Temperature & Pressure • Velocity field components • Magnetic field components (MHD calculations) • Hundreds of time steps saved • Terabytes of data per experiment • Numerical “experiments” • Substantial analysis requirements Yannick Ponty, CNRS 2006 clyne@ncar.ucar.edu

8. Key Component (1) : Domain specific support Only limited support for: • Grid & data types • Cartesian grids, stretched and uniform sampling • AMR grids • Scalar and vector quantities • Visualization algorithms • Volume rendering, flow visualization, cutting planes/probe • Misc. • Publication quality graphics • Filters • File formats (one!) • Extensive support for: • Time varying data • Uniform as well as non-uniform sampling • Missing time steps • Quantitative investigation • Mathematical operators and data manipulators • Science driven specialized features Keep it simple! Keep it focused! Make it scientist friendly! clyne@ncar.ucar.edu

9. Interactive exploration of time varying data • Reduce bandwidth requirements • Regions of interest • Multiresolution • Caching clyne@ncar.ucar.edu

10. VAPOR Interactive visual browsing Future??? IDL Data manipulation & analysis VAPOR Data Collection Multi-resolution access and rapid sub-region extraction Disk Array Key Component (2) : Coupled visualization, quantitative analysis and manipulation capabilities • IDL - array based 4GL for scientific data processing • Thousands of mathematical functions • Basic 2D plotting • Array manipulation clyne@ncar.ucar.edu

11. Wavelet transformed data Two parameter linear function decomposition Hierarchical data representation Invertible and lossless Numerically efficient (O(n)) forward and inverse transform No additional storage cost Enable speed/quality tradeoffs Key component (3) : Multiresolution data access 504x504x2048 Full 252x252x1024 1/8 126x126x512 1/64 63x63x256 1/512 clyne@ncar.ucar.edu

12. Visual comparison of a 5123 compressible convection simulation M. Rast, 2002 1283 coarsened 5123 native clyne@ncar.ucar.edu

13. Performance of forward and inverse Haar wavelet transform • Data • Scalar • Single precision • System • Linux RHEL 3.0 • 2 x Intel 3.4 GHz Xeon EMT64 • 8 GBs RAM • 1Gb/sec Fibre Channel storage Gains in microprocessor technology enable transforms at very low cost clyne@ncar.ucar.edu

14. VAPOR Demo clyne@ncar.ucar.edu

15. Summary • VAPOR is a domain-specific platform for analysis, not a general purpose visualization tool • Target users: fluid flow researchers in earth sciences • Limited value for medical, oil & gas, aerospace, etc. • Desktop data exploration of terabyte data possible • Visualization enables rapid ROI identification • Multiresolution enables speed/quality tradeoffs clyne@ncar.ucar.edu

16. Steering Committee Nic Brummell - CU Yuhong Fan - NCAR, HAO Aimé Fournier – NCAR, IMAGe Pablo Mininni, NCAR, IMAGe Aake Nordlund, University of Copenhagen Helene Politano - Observatoire de la Cote d'Azur Yannick Ponty - Observatoire de la Cote d'Azur Annick Pouquet - NCAR, ESSL Mark Rast - CU Duane Rosenberg - NCAR, IMAGe Matthias Rempel - NCAR, HAO Geoff Vasil, CU Developers Alan Norton – NCAR, SCD John Clyne – NCAR, SCD Kenny Gruchalla - CU Research Collaborators Kwan-Liu Ma, U.C. Davis Hiroshi Akiba, U.C. Davis Han-Wei Shen, Ohio State Liya Li, Ohio State Systems Support Joey Mendoza, NCAR, SCD Acknowledgements clyne@ncar.ucar.edu

17. Questions??? www.vapor.ucar.edu clyne@ncar.ucar.edu

18. Inverse Haar transform with 1/8th volume subregion extraction • System • Linux RHEL 3.0 • 2 x Intel 3.4 GHz Xeon EM64 • 8 GBs RAM • 1Gb/sec Fibre Channel storage • Data • Scalar • Single precision Data blocking permits rapid subregion extraction clyne@ncar.ucar.edu

19. The Lifting Method of wavelet construction in the spatial domain[Sweldens, 95] A signal λjconsisting of 2j samples Split signal into even (λ) and odd (γ) coefficients. λ will contain low frequency information, γ will contain high frequency information. 1) Split: 2) Predict: Local correlation permits prediction of odd samples by even using a prediction operator, P. Capture difference between prediction and actual coefficient value. 3) Update: Update λ coefficients to preserve a property (e.g. mean) of original signal. Update Split Predict Transform 2 Transform j Transform 1 clyne@ncar.ucar.edu

20. Haar operators 1 7 3 1 6 0 9 5 γ2,k 6 -2 -6 -4 4 2 3 7 -2 4 3 5 2 4 Example: Lifting Method with the Haar Wavelet λ3,k λ2,k γ1,k λ1,k λ0,k γ0,k clyne@ncar.ucar.edu

21. NCAR Historical Estimated Sustained GFLOPS (Batch Production Systems) clyne@ncar.ucar.edu

22. NCAR Historical Estimated Sustained GFLOPS (Interactive Production Systems) • Current NCAR visualization and analysis resources • ~32 processors • 8 nodes (6 with gfx) • ~100 TB on-line storage • ~800 MBs/sec aggregate storage bandwidth • ~100 users (99 of which will not leave office) clyne@ncar.ucar.edu