1 / 15

Fast Adaptive Storage and Retrieval

Fast Adaptive Storage and Retrieval. Scott B. Baden Department of Computer Science and Engineering University of California, San Diego. Motivation. Some applications are able to distinguish interesting features from background data using on-line analysis. Features. Animation.

jera
Download Presentation

Fast Adaptive Storage and Retrieval

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Fast Adaptive Storage and Retrieval Scott B. Baden Department of Computer Science and Engineering University of California, San Diego

  2. Motivation Some applications are able to distinguish interesting features from background data using on-line analysis

  3. Features

  4. Animation

  5. Fast Adaptive Storage and Retrieval • If the volume fraction of interesting data is small, then we can reduce storage, memory, and network bandwidth requirements significantly by storing only what is “needed” • We call a scheme that realizes this capabilityAdaptive Storage and Retrieval (FASTR) • This is a new paradigm for scientific users, since they are reluctant to part with their data • We use resources only to the extent that we require them: remote knowledge discovery and data browsing

  6. The KeLP Project • C++ run time libraries for parallel application & library development • Hide low level details without sacrificing performance • Irregular block structured data • Express communication at a high level using intuitive geometric set operations • Also applies to data intensive applications • KeLP I/O: out of core (Bradley Broom, Rice)

  7. Data intensive application of KeLP • KDistuf • Turbulent flow with Direct Numerical Simulation • Collaboration involving K. Nomura (UCSD MAE), W. Kerney and D. Shalit (UCSD CSE),G. Balls (UCSDSC), P. Diamessis (USC) • Content-based data compression • Borrow structured adaptive mesh refinement grid techniques to… • Capture features at full resolution • Discard remaining background data

  8. More about the application • Turbulent mixing in stably stratified flow under the influence of background shear • Solve the incompressible Navier Stokes equations • Follow the time evolution of regions of overturned dense fluid, which are the main agents of stirring and mixing “The efficiency of mixing in turbulent patches: inferences from direct simulations and microstructure observations,” in press, J. Phys. Ocean. Smyth, Moum, and Caldwell, 2001.

  9. Information discovery • Oceanographic observations are incomplete: restricted to 1 dimensional observations • Discovery: time evolution, energy dissipation and lifetime of overturn regions, which have irregular shapes Bill Smyth, Dept. Oceanic & Atmospheric Sciences,Oregon State University

  10. Fast Adaptive Storage and Retrieval • Compression depends on the data, currently on 1283 • Best case ~ 20:1 compression (10 GB  500 MB), worst case ~ 2.8:1 • Lempel-Ziv (gzip) give us only 10%

  11. Further savings: another application • Use volume tracking [Silver, Rutgers] to follow individual features • FASTR permits us to extract only the data we need out of the many features present • Computational volume: 2M pts • Average feature size: 1K points • Maximum feature size: 20K pts • Saves additional two orders of magnitude in communication bandwidth requirements • Perform local analysis on a workstation

  12. Future plans • Develop remote analysis capability • Integrate with DTF data handling infrastructure • Larger scale simulations on Blue Horizon and on clusters: 2563 • Study vortex pairs in a stratified turbulent environment • Improved understanding of aircraft wake vortices • Practical importance for air traffic control

  13. Remote analysis capability • Perform analysis on data sets stored remotely, e.g. Data Cutter • We can perform some data analysis on a local workstation • For highly intensive data analysis, we can use higher end resources, but again we access only the data we need

  14. Publications and people • FASTR is based on a research prototype called MOLD, which is the M.S. thesis research of UCSD CSE student William Kerney “MOLD: A System for Breaking Down Large Visualization and Post-Processing Problems.” Expected March 2002. • Peter Diamessis, then a PhD student with Keiko Nomura (UCSD MAE Dept), used MOLD to carry out an exploration of overturns • An Investigation of Vortical Structures and Density Overturns in Stably Stratified Homogeneous Turbulence by Means of Direct Numerical Simulation, P. Diamessis, PhD thesis, 2001 • “Automated Tracking of Turbulent Structures in Direct Numerical Simulation,” P. Diamessis et al, PARA 2002, Helsinki, Finland. To appear.

  15. Software availability • FASTR- contact us • KeLP • Hardened version of KeLP, AKA KeLP1.4 • http://www.cse.ucsd.edu/groups/hpcl/scg/kelp • NPACI Blue Horizon, Sun HPC, Cray T3E, Linux clusters • Workstations: Solaris, Linux, etc. • Dual tier variant, KeLP2.1: hierarchical KeLP for SMP clusters and SMP based machines (e.g. BH)

More Related