1 / 12

Out-of core Streamline Generation Using Flow-guided File Layout

Out-of core Streamline Generation Using Flow-guided File Layout. Chun-Ming Chen 788 Project. Background. Visualize flow fields with streamlines Scientific data is huge Traditional: Compute in clusters Drawbacks: High Equipment Cost Inter-node communication.

holleb
Download Presentation

Out-of core Streamline Generation Using Flow-guided File Layout

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Out-of core Streamline Generation Using Flow-guided File Layout Chun-Ming Chen 788 Project

  2. Background • Visualize flow fields with streamlines • Scientific data is huge • Traditional: Compute in clusters • Drawbacks: • High Equipment Cost • Inter-node communication http://vidi.cs.ucdavis.edu/publications http://nuweb2.neu.edu/

  3. Background • Nowadays: multi-core CPU on single machine • May not have enough memory capacity • Out-of-core computation is needed • Out-of-core: data cannot be fully loaded into main memory

  4. Goal Compute streamlines on a lower-cost multi-core machine with limited memory, given arbitrary seeds

  5. Demand Paging Algorithm • Preparation Stage: • Break flow fields into blocks • Streamline Generation Stage: • Only load needed blocks during computation • Release least recently used (LRU) block when memory full Load data from Disk Store data in memory pool Compute Release data (LRU)

  6. Multi-core streamline computation Job Queue Seeds for block1 Seeds for block 2 Threaded Computation Threaded Computation Seeds for block 3 Seeds for block 4 New seeds generated from block 1

  7. Problem of Out-of-core Computation • Earlier tests: 1Gb Data • Environment: • 8-core Intel Machine • Limit 25Mb memory usage • Time Generating streamlines: 8.436 s • Time Loading flow field : 32.524 s • IO is the bottle neck

  8. More tests • Read all blocks in a 6Gb data • Unit block size: float 16x16x16 (49152 bytes)Total 131,072 blocks • Random access: 249.146794 sec • Sequential read:13.594309 sec • Reverse-Sequential read: 264.262752 sec • Sequential read can be 20 times faster • Reason: Disk Prefetching

  9. File Layout • Re-arrange data to increase more sequential reads • Hilbert Curve Layout:

  10. Result of Scheduling for Hilbert Curve Layout • Scheduler: only read forward • Test: 1Gb Data • Environment: • 8-core Intel Machine • Limit 25Mb memory usage • Old test: • Time Generating streamlines: 8.436 s • Time Loading flow field: 32.524 s • Hilbert layout: • Time Generating streamlines: 7.890 s • Time Loading flow field : 0.610 s

  11. Layout By Flow Direction

  12. Next And Conclusion • Next: • Better layout? • Re-arrange data based on flow direction • NP-hard Problem • Conclusion: • If we want to analyze large scientific data in a single machine, out-of-core computation is required now and also in the future • Good File layout is important for out-of-core computation

More Related