1 / 24

Your Results From Disk to Device (Preview)

Your Results From Disk to Device (Preview). Amit Chourasia Mona Wong- Barnum Michael L. Norman San Diego Supercomputer Center, UCSD Presented at: XSEDE 13, San Diego, CA, Jul 23, 2013. S tream E ncode E xplore D isseminate M y E xperiments. Computation Cycle. Abstract ?.

vega
Download Presentation

Your Results From Disk to Device (Preview)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Your Results From Disk to Device(Preview) Amit Chourasia Mona Wong-Barnum Michael L. Norman San Diego Supercomputer Center, UCSD Presented at: XSEDE 13, San Diego, CA, Jul 23, 2013

  2. Stream Encode Explore Disseminate My Experiments

  3. Computation Cycle Abstract ? Monitor? Validate?

  4. Monitoring • DerivedProducts • Abstraction • Validation

  5. Direct Access • Not All Members Have HPC Access • Direct Sharing • Security Implications On HPC • DerivedProducts

  6. Concise derived products • Plots & Text Documents • Image Sequences & Videos Share with the research group • Download derived products • Share with collaborators • Email snippets • Set up a webpage • Add content • Email

  7. Derived Visual Content Survey Derived content for a single research group in respective domain

  8. Pitfalls in sharing derived content • Download Upload Round Trip • Video Encoding Complexity • Web Setup • Process Duplication • Scalability

  9. Direct Transfer • Easy Sharing • Reusable • Ubiquitous Gaps SeedMe.org • Download Upload • Video Encoding • Web Setup • Process Duplication • Scalability • Direct Access • Not All Members Have HPC Access • Direct Sharing • Security Implications On HPC

  10. Why not use existing tools?

  11. SeedMe Architecture Ingestion (Atomic or Recurring) • Instant Processing • (Text, Images) • Notification • Responsive Content Access • Batch Processing • (Videos)

  12. SeedMe Hardware Ingestion (Atomic or Recurring) • Instant Processing • (Text, Images) • Notification • Responsive Content Access • Batch Processing • (Videos) 20 gbe Web Cluster (2-8 nodes) : Drupal 20 gbe 20 gbe Phones Ingest Temp Storage (10TB) Mirrored Storage (50 TB) : ZFS Tablets Encode Cluster (16 nodes) : Apple Compressor + ffmpeg Desktops

  13. Compressor Encoding Benchmark

  14. SeedMe Collection Ticker* • Plots • Image Sequences* • Videos Access: Public, Group, Private Collaborators Meta Data * Recurring: periodic update

  15. SeedMe: Interaction Process Create New Collection Set Access Control (Public, Group, Private) Specify collaborators* and meta data Upload/Update content http/https, ftp, Globus Online Interactively on Web, Command Line or REST Services View Collection on Web

  16. SeedMe: Interaction Modality • Command Line Interface (using curl) • Web Interface (WYSIWYG) • RESTful API (integrate your tool chain) • JSON

  17. { "email":”jdoe@sdsc.edu", "api_key":"XXXXXXXXXXX", ”view_permission":"public", ”viewer_emails":["a1@a.com","b1@b.com"], "title":"simulation XYZ", "description":"input conditions", "credits":"Visservices group, SDSC", "key_values":{“key1”: "v1”,"key2":"v2”}, "expire_date":"2013-05-01", "tags":["x","y","z"], "ticker":["time step 100","time step 200"], "files":{ ”docs":["a.pdf",”b.txt"], "plots":["a.png","b.jpg","c.tiff"], "image_sequences":{ "ash":["ash1.jpg","ash2.jpg","ash3.jpg"], "air":["air1.jpg","air2.jpg","air3.jpg"] }, "movies":["m1.mov","m2.mov »] } } Authentication Permissions Meta Data Content Text Ticker Doc Files Plot Files Image Sequences Videos

  18. Application Scenarios • Monitor progress with text tickers and images • Encode and stream videos from image sequences • Share & collect visualization results • Use the collections for education and outreach Content accessible on most devices

  19. Work in progress How it Works Examples #1 & #2 Periodic update at simulation run time from a Magneto Hydrodynamics Simulation Courtesy of Mahidhar Tatineni and Homa Karimabadi, UCSD

  20. Future Work • Production SeedMe Service • Example Scripts& Documentation • Globus integration with SeedMe

  21. Acknowledgements Andrew Ferbert, Michael Dwyer, Kristen Levy & Doug Weimer, SDSC :Hardware/software setup Mahidhar Tatineni, SDSC & Homa Karimabadi, UCSD: in-situ tests Apple Inc. : Provided test hardware/software on loan during project conceptualization phase National Science Foundation This material is based upon work supported by the National Science Foundation under Grant No. OCI-1235505

  22. Sign up Invitation For notification and early access (we won’t spam) Contact: amit @ sdsc.edu

  23. Persistence?

More Related