1 / 9

Computing Infrastructure

Computing Infrastructure. R.Dubois richard@slac.stanford.edu. Potential Work Areas. Management of simulations Sources of cycles Manpower to run them workflow engine to minimize effort? Managing the data that results: catalogue and access Look at usability of tools

boyce
Download Presentation

Computing Infrastructure

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Computing Infrastructure R.Dubois richard@slac.stanford.edu Computing Parallel Session

  2. Potential Work Areas • Management of simulations • Sources of cycles • Manpower to run them • workflow engine to minimize effort? • Managing the data that results: catalogue and access • Look at usability of tools • Connection between cosmosims and ImSim • Harnesses, config tools for running ImSim • Long term support for ImSim? • Collaborative tools: foster communications and sharing • Forum, wiki Computing Parallel Session

  3. Managing Simulations • Clusters at collaboration sites • Grid cycles • John asking for 3 FTEs to run sims • Data catalogue to track simulated data • Probably no off-the-shelf tool currently to track data across arbitrary sites and provide access tools • Examples from Fermi: • Catalogue – being adapted to LSST camera test data as demo • Workflow engine – web based submission/tracking tool • Central submission to multiple sites with addition of simple interface to local batch system • Being used at SLAC for ImSim runs now as demo Computing Parallel Session

  4. Catalogue Example Computing Parallel Session

  5. Example Work Flow Engine Computing Parallel Session

  6. Running ImSim • User Friendliness: • 10 CPU-hrs per visit • Several steps • Understand DM tools: • Best path to catalogues Computing Parallel Session

  7. ImSim Usability Aids • Have a recipe for running – needs user intervention between steps • Can we streamline the recipe? • Might the code benefit from profiling? May be some low hanging fruit to make it go faster than 10 hrs per visit? • Might not be hard to keep a repository of compiled versions of ImSim for popular OSes • Fetch everything needed via an installer • If we hope for a broader developer community • Maybe look at packaging and internals to enable/encourage new developers to get involved Computing Parallel Session

  8. Collaborative Tools • Common repository for collaboration code • Can create a mix of “user contributions” which can evolve into a collection of integrated tools in the same repository • Ease of collaboration: • Use a forum for discussion • More powerful than mailing lists • Use good wiki tool for sharing and discussing documents, analyses etc Computing Parallel Session

  9. Some Open Questions • Can there be a common framework defined for analysis tools? At what level? FITS atomic model with files as interfaces? What does LSST project framework look like? Can/should it be emulated? • Should there be common tools defined for things like plotting etc? • Any connection to other surveys? How to share/store other surveys' data? • What about manpower for supporting all this? • Analysis tasks & systematics session • Verification & Validation of project pipelines • Plea for cross experiment computing resources • What is 'deliverable' for systematics studies? Could there be data flow diagrams to help identify any? • Non-ImSim sims?? • Need fast sim capability Computing Parallel Session

More Related