1 / 66

Dr. Frederica Darema Senior Science and Technology Advisor

Dynamic Data Driven Application Systems (DDDAS) A new paradigm for applications/simulations and measurement methodology ( Symbiotic Measurement&Simulation Systems ). Dr. Frederica Darema Senior Science and Technology Advisor Director, Next Generation Software Program

garson
Download Presentation

Dr. Frederica Darema Senior Science and Technology Advisor

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DynamicData Driven Application Systems (DDDAS) A new paradigm for applications/simulations and measurement methodology (Symbiotic Measurement&Simulation Systems) Dr. Frederica Darema Senior Science and Technology Advisor Director, Next Generation Software Program Director, Biological Information technology Systems NSF

  2. What is DDDAS OLD (serialized and static) NEW PARADIGM (Dynamic Data-Driven Simulation Systems) Simulations (Math.Modeling Phenomenology Observation Modeling Design) Theory (First Principles) Simulations (Math.Modeling Phenomenology) Theory (First Principles) Experiment Measurements Field-Data User Experiment Measurements Field-Data User Dynamic Feedback & Control Loop Challenges: Application Simulations Development Algorithms Computing Systems Support

  3. Examplesof Applications benefiting from the new paradigm • Engineering (Design and Control) • aircraft design, oil exploration, semiconductor mfg, structural eng • computing systems hardware and software design (performance engineering) • Crisis Management • transportation systems (planning, accident response) • weather, hurricanes/tornadoes, floods, fire propagation • Medical • customized surgery, radiation treatment, etc • BioMechanics /BioEngineering • Manufacturing/Business/Finance • Supply Chain (Production Planning and Control) • Financial Trading (Stock Mkt, Portfolio Analysis) DDDAS has the potential to revolutionize science, engineering, & management systems

  4. Outline * Background and New Directions • Examples of Dynamic Data-Driven Application Systems (DDDAS) • CurrentTechnology Trends • Applications, Platforms ( Grids) • Why now is the time for DDDAS * Enabling DDDAS - Challenges and Approaches • Systems’ Software • Performance Engineering/ Systematic methods for building sw/hw systems • Dynamic application composition and Run-Time support • DDDAS application efforts • Algorithms for DDDAS * Agency Efforts • Existing programs • Future Initiatives * Technology transfer to industry

  5. NSF March 2000 Workshop on DDDAS(Co-Chairs: Craig Douglas, UKy; Abhi Desmukh, UMass)Invited Presentations • New Directions on Model-Based Data Assimilation (Chemical Appl’s) Greg McRae, Professor, MIT • Coupled atmosphere-wildfire modeling Janice Coen, Scientist, NCAR • Data/Analysis Challenges in the Electronic Commerce Environment Howard Frank, Dean, Business School, UMD • Steered computing - A powerful new tool for molecular biology Klaus Schulten, Professor, UIUC, Beckman Institute • Interactive Control of Large-Scale Simulations Dick Ewing, Professor, Texas A&M University • Interactive Simulation and Visualization in Medicine: Applications to Cardiology, Neuroscience and Medical Imaging Chris Johnson, Professor, University of Utah • Injecting Simulations into Real Life Anita Jones, Professor, UVA Workshop Report: www.cise.nsf.gov/eia/dddas

  6. Fire Model • Sensible and latent heat fluxes from ground and canopy fire -> heat fluxes in the atmospheric model. • Fire’s heat fluxes are absorbed by air over a specified extinction depth. • 56% fuel mass -> H20 vapor • 3% of sensible heat used to dry ground fuel. • Ground heat flux used to dry and ignite the canopy. Kirk Complex Fire. U.S.F.S. photo Slide Courtesy of Cohen/NCAR

  7. Coupled atmospheric and wildfire models Slide Courtesy of Cohen/NCAR

  8. Gas Phase Reactions SiCl3H  HCl + SiCl2 SiCl2H2 SiCl2 + H2 SiCl2H2 HSiCl + HCl H2ClSiSiCl3 SiCl4 + SiH2 H2ClSiSiCl3 SiCl3H + HSiCl H2ClSiSiCl3 SiCl2H2 + SiCl2 Si2Cl5H  SiCl4 + HSiCl Si2Cl5H  SiCl3H + SiCl2 Si2Cl6 SiCl4 + SiCl2 Surface Reactions SiCl3H + 4s  Si(B) + sH + 3sCl SiCl2H2 + 4s  Si(B) + 2sH + 2sCl SiCl4 + 4s  Si(B) + 4sCl HSiCl + 2s  Si(B) + sH + sCl SiCl2 + 2s  Si(B) + 2sCl 2sCl + Si(B)  SiCl2 + 2s H2 + 2s  2sH 2sH  2s + H2 HCl + 2s  sH + sCl sH + sCl  2s + HCl AMAT Centura Chemical Vapor Deposition Reactor Operating Conditions Reactor Pressure 1 atm Inlet Gas Temperature 698 K Surface Temperature 1173 K Inlet Gas-Phase Velocity 46.6 cm/sec Slide Courtesy of McRae/MIT

  9. Some Technology Challenges in Enabling DDDAS • Application development • interfaces of applications with measurement systems • dynamically select appropriate application components • ability to switch to different algorithms/components depending on streamed data • Algorithms • tolerant to perturbations of dynamic input data • handling data uncertainties • Systems supporting such dynamic environments • dynamic execution support on heterogeneous environments • GRID Computing, andBeyond!!!

  10. Why Now is the Time for DDDAS • Technological progress prompted advances in some of the challenges • Computing speeds advances (uniprocessor and multiprocessor systems), Grid Computing • Systems Software • Applications Advances (parallel & grid computing) • Algorithms advances (parallel &grid computing, numeric and non-numeric techniques) • Examples of efforts in: • Systems Software • Applications • Algorithms

  11. What is Grid Computing? coordinated problem solving on dynamic and heterogeneous resource assemblies DATA ACQUISITION ADVANCEDVISUALIZATION ,ANALYSIS COMPUTATIONALRESOURCES IMAGING INSTRUMENTS LARGE-SCALE DATABASES Example:“Telescience Grid”, Courtesy of Ellisman & Berman /UCSD&NPACI

  12. Application Directions Past • Monolithic • One programming language • Computation Intensive • Batch • Hours/Days • Computation Intensive • Data Intensive • Few Minutes/hours • Real Time Turn-around • Visualization (real time) • Interactive Steering by user • and ... DDDAS Present / Future • Multi-Modular • Multi-Source Data • Multi-Language • Multiple Developers Such characteristics require new capabilities in systems software

  13. Some Examples of Today’s Applications

  14. The e-Business / (CIM, CIE) Distributor Channel Order Processing Customer Service Sales Management Manufacturing Product DBs Inventory Shipping Application Integration Interoperability Process Coordination Management & Monitoring Business to Business Enterprise Messaging Data Integration Interoperability Mobile Workers Knowledge Workers Business Communications Business to Customer Web e-commerce

  15. Compare withClassical (Old) Supply Chain Manufacturing Manufacturing Manufacturing Distribution Distribution Distribution Retail Retail Retail Customer Customer Customer Customer Customer Customer Parts Supplier Parts Supplier Transportation Supplier

  16. TREES H2O GRASS ROAD Target & Clutter Database ROI Hypothesis TREES TREES y GRASS  BMP-2 Local Scene Map x ROI Hypothesis TREES Shadow (?) TREES y GRASS  BMP-2 Local Scene Map x MSTAR (DARPA)(Moving and Stationary Target Acquisition and Recognition) Focus of Attention Index Database (created off-line) ... Search Tree Regions of Interest (ROI) Segmented Terrain Map SAR Image & Collateral Data - DTED, DFAD - Site Models - EOSAT imagery ... Indexing Target & Scene Model Database (created off line) Task Predict Task Extract Statistical Model Search Extract Predict Clutter Database CAD Match Results Tree Clutter Semantic Tree Form Associations Analyze Mismatch Refine Pose & Score Shadow Obscuration ? x2,y2,  x1,y1,  Score = 0.75 Ground Clutter Feature-to-Model Traceback Match

  17. Platform Directions tac-com fire cntl alg accelerator data base data base fire cntl SAR Past • Vector Processors, SIMD MPPs • Distributed Memory MPs • Shared Memory MPs • Distributed Platforms, Heterogeneous Computers and Networks • Heterogeneity • architecture (compute &network) • node power (supernodes, PCs) Present/Future • Latencies • variable (internode, intranode) • Bandwidths • different for different links • different based on traffic GiBs Grids Petaflops Platform (Grid-in-a-Box) Distributed Platform …. MPP NOW SP

  18. OC-12 vBNS Abilene MREN OC-12 OC-3 TeraGrid: 13.6 TF, 6.8 TB memory, 79 TB internal disk, 576 network disk ANL 1 TF .25 TB Memory 25 TB disk Extreme Blk Diamond Caltech 0.5 TF .4 TB Memory 86 TB disk 574p IA-32 Chiba City 256p HP X-Class 32 32 24 32 32 128p HP V2500 128p Origin 24 32 24 92p IA-32 32 HR Display & VR Facilities 5 4 8 5 8 HPSS HPSS NTON OC-48 OC-12 Calren ESnet HSCC MREN/Abilene Starlight Chicago & LA DTF Core Switch/Routers Cisco 65xx Catalyst Switch (256 Gb/s Crossbar) Juniper M160 OC-12 ATM OC-48 OC-12 GbE NCSA 6+2 TF 4 TB Memory 240 TB disk SDSC 4.1 TF 2 TB Memory 225 TB SAN vBNS Abilene Calren ESnet OC-12 OC-12 OC-12 OC-3 Myrinet 4 8 HPSS 300 TB UniTree 2 Myrinet 4 10 1024p IA-32 320p IA-64 1176p IBM SP 1.7 TFLOPs Blue Horizon 14 Sun Server 15xxp Origin 4 16 2 x Sun E10K Slide Courtesy of Berman/NPACI

  19. Grids Form the Basis of a National Information Infrastructure August 9, 2001: NSF Awarded $53,000,000 to SDSC/NPACI and NCSA/Alliance for TeraGrid TeraGrid will provide in aggregate • 13.6 trillion calculations per second • Over 600 trillion bytes of immediately accessible data • 40 gigabit per second network speed • Provide a new paradigm for data-oriented computing • Critical for disaster response, genomics, environmental modeling, etc. Slide Courtesy of Berman/NPACI

  20. DARPA SF Express(Synthetic Forces Express) LargeScale distributed, interactive, battle simulation Simulation decomposed terrain contiguously among supercomputers Simulation of 50,000 entities in 8/97, 100,000 entries in 3/98 NSF and DoE CACTUS/GriPhyN (ITR, NGS, SciDAC) Toolkit for Large-Scale Relativity Simulations Largest Simulations for Colliding Black Holes International Team/Grid Examples Other Agencies Grid Efforts NASA’s Information Power Grid

  21. Why Now is the Time for DDDAS ? • Technological progress prompted advances in some of the challenges • Computing speeds advances (uniprocessor and multiprocessor systems), Grid Computing • Applications Advances (parallel & grid computing) • Algorithms advances (parallel &grid computing, numeric and non-numeric techniques) • Examples of efforts in: • Systems Software • Applications • Algorithms

  22. Agency Efforts • NSF • NGS: The Next Generation Software Program • develops systems software supporting dynamic resource execution • ITR: Information Technology Research (NSF-wide) • has been used as an opportunity to support DDDAS related efforts • 46 DDDAS pre-proposals; many meritorious • about 24 proposals; 8 were awarded … more on this, next slide…. • Gearing towards a DDDAS initiative • expect participation from all NSF Directorates: CISE, MPS, ENG, BIO, GEO, SBE, HER • DARPA, NASA, DoE • have related programs (NASA/IPG, DoE/SciDAC) • and interested in DDDAS

  23. “DDDAS” proposals awarded in FY01 ITR Competition • Biegler – Real-Time Optimization for Data Assimilation and Control of Large Scale Dynamic Simulations • Car – Novel Scalable Simulation Techniques for Chemistry, Materials Science and Biology • Knight – Data Driven design Optimization in Engineering Using Concurrent Integrated Experiment and Simulation • Lonsdale – The Low Frequency Array (LOFAR) – A Digital Radio Telescope • McLaughlin – An Ensemble Approach for Data Assimilation in the Earth Sciences • Patrikalakis – Poseidon – Rapid Real-Time Interdisciplinary Ocean Forecasting: Adaptive Sampling and Adaptive Modeling in a Distributed Environment • Pierrehumbert- Flexible Environments for Grand-Challenge Climate Simulation • Wheeler- Data Intense Challenge: The Instrumented Oil Field of the Future

  24. Enabling DDDAS Dynamic Data-Driven Application Systems -- Symbiotic Measurement&Simulation Systems New Systems Software Technology NGS Program Dynamic Compilers & Application Composition Performance Engineering

  25. Enabling DDDAS Dynamic Data-Driven Application Systems -- Symbiotic Measurement&Simulation Systems Dynamic Compilers & Application Composition Performance Engineering

  26. Application Languages API Compilers & Libraries Runtime Services Tools Global Management Computing Engine The NGS Program develops Performance Engineering Technology Performance Models & Measurements Distributed Applications Collaboration Visualization Environments Scalable I/O Authenication / Data Management Authorization Archiving/Retrieval Dependability Performance Engineered Design Technology Services Services . . . Other Services Distributed Systems Management Distributed, Heterogeneous, Dynamic, Adaptive Computing Platforms and Networks Components Technology CPU Device Memory . . . Technology Technology Technology

  27. Application Models Languages Compilers Libraries Tools Enables Analysis in Multiple views of the system (The applications’ view) Distributed Applications . . . Collaboration Visualization Environments Scalable I/O Authenication / Data Management Authorization IO / File Archiving/Retrieval Dependability Models Services Services . . . OS Other Services Scheduler Models Distributed Systems Management Architecture / Distributed, Heterogeneous, Dynamic, Adaptive Network Computing Platforms and Networks Models Memory CPU Device Memory . . . Models Technology Technology Technology

  28. Enabling DDDAS Dynamic Data-Driven Application Systems -- Symbiotic Measurement&Simulation Systems Dynamic Compilers & Application Composition Performance Engineering

  29. Technology Gap Example case: Distributed Application PlatformProgramming ModelConstraint • Message passing • Inefficient • load-balancing • Static partition • Shared queue • Application • “re-write” • required • Dynamic allocation • of work • Message-passing • across SMPs • Shared queue • within SMP Distributed Platform tac-com fire cntl alg accelerator …. data base data base fire cntl SAR MPP NOW SP Dynamic Analysis Situation • Network of • Workstations • (NOW) • Symmetric • Multiprocessor • (SMP) Launch Application(s) • Application cannot • be repartitioned dynamically • when problem size or • number of SMPs changes • Cluster of • SMPs Distributed Computing Resources Adaptable Systems Infrastructure

  30. The NGS Program developsTechnology for integrated feedback & control Runtime Compiling System (RCS) and Dynamic Application Composition tac-com fire cntl alg accelerator …. data base data base fire cntl SAR MPP NOW SP Application Model Dynamic Analysis Situation Distributed Programming Model Application Program Compiler Front-End Application Intermediate Representation Compiler Back-End Launch Application (s) Performance Measuremetns & Models Dynamically Link & Execute Application Components & Frameworks Distributed Computing Resources Distributed Platform Adaptable computing Systems Infrastructure

  31. Performance feedback Perf problem Software components Realtime perf monitor Scheduler/ Service Negotiator Grid runtime System (Globus) Config. object program Source appli- cation whole program compiler P S E negotiation Dynamic optimizer libraries Example of NGS supported effort: • The GrADS Project (Grid Application Development Software) • Design and development of a Grid program development and execution environment • Tight coupling between program preparation and program execution environment • Contract-based performance economy Slide Courtesy of GRADS group

  32. NGS fosters Employing Performance Engineering Technology for: Application Composition and Run-Time Support on Dynamic, Heterogeneous Computing Platforms so that the users “Shouldn’t Have to be Heroes to Achieve Grid Program Performance” and... because heroism is not enough

  33. Enabling DDDAS Dynamic Data-Driven Application Systems -- Symbiotic Measurement&Simulation Systems Dynamic Compilers & Application Composition Performance Engineering

  34. Challenges • Application development • develop interfaces of applications with measurement systems • dynamically select appropriate application components • need to switch to different algorithms/components depending on streamed data • Algorithms • tolerant to perturbations of dynamic input data • handling data uncertainties • Systems supporting such dynamic environments • need Performance Engineering technology • ApplicationComposition Frameworks • Dynamic Run-Time Support

  35. Some more Challenges on Applications Development Issues • Handling Data Streams in addition to Data Sets • Handling different data structures – semantic information • Interfaces to Measurement Systems - Interactive Visualization and Steering • Standards for data exchange • Combining Local and Global Knowledge • Model Interactions • Application control of measurement systems • Dynamic Application Composition and Runtime support Examples from ITR supported efforts:

  36. NSF ITR Project A Data Intense Challenge: The Instrumented Oilfield of the Future PI: Prof. Mary Wheeler, UT Austin Multi-Institutional/Multi-Researcher Collaboration Slide Courtesy of Wheeler/UTAustin

  37. Highlights of Instrumented Oilfield Project • Motivation: • Field instrumentation for information technology and computational science essential for monitoring and optimizing oil and gas production. • Integration yields: THE INSTRUMENTED OILFIELD • Field Technologies: • Time-lapse surface and borehole seismic, permanent downhole sensors, intelligent well completions, fiber optics, and remote control operations Slides Courtesy of Wheeler/UTAustin

  38. Highlights of Instrumented Oilfield Proposal • IT Technologies: • Data management, data visualization, parallel computing, and decision-making tools such as new wave propagation and multiphase, multi- component flow and transport computational portals, reservoir production: THE INSTRUMENTED OILFIELD • Major Outcome of Research: • Computing portals which will enable reservoir simulationand geophysical calculations to interact dynamically with the data and with each other and which will provide a variety of visual and quantitative tools. Test data provided by oil and service companies

  39. Economic Modeling and Well Management Production Forecasting Well Management Reservoir Performance Simulation Models Visualization Data Analysis Multiple Realizations Field Measurements Data Management and Manipulation Reservoir Monitoring Field Implementation Data Collections from Simulations and Field Measurements

  40. Highlights of Instrumented Oilfield Proposal Simulation Framework

  41. ITR Project • A Data Intense Challenge: • The Instrumented Oilfield of the Future • Industrial Support (Data): • British Petroleum (BP) • Chevron • International Business Machines (IBM) • Landmark • Shell • Schlumberger

  42. Rapid Real-Time Interdisciplinary Ocean Forecasting: Adaptive Sampling and Adaptive Modeling in a Distributed Environment Nicholas M. Patrikalakis, Henrik Schmidt, MIT Allan R. Robinson, James J. McCarthy, Harvard http://czms.mit.edu/poseidon Poseidon

  43. Ocean Science Issues • Data driven simulations via data assimilation • Simulation driven adaptive sampling of the ocean • Interdisciplinary ocean science: interactions of physical, biological, acoustical phenomena • Extend state-of-the-art via feedback from acoustics to physical&biological oceanography • Application in fisheries management, but also in oil-slick containment

  44. Interdisciplinary Ocean Science

  45. System Architecture

  46. Data Driven Design Optimization in EngineeringUsing Concurrent Integrated Experiment and SimulationDoyle Knight, Rutgers-The State University of New JerseyKhaled Rasheed, University of Georgia • Channel or enclosure with isolated heat sources (i.e., electronic components) • The maximum surface temperature must be maintained below a specified level by the flow of air or dielectric liquid • Control flow of air or liquid for optimal heat dissipation

  47. Gas Phase Reactions SiCl3H  HCl + SiCl2 SiCl2H2 SiCl2 + H2 SiCl2H2 HSiCl + HCl H2ClSiSiCl3 SiCl4 + SiH2 H2ClSiSiCl3 SiCl3H + HSiCl H2ClSiSiCl3 SiCl2H2 + SiCl2 Si2Cl5H  SiCl4 + HSiCl Si2Cl5H  SiCl3H + SiCl2 Si2Cl6 SiCl4 + SiCl2 Surface Reactions SiCl3H + 4s  Si(B) + sH + 3sCl SiCl2H2 + 4s  Si(B) + 2sH + 2sCl SiCl4 + 4s  Si(B) + 4sCl HSiCl + 2s  Si(B) + sH + sCl SiCl2 + 2s  Si(B) + 2sCl 2sCl + Si(B)  SiCl2 + 2s H2 + 2s  2sH 2sH  2s + H2 HCl + 2s  sH + sCl sH + sCl  2s + HCl AMAT Centura Chemical Vapor Deposition Reactor Operating Conditions Reactor Pressure 1 atm Inlet Gas Temperature 698 K Surface Temperature 1173 K Inlet Gas-Phase Velocity 46.6 cm/sec Slide Courtesy of McRae/MIT

More Related