Army High Performance Computing Research Center

1. Army High Performance Computing Research Center Overview of Research Activities Vipin Kumar Director, AHPCRC

2. AHPCRC • High performance computing (HPC) plays a key role in bringing about a transformation from the heavily mechanized, slow to deploy forces, to an objective force. • Mission: • Leading edge HPC research in support of the Army’s science and technology goals for the Objective Force and Army Transformation. • Transfer of technology to the Army via direct interactions with Army scientists and through an Infrastructure Support program. • Education and outreach programs that include the Summer Institute for undergraduate students and a series of international/national workshops and conferences. • Forging of synergistic relationships among partner institutions via strong, collaborative research. Overview of AHPCRC Research Activities

3. AHPCRC Team Clark Atlanta University (CAU) Florida Agricultural and Mechanical University (FAMU) Howard University (HU) Jackson State University (JSU) University of Minnesota (UM) University of North Dakota (UND) NetworkCS (NCS) Overview of AHPCRC Research Activities

4. AHPCRC – Current Research

5. Research Program Overview of AHPCRC Research Activities

6. Objectives: Develop HPC modeling and analysis techniques for low observable technologies to obtain an improved understanding of the entire spectrum of signatures of combat systems in a realistic battlefield environment. Computational modeling of radio frequency (RF) and infrared (IR) signatures, and the effects of coatings on these signatures. Understanding the effect of atmospheric on gathering and detecting acoustic and near infrared (IR) signatures. Portfolio: ELECTROMAGNETIC SIGNATURE MODELING IN THE SYNTHETIC BATTLEFIELD Overview of AHPCRC Research Activities

7. Projects:  RCS Modeling of Multilayered Material Structures/Coatings. Solution of Electromagnetic Field Equations Using Basis Sets in Time and Space and Surface Interactions. Signature Modeling Applications of an Innovative Objective Analysis Technique. Fine Scale Modeling and Product Generation for Visualization of Atmospheric Dynamics and Phenomena. Algorithms For Signal and Image Processing, and ATR. Portfolio: ELECTROMAGNETIC SIGNATURE MODELING IN THE SYNTHETIC BATTLEFIELD Overview of AHPCRC Research Activities

8. Radar Cross Section (RCS) Modeling of Multilayered Material Structures/Coatings Objectives Determine the radar signatures of military hardware, and the use of various methods to control them, including the shaping of the geometry to reduce electromagnetic signatures, and the use of coatings or shields which are absorptive or which scatter incoming signals to reduce these returns. Methodology The program is focusing on the formulation of a boundary integral approach for the solution of 3-D electromagnetic scattering and on the development of a corresponding code. Overview of AHPCRC Research Activities

9. Extend current generalized response function framework to multiple dimensions Theoretical framework Testing and Validation Develop variationally-based filter Theoretical framework Evaluation Develop software that applies the new filter Initial (preliminary) incorporation of the new filter into Local Analysis and Prediction System (LAPS) Atmospheric Effects on Signature Modeling Overview of AHPCRC Research Activities

10. Objectives: Multi-disciplinary HPC modeling and simulations technology for projectile-armor/anti-armor target interaction (including blast-target interaction) in support of Future Combat Systems (FCS). Computational fluid dynamics and aerothermodynamics for external and internal flows in anti-tank projectiles and missiles. Structural dynamics, contact-impact and penetration. Energetic materials. Portfolio: PROJECTILE TARGET INTERACTION Overview of AHPCRC Research Activities

11. Projects: Computational Structural Mechanics and Dynamics. Discontinuous Galerkin Method In Linear/Nonlinear Structural Mechanics. Developments Of Stable, Locking Free Finite Elements for Nonlinear Structural Problems Without Hourglass Control. Characterization, Aging, and Dynamics of Energetic Materials. Energetic Nano-particle Architecture and Properties. Modeling and Simulation of Nano-particle Growth in Turbulent Flows. Mining Computational Chemistry Data Sets. Complex Configuration Aerodynamics with Time-Dependant Interactive Geometries. Computational Analysis of Transitional and Turbulent High-Speed Flows. Numerical Simulation of Turbulent Reacting Flows. Scalable Parallel Algorithms for Partitioning Multi-physics and Multi-phase Computations. Molecular Dynamic Simulation of Atomic-Scale Friction and Wear for Micro Electro Mechanical Systems (MEMS) Applications. Portfolio: PROJECTILE TARGET INTERACTION Overview of AHPCRC Research Activities

12. Monte-Carlo based simulation and classical molecular dynamics (MD) approaches to characterize the evolution of surface to volume ratio and properties of energetic nanoparticles. Model includes nanoparticle coagulation and finite coalescence, as well as energy release and non-isothermal effects resulting from coalescence. Surface energy and kinetic rates for caolescence determined from MD calculations. Objective Computational Approach Synthesis and Properties of NanoEnergetic Materials Develop a hierarchical computational approach to characterize the thermophysical properties of nanoparticles and their manufacture in areas of interest to the Army, including: -- Predicting the properties of nanoparticles for application in energetic materials -- Predicting the morphology and architecture of nanoparticles grown from vapor -- Augmenting the experimental program of the Army funded Center for Nanoenergetics Research (CNER) Current Modeling Current Modeling • 20-40 Atoms (quantum mechanics) • 104-105 Atoms (molecular dynamics) • Simple idealized interactions • Limited Spatial and Time scales • Perfect Crystals, 2-Dimensional • Build a model capable of modeling the morphological evolution of aluminum nanoparticles using bulk property data. • Develop a simulation capability for Aluminum nanoparticles using an in-house code or with “Temperature Accelerated Dynamics.” • Conduct detailed comparisons to experimental results whenever feasible. Overview of AHPCRC Research Activities

13. Modeling and Simulation of Nanoparticle Growth inTurbulent Flows • Goal: Develop an analytical and algorithmic framework to facilitate the prediction of nanoparticle formation and growth in turbulent flows. • Zero-dimensional simulations • Laminar flow approximations • Approaches: moment methods, direct simulation, and discrete / sectional methods Instantaneous concentration of 4nm diameter particles Turbulence/transport effects significant downstream Overview of AHPCRC Research Activities

14. Numerical Simulation of Turbulent, High-Speed Flows • Goal: Develop "subgrid-scale" SGS closures for turbulent reacting flows, and to implement these closures in flows representative of those encountered in projectile-target interactions. • Reynolds-averaged Navier-Stokes Simulation • Computationally affordable • Closures model wide range of length scales • Large Eddy Simulation • Capture “large” scale - Model “small” scale • Computationally affordable? Three-dimensional round jet of propane issuing into air. Non-reacting (hydrodynamic and hydrochemical interactions only) Overview of AHPCRC Research Activities

15. Parallel Software Developments • Critical stages • Scalable parallel graph partitioning – Mapping and load balancing • Sub-domain integration – Domain decomposition techniques • Time integration operators • Scalable computations - arbitrary large problems and processors • Congruent amongst sub-domains • Reduced solution times • Numericalscalability – no degradation in convergence of numerical algorithms. • Non-linear solvers and linear solvers • Parallel - ability to deliver speedups in terascale range • Memory utilization • REduced Complexity In Programming Environment – [RECIPE] paradigm • Few lines of code - unified and integrated implementation, increased functionality, arbitrary increase in order of accuracy of time integrators • Code optimization – reduced development efforts SGI Origin bristled fat hypercube interconnection network Cray T3E 3D bi-directional torus interconnection network Overview of AHPCRC Research Activities

16. Objectives: Develop a virtual computing environment for Future Combat Systems which includes a synthetic battlefield and the creation of a multidisciplinary computing environment for virtual HPC design. Portfolio: VIRTUAL COMPUTING ENVIRONMENT FOR FUTURE COMBAT SYSTEMS Visualization Scene Generation Overview of AHPCRC Research Activities

17. Projects: Visualizing Spatial/Temporal Data for Battlefield Visualization. High Performance Geographic Information Systems for Battlefield Visualization in Future Combat Systems. The Virtual Data Grid: An Infrastructure to Support Distributed Data-centric Applications. Rapid Physical Prototyping in Support of Battlefield Visualization and Signature Modeling. Innovative HPC Design and Analysis Approaches for Flexible Multi-body Structures. Portfolio: VIRTUAL COMPUTING ENVIRONMENT FOR FUTURE COMBAT SYSTEMS Overview of AHPCRC Research Activities

18. High Performance Geographic Information Systems (GIS) for Battlefield Visualization in Future Combat Systems Maps are as important to soldiers as guns Shooters Network Sensor Network HPGIS National Assets, e.g. Maps Example Usage of Geographic Info. Systems (GIS) in Battlefield : • Rescue of pilots after their planes went down (recently in Kosovo) • Precision targeting e.g. avoid civilian casualities (e.g. friendly embassies) • Logistics of Troop movements, avoid friendly fires Overview of AHPCRC Research Activities

19. RPP is used to create physical prototypes of 3D solids from their digital representations, using a “3D printer” attached to a workstation. Model Acquisition • CAD Software • CT Scans • Laser Scanning • 3D Photography Computer-Aided Process Planning • File repair • Model orientation • Slicing • Support creation Model Building via Layered Manufacturing Postprocessing • Remove supports • Improve finish • Inspect model LAN or Internet Rapid Physical Prototyping RPP “prints” the 3D model as a stack of 2D layers, using a technique called Layered Manufacturing. Goal: Create, on demand, physical scale models of enemy terrains and assets to help mission planners and field commanders develop and evaluate different combat strategies. Overview of AHPCRC Research Activities

20. hpc simulations can be launched by VDG clients to “produce” VDG data information grid/sensor grid/shooter grid national assets Internet sources hpc simulations sensor networks VDG API for data producers data can be pushed or pulled into VDG VDG server VDG server VDG server wired or wireless access VDG API for client access battlefield simulation battlefield visualization VDG user The Virtual Data Grid: An Infrastructure to Support Distributed Data-centric Applications VDG is a persistent data network designed to support military applications both in the field and at the Army labs Key Features: Reliable, Efficient, Security, Heterogeneity ET metacomputing portfolio connections may be intermittent Overview of AHPCRC Research Activities

21. Objectives: An understanding of protein interactions with toxins and pathogens at the atomistic level to help counteract chem-bio threats. Modeling of the adsorption, transport, diffusion, and dispersion of chemical and biological agents within, across, or into a variety of media, i.e., the atmosphere, water, soil, clothing, building materials, and vegetation A201 E199 G118 Bound Sarin G119 H440 E327 F290 F288 Millard . (1999) 38 , 7032-9 et al Biochemistry Portfolio: CHEM-BIO DEFENSE AND ENVIRONMENTAL MODELING Overview of AHPCRC Research Activities

22. Projects: Chemical/Biological Defense. Wireless GIS-Based High Performance Simulation of Dispersions. Chemical-Biological Applications of Mesoscale Atmospheric Modeling. Mechanics of Colloidal Transport in Contaminant Dispersion. Evaluation of the Behavior of Chemical and Biological Agents in the Soil and Deep Subsurface Environments Air Quality, Dispersion and Atmospheric Radiative Properties. Environmental Quality Modeling. Portfolio: CHEM-BIO DEFENSE AND ENVIRONMENTAL MODELING Overview of AHPCRC Research Activities

23. FAS2 Human A201 AChE E199 G118 Bound Sarin G119 H440 E327 F290 F288 Kryger . Sussman (2000) D56 , 1385-9 et al Acta Cryst Millard . (1999) 38 , 7032-9 et al Biochemistry Current Modeling Approach uses a combination of experimental and theoretical techniques. The above figures are derived from X-ray crystal structures. The structures are then computationally modeled to develop an understanding of the process by which the nerve agents bind to the enzyme. Computational Approach The target of many nerve agents is the enzyme, acetylcholinesterase (AChE). Molecular dynamics tools are used to study the molecular motions that occur inside the structure of the target, when it reacts with chemical “nerve agents,” or is inhibited by FAS2. Objective Develop effective countermeasures to protect personnel from nerve agents such as sarin or the biological snake toxin, FAS2 Finding Reactive Sites in Proteins Nerve Agent Snake Toxin Nerve Agent Snake Toxin Sarin - AChE Complex FAS2- AChE Complex Sarin - AChE Complex FAS2- AChE Complex Overview of AHPCRC Research Activities

24. GIS-Based High Performance Simulation of Dispersion Approach • Geometry Representation • Compatible Geometry Representation for Automatic Mesh Generation • Accurate Flow Solver • Boundary Condition + Initial Condition Overview of AHPCRC Research Activities

25. CFD Modeling of Contaminant Dispersion in Urban Environments • Governing equations: • Incompressible N-S equations • Boussinesq approximation for r • LES model for turbulent dissipation • Numerical method: • Standard predictor-corrector method • Solve Poisson equation for pressure • Use grid masking for solid objects: • Enforce zero flux through surface • Modify equations within object • Solve pressure equation in full domain • Inexpensive, flexible Overview of AHPCRC Research Activities

26. Mechanics of Colloidal Transport in Contaminant Dispersion Problem Statement: • Contaminants adsorb to colloidal particles (e.g. clay fragments in groundwater, aerosol/droplet inclusions). • Colloids are transported greater distances than contaminants dissolved in the fluid or gas phase. Governing Equations: • Flow is resolved on the length scale of typical pore spaces using 3-D Navier-Stokes equations. • Transport is modeled by the microscale convection-diffusion equation. No tunable parameters. Overview of AHPCRC Research Activities

27. Chem-Bio Applications of MM5 Address the added complications of land-surface and land-cover at finer resolutions Improve model adaptation to time varying land-surface conditions through GIS LULC extraction Identify model enhancements required that involve physics of flow of water and transport of heat within variably-saturated, variably-frozen soils Implement MM5 on T3E with a grid spacing of 1-km & evaluate the results of the surface condition and boundary layer packages Evaluation of model reliability Assess methods to incorporate chemical-biological dispersion models implicitly with MM5 (with CAU) Chemical-Biological Applications of Mesoscale Atmospheric Modeling Overview of AHPCRC Research Activities

28. Objectives: Conduct basic research in enabling technologies and computational algorithms in support of the Army's scientific and technology goals of the Objective Force and Army transformation Data mining algorithms for discovery of useful patterns in massive data sets. Software infrastructure for metacomputing systems. Mesh partitioning / domain decomposition. Basic research on the development of new computational algorithms in support of interdisciplinary computational research. Portfolio: ENABLING TECHNOLOGY AND COMPUTATIONAL ALGORITHMS Overview of AHPCRC Research Activities

29. Projects: Data Mining Techniques for Large Data Sets. Scheduling and Resource Management in Metacomputing Systems. Computational Algorithms for Time Dependent Problems: Unified Mathematical Framework, Design and Implementation Aspects. New Time Dependent Algorithms For Quantum Mechanics/Molecular Mechanics.  Hybrid Structured/Unstructured Implicit Methods for Complex High-Speed Flows. PSE Method Development for Projectile Transition and Turbulence. Precise Contact-Algorithms for Computational Mechanics. Scalable High Performance Computing Environment for Large-Scale Simulation of Free Surface Flows. Development of Numerical Algorithms for 3D, Viscoelastic, Fluid Flows with Moving Boundaries for Novel Lightweight Materials. Portfolio: ENABLING TECHNOLOGY AND COMPUTATIONAL ALGORITHMS Overview of AHPCRC Research Activities

30. Data Mining for Homeland Defense Objectives: • Information fusion from diverse data sources including intelligence, agencies, law enforcement, profile … • Data mining on this information base to uncover latent models and patterns Overview of AHPCRC Research Activities

31. Data Mining for Network Intrusion Detection Objective:Develop techniques to detect and identify attacks against computers, networks, and the information stored therein. • Misuse Detection - Predictive models • Mining needle in a haystack – models must be able to handle skewed class distribution, i.e., class of interest is much smaller than other classes. • Learning from data streams - intrusions are sequences of events • Anomaly and Outlier Detection • Able to detect novel attacks through outlier detection schemes • Detect deviations from “normal” behavior as anomalies Overview of AHPCRC Research Activities

32. Scheduling and Resource Management in Metacomputing Systems • Metacomputing /Grid technology is a fundamental building block for distributed HPC • interconnect disparate resources, data sources, and computations • virtualization of computing • high performance opportunities • single applications: remote supercomputing, resource aggregation • multiple applications: high throughput • Scheduling is needed to realize this HPC potential Overview of AHPCRC Research Activities

33. Technology Transfer - Collaboration with Army Scientists • AHPCRC has a strong record of joint work with Army scientists and engineers, including numerous scientific papers, workshops, reciprocal short- and long-term visits and software development: • State-of-the-art software developed by AHPCRC researchers has been incorporated into many Army and DoD codes. Such transfers of software are the result of significant interaction with Army scientists. • Many students formerly supported by AHPCRC research programs are now employed as scientists at Army laboratories and various DoD organizations • The Infrastructure Support staff plays an instrumental role in technology transfer, by collaborating in the development of software and applying the latest software developments to specific Army applications. Overview of AHPCRC Research Activities

34. Material Deformations Technology Transfer - Software • Software and tools developed at AHPCRC are being used at the Army and throughout the DoD. • Geometric modeling • Automatic mesh generation • Visualization • Mesh partitioning • Sparse linear systems solvers • Process modeling and simulation • Structural dynamics • Example:METIS and ParMETIS graph partitioning libraries are used world-wide for partitioning unstructured and adaptive graphs and have been incorporated in numerous DoD codes such as CTH/PCTH, Paradyn, ParaAble, ADH, MPMC-NET, ICM-TOXI, and FEMWATER. Overview of AHPCRC Research Activities

35. Summer Institutes are conducted to encourage students, particularly minority and female students, to pursue studies and careers in HPC and defense-critical technology areas: Over 200 students since 1991 Emphasize recruiting from partner schools High success rate Majority of them have entered Master and Ph.D. programs (many at AHPCRC partner institutions) Many placed in internships or have pursued technical careers (ARL, Cray Research, Dupont, Intel Corp, GTE, GM, CEWES, NASA, Dept. of Energy, Lucent Technologies, Dow Chemical, Cigna, Argonne National Laboratory, LLNL, 3M, IBM, TRW, Citibank, Oracle Corp., Lockheed Martin, Konica, Digi International, Diversified Pharmaceuticals) 2 people founded successful companies Summer Institute Overview of AHPCRC Research Activities

36. Outreach - Workshops and Conferences • Provide a forum for the exchange of information among AHPCRC researchers, Army collaborators and other interested DoD researchers: • ARL Intrusion Detection Systems Research WorkshopMarch 19-20, 2002 Higher Education and Applied Technology (HEAT) Center, Aberdeen, Maryland • Computational Electromagnetics (CEM) WorkshopJun 27-28, 2002 Sheraton College Park, Adelphi, Maryland • CFD / CSM for Projectile's Aerodynamics and Propulsion,Aug 13-14, 2002 Clark Atlanta University, Atlanta, GA • Contact Impact Modeling and SimulationSept. 2 (tentative) Minneapolis, Minnesota • Mesoscale Data Integration WorkshopSep 9-10, 2002 University of North Dakota, Grand Forks, ND Overview of AHPCRC Research Activities

37. Outreach - Workshops and Conferences • 10th Conference on Computational Chemistry (JSU, November 2001) Attended by two Nobel Laureates Banquet speaker: Mr. W. Hollis • Second SIAM International Conference on Data Mining(April 11-13, 2002 – Hyatt Regency, Crystal City) • International Conference Series as a Follow-up of the Workshop on Mining Scientific and Engineering Datasets (held at the AHPCRC, 1999,200) Overview of AHPCRC Research Activities

38. Synergistic Relationships Among Partners • Projectile Target Interaction • UM, CAU, FAMU, HU, NCS • Signature Modeling • HU, UND, UM, JSU, NCS • Chem-Bio Defense • CAU, JSU, FAMU, UND, UM, NCS • ET & Computational Algorithms • FAMU, HU, UM • VCE • CAU, UM, NCS Overview of AHPCRC Research Activities

39. Synergistic Relationships Example: Computational Chemistry • Goal is to predict chemical properties of interest. • Sensitivity of energetic materials • Properties of stealth coatings • Conformational and reactive site variations in proteins UM: Geometric modeling and data mining JSU: Computational chemistry and biology FAMU: Energetic materials NCS: Computational chemistry Overview of AHPCRC Research Activities