The TSTT Interface Definition Effort

The TSTT Interface Definition Effort Lori Freitag Diachin Lawrence Livermore National Lab

The TSTT Center • Goal: To enable high-fidelity calculations based on multiple coupled physical processes and multiple physical scales • Adaptive methods • Composite or hybrid solution strategies • High-order discretization strategies • Barrier: The lack of easy-to-use interoperable meshing, discretization, and adaptive tools requires too much software expertise by application scientists The TSTT center recognizes this gap and is addressing the technical and human barriers preventing use of adaptive, composite, hybrid methods

Diverse Partners Overture AOMD/Trellis CUBIT NWGrid/Phys Frontier Mesquite Nek5000 Meshing Technologies Structured/Overset Mixed element hybrid Unstructured hex Tetrahedral Variational approaches Mesh quality improvement Adaptive Technologies Mesh refinement Front tracking Software Frameworks and PDE Solvers Significant Applications Experience Overture Mesh (LLNL) MEGA Boundary Layer Mesh (RPI) NWGrid mesh CUBIT Mesh (Sandia) TSTT brings a broad set of meshing-discretization communities together

The Challenge • These tools all meet particular needs, but • They do not interoperate to form hybrid, composite meshes • They cannot be easily interchanged in an application • In general the technology requires too much software expertise from application scientists • Difficult to improve existing codes • Difficult to design and implement new codes • Our goal is to develop interchangeable and interoperable software components for meshing and discretization

TSTT interoperability goals require services build on a common infrastructure • All tools must build on a common abstract framework and data model • Provides a single reference for PDE simulation data hierarchy • Defining common interfaces is critical • Required for interchangeability of similar tools • Allows vertical integration of tools • Applications write to the TSTT specification once and can use all TSTT services Geometry Information (Level A) Full Geometry Meshes (Level B) Mesh Components (Level C)

TSTT data model abstracts PDE-simulation data hierarchy • Core Data Types • Geometric Data: provides a high level description of the boundaries of the computational domain; e.g., CAD, image, or mesh data • Mesh Data: provides the geometric and topological information associated with the discrete representation of the domain • Field Data: provides access to the time dependent physics variables associated with application solution. These can be scalars, vectors, tensors, and associated with any mesh entity. • Data Relation Managers • Provides control of the relationships among the core data types • Resolves cross references between entities in different groups • Provides functionality that depends on multiple core data types • Each core data type • has a TSTT interface • Mesh: TSTTM • Geometry: TSTTG • Fields: TSTTF

Lowers the burden for adoption of the interface • Maintain data structure neutrality • Create a small set of interfaces that existing packages can support • Small set of ‘core’ functions that must be implemented • Larger set of functions supported by reference implementations • Balance performance and flexibility • Work with a large tool provider and application community to ensure applicability • Target both specific uses of TSTT services and broad dissemination as CCA components Performance is critical for kernel computations involving mesh access; flexibility is critical for covering a broad usage spectrum CCA provides infrastructure and guidance for domain-specific interface definition efforts An overarching philosophy guidesour interface definition efforts

The TSTT mesh data model consists of entities, entity sets and meshes • Entity Definition • Unique type and topology • Canonical ordering defines adjacency relationships • Entity Set Definition • Arbitrary grouping of TSTT mesh entities • There is a single “Root Set” • Relationships among entity sets • Contained-in • Hierarchical • Mesh Definition • Simple meshes are a set of connected, non-overlapping entities related through adjacency relationships • Meshes can contain collections of simple meshes

Core functions must be implemented to be TSTT compliant • The smallest set of functions required to be considered TSTT compliant • Provides basic access to vertex coordinates and adjacency information • Mesh loading and saving • Accessing global information such as the root set, geometric dimension, number of entities of a given type or topology • Primitive array access for entity sets • Global entity arrays for entity sets • Reference implementations can be used to provide advanced capabilities needed for TSTT services • Requires a copy of mesh data • Allows incremental implementation of advanced functions

Advanced traversal functions provide additional flexibility and performance • Single Entity Traversal • Iterator functionality (init, next, reset, end) • Query (topology, type, adjacency) • Workset Traversal • Block iterators allow access of entities in a user-defined array size • Array based query functions for topology, type, adjacency • Increases performance by reducing the number of calls through the interface • Interface is not as intuitive

Advanced functionality is supported through optional interfaces • Entity Sets • Creation/Destruction • Boolean Operations • Relations • Tags • Allows the incorporate user-defined data in the TSTT data model • Associated with entities, meshes, and entity sets • Mesh Modification • Simple mesh geometry and topology modification • Requires simple classification mechanisms against the geometric model • Intended to support higher-level functionality

The geometry interface provides access to the computational domain • Must support • automatic mesh generation • mesh adaptation • tracking domain changes • relating information between alternative discretizations • Builds on boundary representations of geometry • Used to support various underlying representations • Commercial modelers (e.g., Parasolid, ACIS) • Modelers that operate from standard files (e.g. IGES, STEP) • Models constructed from an input mesh

Both basic and advanced functionalities are supported in the geometry interface • Model loading and initiation • Topological queries of entities and adjacencies • Pointwise geometric shape interrogation • Parametric coordinate systems • Model topology modification • Tags

Relating mesh and geometry data is critical for advanced TSTT services • Required for e.g., adaptive loops, mesh quality improvement • Mesh/Geometry Classification Interface • Manages the relationship between the high level geometric description and the mesh • Called by an application that knows about both • Capabilities • For a given mesh entity, get the geometric entity against which it is classified • Establish a classification relationship between a mesh entity and a geometric entity

Working documents describing the mesh and geometry data models as well as the overall vision • TSTT Mesh Interface (version 0.6) complete • Preliminary interfaces for geometry and mesh/geometry classification defined • Implementations • AOMD, FronTier, GRUMMP, MOAB, NWGrid, Overture for mesh • CGM for geometry • C, C++, and Fortran language interoperability through SIDL/Babel (CCA) • Analyzing performance ramifications of SIDL/Babel language interoperability tools (joint with the CCA) Preliminary interface implementations are well underway

Performance studies will provide ‘best practices’ guidelines • Working with SIDL/Babel team to characterize the interface performance • Focus on entity-based access • Single entity iterators • Workset iterators • Compare • Native data structures • Native language interfaces • SIDL/Babel interfaces Workset Iterators Initialize Workset(entity_type, num_entities, &wrk_iter) While (!done){ GetNextWorkset(&entity_handles[ ], &num, wrk_iter); GetInfo(entity_handles); for i=0:num-1 { do_something(entity_handle[i]); } } Destroy Workset(wrk_iter);

Preliminary performance focus on TSTT mesh access patterns • Workset Iterators • Increased workset size amortizes overhead • 15-40% compared to native data structures • SIDL a few percent more than native language • Single entity access via workset iterators 2X-4X for native and SIDL interfaces • Single entity iteratorshave similar performance to a workset size of 5-10 • Array based access for NWGrid has .5% overhead independent of mesh size • Ongoing research to further improve performance when interoperability is not critical (w/ PERC)

The TSTT mesh interface working group uniquely engages a large number of people • External (monitoring) • UIUC: Guoy • Boeing: Michal (UGC) • LLNL: Keasler • SNL: Pebay, Haynes, Edwards, Jung • ElemTech: Meyers • UU: Parker • ORNL: Bernholdt • External (active) • UofC: Loy • UBC: Ollivier-Gooch Internal • ANL: Fischer, Buschelman • LLNL: Freitag, Chand, Henshaw, Dahlgren • PNNL: Trease • RPI: Shephard, Seole • SNL: Tautges, Kim • SUNY SB: Li, Fix • Managing the interactions across institutions • Face to face meetings (all-hands and opportunistic) • Telecons (approximately every three weeks) • Email discussion group (39 participants from 13 institutions) • Working documents and SIDL files • CVS repository

Defining the TSTT interfaces has been an interesting challenge • Interface definition is harder than we anticipated • Cannot achieve the 100 percent solution, so... • What level of functionality should be supported? • How to best support existing (different) packages? • The devil is in the details • Initial interface completed a year ago • Lots of interesting issues • Invariant handles • Consistent error handling across packages • Memory management issues • Core interface functions • Performance, performance, performance

TSTT services support SciDAC applications developers • Domains from multiple geometry source • Use of alternative mesh generators • Use alternative equation discretization methods and tools • Application of adaptive control of the • Geometry - shape optimization • Mesh - discretization error control • Both - evolving geometry simulations • Combining multiple analysis codes to solve multiphysics problems

TSTT services • “Current” TSTT Services • Mesh services libraries (AOMD, MOAB, NWGrid) • Mesh quality improvement (Mesquite) • Discretization libraries (Overture, Trellis) • Parallel mesh generation (MOAB) • Front tracking (FronTier) • Adaptive control loops (combination of tools) • Services Under Development • Geometry interface functions (not doing geometry) • Fields library • Mesh control for shape optimization

Mesquite Mesh Quality Improvement • Stand alone library of mesh quality improvement algorithms • 2D/3D unstructured and hybrid meshes • Triangles, Quads, Tets, Hex elements • State-of-the-art optimization-based improvement schemes • Joint work with TOPS created fastest mesh quality optimizer in Mesquite tool suite – improves 1M elements in 15 seconds • TSTT interface to access mesh and geometry information • Vertex coordinates, element connectivity, tags • Successfully integrated with AOMD, MOAB, Overture, Cubit • Application uses • Surface mesh smoothing on complex geometries for accelerator modeling • Mesh quality analysis for SLAC Tau3P code • Mesh deformation in design optimization problems • Anisotropic smoothing for error minimization

TSTT developed a new interoperable AMR/Front tracking scheme • Combine front tracking in Frontier with adaptive techniques in Overture • Increase problem resolution using AMR • Maintain exact interface representation capabilities • Applications • Richtmeyer-Meshkov instability • Diesel jet spray break up • Using TSTT interface to combine front-tracking schemes with NWgrid capabilities

TSTT/SLAC collaboration provides adaptive mesh capabilities for Omega3P • Omega3P is SLAC’s quadratic FEM eigensolver for computing normal modes in RF cavities • Extremely accurate solutions (0.01% error) are required for requirements of accelerator design • Using new TSTT technology developed at RPI, adaptive mesh capability has been provided for Omega3P • Simulations achieve same accuracy with 1/3 the number of unknowns 3 levels of refinement Initial Mesh First Refinement Final Adaptive Mesh

Using TSTT field operatorsallows easy construction of alternative error estimators Components used in construction of adaptive loop for SLAC Using geometry operatorsmeans alternate solid modelers can be inserted Using TSTT mesh operatorsmeans alternate mesh generatorsand mesh adaptation procedurescan be inserted Mesh adaptation based on local modification linkeddirectly to CAD Unaltered SLAC code Projection-based error estimatorused to construct new mesh sizefield given to mesh modification Error estimatorsfrom RPI and SLAC

Contact Information • TSTT Web Site: www.tstt-scidac.org • David Brown: dlb@llnl.gov • Lori Freitag Diachin: diachin2@llnl.gov • Jim Glimm: glimm@ams.sunysb.edu

The TSTT Interface Definition Effort