TAXI Code Overview and Status

1. TAXI Code Overview and Status Timmie Smith November 14, 2003

2. Problem Specification • Given a configuration at time T • Spatial domain divided into cells • Energy range divided into groups • Quadrature set chosen for angular integrals • Fixed Sources, Boundary Conditions, and Initial Conditions • Compute particle flux at time T + DT • Particle flux at time T + DT • Reaction rates, etc., during time step

3. Grid Cell Element Element Element Element TAXI Data Structures • Spatial domain decomposed into Cells • Cells are stored in a grid • Cell contains Elements • Elements are the base spatial data structure • Material definitions stored as in time-dependent neutronics problems • Number densities stored by cell • Microscopic cross sections stored by isotope

4. Aggregation in TAXI • Angle Sets • Angles in AngleSet generate same dependences • Energy Group Sets • Cell Sets • Code computes aggregation • User can specify an aggregation if they want • Work function operates on GroupSet-AngleSet-CellSet combination

5. STAPL Overview • Standard Template Adaptive Parallel Library • Superset of C++ STL • pContainers – generic distributed containers • pAlgorithms – generic parallel algorithms • pRange - abstract view of partitioned data • binds pContainers and pAlgorithms • view of dist. data, random access, data dependences Example: parallel sorting a distributed vector stapl::pvector<int> pv(100); … initialize pv … stapl::p_sort( pv.get_prange() );

6. User Code pAlgorithms pContainers pRange ARMI Pthreads OpenMP MPI Native The STAPL Programming Environment

7. Grid Overview • ArbitraryPGrid is derived from • BaseSpatialGrid • pGraph • Grid is templated on • cell type (brick, polyhedral, etc.) • element type (whole_cell, corner, ...) • It contains instantiations of • cells (one per graph_vertex) • faces (really graph_edges) • It contains methods for • putting outgoing-surface intensities into graph_edges • getting incoming-surface intensities from graph_edges • both of these are (anglesetgroupset) at a time Graph Vetex Graph Edge Cell

8. Cell Overview • Cell class doesn’t contain much. • list of isotopes and their number densities • list of elements • list of vertex indices • The “element” concept gives us great generality and flexibility. • There is one element for each “fundamental” spatial unknown. For example: • simple methods with one unknown per cell use the “whole-cell” element • methods with one unknown per vertex per cell use the “corner” element

9. Element Overview • Elements contain: • volumetric sources (scattering and total) • the solution (angular moments for generating the scattering source; also the angular intensities in a time-dependent problem) • Most of the code is unaware of problem geometry or discretization method. • It just loops over elements • Work function processes the Cells in a STAPL pRange

10. TAXI Setup • Energy and Angle objects constructed • Grid partition decided • Specialized STAPL scheduling algorithms used • Partition information distributed • Grid constructed in parallel • Cells and Elements created and inserted • pRanges built on grid • Specialized STAPL scheduler determines pRanges • pRange dependence graphs are directed

11. TAXI Implementation History • C++ selected for implementation • Wanted to use STL and STAPL • Prototype implemented with C++ and MPI • STAPL components weren’t fully implemented • STAPL didn’t support distributed memory systems • STAPL has advanced • ARMI Runtime System provides a consistent base for development • Allows STAPL to run on distributed memory systems • pContainers provide data distribution and parallel operations • pRange supports hierarchical decomposition of data • Scheduling components incorporated into STAPL

12. STAPLing TAXI • Custom Data Structures • Distributed 3D vector class • User managed distribution • 3D indexing (e.g. foo[x][y][z]) • Stored Cells and CellSets • CellSet class • Explicit list of graph vertex ids (Cells in the CellSet) • List of ids provided cell sweep order

13. STAPLing TAXI • pGraph stores Cells • Each pGraph vertex is a Cell • pGraph edges identify neighboring cells • Arbitrary grids now possible • 2D grids also possible • pRange used to represent CellSet • General representation of dependences between CellSets • Multiple execution orders of cells in subrange possible

14. 1 4 32 29 2 5 3 8 31 28 30 25 3 6 9 2 7 12 30 27 24 31 26 21 4 7 10 13 1 6 11 16 29 26 23 20 32 27 22 17 8 11 14 17 5 10 15 20 25 22 19 16 28 23 18 13 12 15 18 21 9 14 19 24 21 18 15 12 24 19 14 9 16 19 22 25 13 18 23 28 17 14 11 8 20 15 10 5 20 23 26 29 17 22 27 32 13 10 7 4 16 11 6 1 24 27 30 21 26 31 9 6 3 12 7 2 28 31 25 30 5 2 8 3 32 29 1 4 pRange Dependence Graph angle-set A angle-set B angle-set C angle-set D • Numbers are cellset indices • Colors indicate processors

15. angle-set A angle-set B angle-set C angle-set D 1 4 32 29 2 5 3 8 31 28 30 25 3 6 9 2 7 12 30 27 24 31 26 21 4 7 10 13 1 6 11 16 29 26 23 20 32 27 22 17 8 11 14 17 5 10 15 20 25 22 19 16 28 23 18 13 12 15 18 21 9 14 19 24 21 18 15 12 24 19 14 9 16 19 22 25 13 18 23 28 17 14 11 8 20 15 10 5 20 23 26 29 17 22 27 32 13 10 7 4 16 11 6 1 24 27 30 21 26 31 9 6 3 12 7 2 28 31 25 30 5 2 8 3 32 29 1 4 Adding a reflecting boundary

16. angle-set A angle-set B angle-set C angle-set D 1 4 32 29 2 5 3 8 31 28 30 25 3 6 9 2 7 12 30 27 24 31 26 21 4 7 10 13 1 6 11 16 29 26 23 20 32 27 22 17 8 11 14 17 5 10 15 20 25 22 19 16 28 23 18 13 12 15 18 21 9 14 19 24 21 18 15 12 24 19 14 9 16 19 22 25 13 18 23 28 17 14 11 8 20 15 10 5 20 23 26 29 17 22 27 32 13 10 7 4 16 11 6 1 24 27 30 21 26 31 9 6 3 12 7 2 28 31 25 30 5 2 8 3 32 29 1 4 A second reflecting boundary

17. angle-set A angle-set B angle-set C angle-set D 1 4 32 29 2 5 3 8 31 28 30 25 3 6 9 2 7 12 30 27 24 31 26 21 4 7 10 13 1 6 11 16 29 26 23 20 32 27 22 17 8 11 14 17 5 10 15 20 25 22 19 16 28 23 18 13 12 15 18 21 9 14 19 24 21 18 15 12 24 19 14 9 16 19 22 25 13 18 23 28 17 14 11 8 20 15 10 5 20 23 26 29 17 22 27 32 13 10 7 4 16 11 6 1 24 27 30 21 26 31 9 6 3 12 7 2 28 31 25 30 5 2 8 3 32 29 1 4 Opposing reflecting boundary

18. STAPLing TAXI • Abstracting Communication • Explicit MPI calls in driver function replaced with calls to ARMI methods • pGraph uses ARMI to transfer information between threads when necessary • ARMI fences replace MPI barriers • Fences required during problem initialization • Most fences handled inside STAPL

19. Effects of STAPLing • Problem Time to Completion • Average 2.16 times slower but improving rapidly • Currently 15 global barriers in the ASCI code • There is opportunity for improvement

20. Effects of STAPLing • Reduced code size • Code reduced ~6,000 lines. • Currently 13,000 lines and shrinking • Elimination of classes • MPI wrappers • XYZSpace • CellSet • Code Reorganization • Problem class reorganized • Eliminated explicit grid distribution • Use of pRanges

21. TAXI Driver void driver() { BaseProblem* problem = read_problem(*input_stream, *output_stream, *error_stream); problem->solve(); problem->report(); } BaseProblem* read_problem(istream& input_stream, ostream& output_stream, ostream& error_stream){ ProblemInput* pinput = new ProblemInput(input_stream); Hex_cell_creator creator(*pinput); BaseEnergyAggregation egs; XMLObject xml_input = pinput->getTagObject("prototype")[0]; egs.init_from_input(xml_input, output_stream); asci_preprocessing::ASCI_Regular_pRange_Preprocessor* sched = build_brick_scheduler(creator.dimx(), creator.dimy(), creator.dimz(), egs, *pinput); BaseProblem* bp= new GenericSweepProblem<LogicallyRectangularGrid, Wt_Diamond_Diff_Method>(pinput, creator, egs, sched, output_stream, error_stream); return bp; }

22. Questions?