Towards Scalable Performance Analysis and Visualization through Data Reduction

1. Towards Scalable Performance Analysis and Visualization through Data Reduction CheeWai Lee, Celso Mendes, L. V. Kale University of Illinois at Urbana-Champaign

2. Motivation Why? Event trace-based performance tools help applications scale well. As applications scale, so must performance tools.

3. Nature of Event Traces Tend to be thread or processor-centric. Volume of data per thread proportional to number of performance events encountered. Number of performance events per thread depends on duration of run and frequency of events. Strong Scaling: More threads, more communication events. Weak Scaling: More threads, more communication events, more work per thread. More events = more work for Performance Tools.

4. Reducing the data: Part 1 • Cut inconsequential event-blocks (e.g. initialization/end) Keep important snapshots (e.g. important iteration blocks) NAMD Startup First 300 steps with Load Balancing Steps 300-500 with a load refinement Baseline: Record events of the entire run. What are simple ways of reducing the volume of performance data?

5. Quantifying the Problem NAMD molecular dynamics simulations and event trace volume as generated by Projections performance tool over 200 (“interesting”) time steps. Strong Scaling Weak Scaling

6. Reducing the data: Part 2 Our Approach: Drop “uninteresting” processors (Threads) Drop “uninteresting” or some specific classes of events. Compress and/or characterize event patterns.

7. Our Approach Which? Why? How? • Choose a subset of processors: • Representatives • Outliers • Employ k-Means Clustering for Equivalence-Class discovery. • Chosen processors’ performance data are written to disk at end of run.

8. Equivalence Class Discovery Euclidean Distance Metric Y Outliers Representatives Metric X

9. Things to Consider Distance measures may require normalization. Whether certain metrics are strongly correlated to one another. Number of initial seeds. Placement of initial seeds. Number of representatives chosen. Number of outliers chosen.

10. Experimental Methodology Tuned NAMD Problem Injected NAMD (NAnoscale Molecular Dynamics) task grain-size performance problem (2002). Roll-back a performance improvement we made in 2002 to address this problem.

11. Experimental Methodology (2) • 1 million atom simulation of the Satellite Tabacco Mosaic Virus. • 512 processors to 4096 processors on PSC’sBigben Cray XT3 supercomputer. • Two criteria for validation: • Amount of data reduced. • Quality of the reduced dataset.

12. Histogram Quality Measure Original Data: 1000 pe How close is Hri/Hoi to 0.100 on average? Hoi Reduced Data: 100 pe Hri … … … … Bariorig Barireduced

13. Results: Data Reduction

14. Results: Quality

15. Conclusion Approach offers a potential way of controlling volume of performance data generated. Heuristics have been reasonably good at capturing performance characteristics of the NAMD grain-size problem.

16. Future Work Conduct experiments on more problem types and classes for verification. Find better (more practical) ways for equivalence class discovery.