240 likes | 411 Views
Profiling, Tracing, Debugging and Monitoring Frameworks. Sathish Vadhiyar. Courtesy: Dr. Shirley Moore (University of Tennessee). Outline. Profiling ,Tracing, Debugging Tools – Main thrust on Vampir and TotalView Debugging – TotalView. Definitions – Profiling and Tracing.
E N D
Profiling, Tracing, Debugging and Monitoring Frameworks Sathish Vadhiyar Courtesy: Dr. Shirley Moore (University of Tennessee)
Outline • Profiling ,Tracing, Debugging Tools – Main thrust on Vampir and TotalView • Debugging – TotalView
Definitions – Profiling and Tracing • Profiling – Summary information about various statistics - Number of function calls made - Execution times for functions - Number of floating point instructions - FLOP count etc. • Tracing –Information about events - Time stamps about entering/exiting calls - Interaction between processes (send and recv traces)
Vampir • For performance analysis of parallel applications • Event-based and trace-based performance analysis tool • Different statistical displays – - timeline displays - profiling statistics (execution times of routines) - communication statistics • Can be analyzed at different levels – zoom in facility • Needs VampirTrace for instrumentation
Problems Identified • Excessive communication cost • load (im)balance • Communication bottle-necks • Shows percentage spent in all MPI procedures and actual applications • Shows percentage time spent for each MPI procedure
VampirTrace • Vampir graphical tool uses VampirTrace tracing library • VampirTrace uses MPI profiling interface to intercept MPI calls • E.g.: mpicc –o my_program my_program.c -L<vampirtrace_dir>/lib –lVT Gives rise to my_program.stf
My VAMPIR results • [secvss@altix openmp]$ vampir /opt/vampir/examples/redblack_sndrcv.stf • VAMPIR: Could not find matching license. • VAMPIR: Need license for product VA40. • VAMPIR: platform IA64-LIN • VAMPIR: hostid 0x10908e31, network address 144.16.49.142, userid 2000 • VAMPIR: Environment variable PAL_ROOT set to • /opt/vampir. • VAMPIR: Environment variable PAL_LICENSEFILE set to • /opt/vampir/etc/license.dat. • VAMPIR: Environment variable VT_ROOT set to • /opt/vampir_trace.
Some more Parallel Profiling Libraries • GuideView (for OpenMP from Pallas) • Upshot / Jumpshot (MPI from Argonne) • SvPablo (UIUC for MPI and OpenMP) • DynaProf (UT)
Parallel Debugging – TotalView (Etnus) • Source level parallel debugger • Multithreaded, multiprogram debugger for MPI, PVM, OpenMP • Examine and change data • Can attach to all or some of running processes • Separate window for each process • Can change source code while debugging • Start/stop execution
TotalView Introduction • Detailed information by diving • Computational steering by changing variable values or by using expressions • Displaying variables across processes/threads
Action Points • Breakpoints • Barrier breakpoints • Conditional breakpoints • Evaluation point • Watchpoints
MPI and TotalView • mpirun[ MPICH-arguments ] -tv program [ program-arguments ] • totalview poe -a program[ arguments ][ PE_arguments ]
Break point Example 1 pid = fork(); 2 if (pid == -1) 3 error ("fork failed"); 4 else if (pid == 0) 5 children_play(); 6 else 7 parents_work(); Conditional Breakpoint: if (i > 20 && i < 25) $stop;
Research Areas • Compact representation of trace files • Minimize overhead due to tracing
Further Reading • Review of performance analysis tools for MPI programs (www.netlib.org/utk/people/JackDongarra/PAPERS/perftools-review2.pdf) • TotalView Hands-on Exercises (http://www/llnl/gov/computing/tutorials/workshops/workshop/total_view/MAIN.html