1 / 23

Profiling, Tracing, Debugging and Monitoring Frameworks

Profiling, Tracing, Debugging and Monitoring Frameworks. Sathish Vadhiyar. Courtesy: Dr. Shirley Moore (University of Tennessee). Outline. Profiling ,Tracing, Debugging Tools – Main thrust on Vampir and TotalView Debugging – TotalView. Definitions – Profiling and Tracing.

isabella-paul
Download Presentation

Profiling, Tracing, Debugging and Monitoring Frameworks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Profiling, Tracing, Debugging and Monitoring Frameworks Sathish Vadhiyar Courtesy: Dr. Shirley Moore (University of Tennessee)

  2. Outline • Profiling ,Tracing, Debugging Tools – Main thrust on Vampir and TotalView • Debugging – TotalView

  3. Definitions – Profiling and Tracing • Profiling – Summary information about various statistics - Number of function calls made - Execution times for functions - Number of floating point instructions - FLOP count etc. • Tracing –Information about events - Time stamps about entering/exiting calls - Interaction between processes (send and recv traces)

  4. Vampir • For performance analysis of parallel applications • Event-based and trace-based performance analysis tool • Different statistical displays – - timeline displays - profiling statistics (execution times of routines) - communication statistics • Can be analyzed at different levels – zoom in facility • Needs VampirTrace for instrumentation

  5. Problems Identified • Excessive communication cost • load (im)balance • Communication bottle-necks • Shows percentage spent in all MPI procedures and actual applications • Shows percentage time spent for each MPI procedure

  6. Displays

  7. Displays (Summary Charts)

  8. Activity Charts (Identifying Load Balance)

  9. Communication Statistics

  10. Timeline Diagram

  11. VampirTrace • Vampir graphical tool uses VampirTrace tracing library • VampirTrace uses MPI profiling interface to intercept MPI calls • E.g.: mpicc –o my_program my_program.c -L<vampirtrace_dir>/lib –lVT Gives rise to my_program.stf

  12. My VAMPIR results • [secvss@altix openmp]$ vampir /opt/vampir/examples/redblack_sndrcv.stf • VAMPIR: Could not find matching license. • VAMPIR: Need license for product VA40. • VAMPIR: platform IA64-LIN • VAMPIR: hostid 0x10908e31, network address 144.16.49.142, userid 2000 • VAMPIR: Environment variable PAL_ROOT set to • /opt/vampir. • VAMPIR: Environment variable PAL_LICENSEFILE set to • /opt/vampir/etc/license.dat. • VAMPIR: Environment variable VT_ROOT set to • /opt/vampir_trace.

  13. Some more Parallel Profiling Libraries • GuideView (for OpenMP from Pallas) • Upshot / Jumpshot (MPI from Argonne) • SvPablo (UIUC for MPI and OpenMP) • DynaProf (UT)

  14. Parallel Debugging – TotalView (Etnus) • Source level parallel debugger • Multithreaded, multiprogram debugger for MPI, PVM, OpenMP • Examine and change data • Can attach to all or some of running processes • Separate window for each process • Can change source code while debugging • Start/stop execution

  15. TotalView Introduction • Detailed information by diving • Computational steering by changing variable values or by using expressions • Displaying variables across processes/threads

  16. Action Points • Breakpoints • Barrier breakpoints • Conditional breakpoints • Evaluation point • Watchpoints

  17. MPI and TotalView • mpirun[ MPICH-arguments ] -tv program [ program-arguments ] • totalview poe -a program[ arguments ][ PE_arguments ]

  18. Message Queue Displays

  19. Break point Example 1    pid = fork(); 2    if (pid == -1) 3        error ("fork failed"); 4    else if (pid == 0) 5        children_play(); 6    else 7        parents_work(); Conditional Breakpoint: if (i > 20 && i < 25)        $stop;

  20. Visualization of Data

  21. Research Areas • Compact representation of trace files • Minimize overhead due to tracing

  22. Further Reading • Review of performance analysis tools for MPI programs (www.netlib.org/utk/people/JackDongarra/PAPERS/perftools-review2.pdf) • TotalView Hands-on Exercises (http://www/llnl/gov/computing/tutorials/workshops/workshop/total_view/MAIN.html

More Related