Overview of DynTG

1. Overview of DynTG Adam Leko UPC Group HCS Research Laboratory University of Florida Color encoding key: Blue: Information Red: Negative note Green: Positive note

2. Basic Information • Name: DynTG • Developer: Center for Applied Scientific Computing, Lawrence Livermore National Laboratory • Current Version: • (Unreleased) • Website: • http://www.llnl.gov/CASC/tool_gear/ • Contact: • Martin Schulz (schulzm@llnl.gov) • John May (johnmay@llnl.gov) • John Gyllenhaal (gyllen@llnl.gov)

3. Overview of DynTG • Not yet publicly available yet • A release should be available “in the next few weeks” as of 5/26/2005 according to developers • Still nothing available • Information in this overview based on [1] • DynTG brings together two existing tools: DPCL and Tool Gear • Lets the user instrument their program by viewing source code and choosing instrumentation points at runtime • Uses Tool Gear to provide source browser and metric display to user • Uses DPCL to perform dynamic instrumentation at runtime

4. Related Project: DPCL • DPCL – Dynamic Probe Class Library [2] • Based on DynInst from Paradyn • Meant to provide performance tool developers with a library that handles instrumenting distributed programs during runtime • Also has built-in security features • Architecture of DPCL shown right • Analysis tool uses DPCL “super daemon” to establish secure connection to DPCL daemons on each host • DPCL daemons (one per user per host) use DynInst to perform instrumentation of running programs and handle sending data back to the analysis tool • Currently only works on Linux and AIX platforms

5. Related Project: Tool Gear • Tool Gear – “Software infrastructure for developing performance analysis and debugging tools for large scale parallel programs” [3] • Basic architecture shown below • Tool Gear provides a “database” by which • Performance tools give information to the tool gear viewers • Viewers pass asynchronous events (such as instrumentation requests) to a tool • Architecture handles both post-mortem and runtime analysis • But Tool Gear is much more useful for tools that perform online analysis (and hence DynTG) • However, only a few viewers that display metrics inline with source code are currently available • Tool Gear is used in mpiP’s mpipview (see mpiP evaluation)

6. Related Project: Tool Gear (2) • A good description of the purpose of Tool Gear (from [1], pg. 5): “Source code viewers are complex and difficult to implement, yet required by a large number of tools. The Tool Gear project was born based on this observation and provides a versatile framework for the development of interactive tools. As its core it provides a scalable source viewer, which can display both the source code itself as well as multiple performance metrics on a per source line basis. The performance data itself is stored inside Tool Gear using a database. Data can directly be added to this database from a collector using the Tool Gear communication library.”

7. DynTG Architecture • DynTG merges DPCL and ToolGear’s architectures together • All analysis, instrumentation, and display are performed while a program is running

8. DynTG Modules • DynTG can load “modules” (shared objects) at startup that report information back to the main viewer • Similar to Dynaprof’s loadable probe interface (see Dynaprof review) • Each probe split into two pieces • Probe module • Contains the DPCL data probes that should be inserted when the user requests a function to be instrumented • Should contain code that sends data to collector modules via DPCL’s Ais_send routine • Collector module • Receives data from the DPCL daemons, analyzes it, and gives it to the Tool Gear viewer • According to paper [1], much low-level functionality is already provided by DynTG • “allows even unexperienced [sic] users not familiar with DPCL or Tool Gear to develop new probe/collector modules”

9. Built-In Modules • Counter • Counts how often a program executes a specific location • Timer • Keeps track of how much time is spent at a location using the UNIX gettimeofday call • PMAPI • Provides direct access to IBM’s hardware counters • Users specify which hardware counters they are interested and specify priorities for each • Module maps user’s request to available hardware resources

10. DynTG GUI • User interacts with Tool Gear GUI to select what parts of their code they want instrumented (see right) • GUI loads modules and inserts probes as previously discussed • It is not clear from the paper how the tool handles starting programs

11. DynTG GUI (2)

12. DynTG Overhead • In paper, authors found runtime overhead of DynTG to be between 1.94% and 12.4% ([1], page 6) • Used two applications from the ASC Purple Benchmark Codes • SMG2000 (60x60x60) • sPPM (64x64x64) • They inserted single instrumentation action into a central location (executed around 4000 times per second) • Based on our own experience with DynInst, overhead is very low for everything but very lightweight functions • However, overhead can easily grow to several hundred percent or more for applications with functions that execute > 105 times per second • Normally these types of functions should be inlined, so not a major concern

13. References [1] M. Schulz, J. May, J. Gyllenhaal. “DynTG: A Tool for Interactive, Dynamic Instrumentation,” ICSS 2005, Atlanta, GA, May 22-25, 2005. [2] L. DeRose, T. Hoover Jr. “The Dynamic Probe Class Library – An Infrastructure for Developing Instrumentation for Performance Tools,” IPDPS 2001, April 2001. [3] Toolgear website: http://www.llnl.gov/CASC/tool_gear/