1 / 32

Dynamic Tuning of Parallel Programs with DynInst

Paradyn/Condor Week, March 2002. Dynamic Tuning of Parallel Programs with DynInst. Anna Morajko, Tomàs Margalef, Emilio Luque Universitat Autònoma de Barcelona. Content. Our goals - dynamic performance tuning Steering loop Dynamic tracing Performance analysis Dynamic tuning DynInst

jenette-miller
Download Presentation

Dynamic Tuning of Parallel Programs with DynInst

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Paradyn/Condor Week, March 2002 Dynamic Tuning of Parallel Programs with DynInst Anna Morajko, Tomàs Margalef, Emilio Luque Universitat Autònoma de Barcelona

  2. Content • Our goals - dynamic performance tuning • Steering loop • Dynamic tracing • Performance analysis • Dynamic tuning • DynInst • Application development framework • Conclusions and future work

  3. Our goals The objective is to: • create a system that improves performance of parallel applications during their execution without recompiling and rerunning them • Our approach to performance improvement: • Phases: • Tracing • Performance analysis • Tuning • All done during run-time

  4. Dynamic performance tuning Application development Source Problem / Solution Modifications Instrumentation Events Application User Exe-cution Tracing Tuning Tool Performance analysis Suggestions

  5. Dynamic performance tuning Machine 1 Machine 2 Apply solutions Task1 Task1 Task2 Change instrumentation Change instrumentation Events Events Machine 3 Tuner Tuner Tracer Tracer pvmd pvmd Analyzer

  6. Dynamic performance tuning Requirements: • all optimization phases done during run time • improve performance changing execution of an application • without source code access, re-compile, re-link and re-run Dynamic instrumentation (DynInst) instrumentation and tracing tuning

  7. Dynamic performance tuning • Analysis • application dependent performance model: • what to measure • how to detect bottlenecks • how to resolve it • simple analysis – decisions must be taken in a short time • Execution modifications • what can be changed - complexity when performing application changes without knowing its internal code • and how it can be changed • Intrusion • minimization of intrusion caused by system modules, instrumentation, event generation and communication

  8. Content • Our goals - dynamic performance tuning • Steering loop • Dynamic tracing • Performance analysis • Dynamic tuning • DynInst • Application development framework • Conclusions and future work

  9. Dynamic tracer Goals of the dynamic tracing tool: • control of the distributed application • application instrumentation management • event generation • intrusion minimization • portability and extensibility Version: • implementation in C++ for PVM based applications • Sun Solaris 2.x / SPARC

  10. Dynamic tracer Machine 1 Task1 Master Tracer pvmd Machine 2 Machine 3 Slave Tracer Task4 Slave Tracer Task2 Task3 pvmd pvmd Distributed application control: • Process control with PVM Tasker service • VM control with Hoster service • Master tracer / Slave tracers • Clock synchronization(event timestamp with global order)

  11. Dynamic tracer Machine N Task2 Task1 lib lib Tracer Config Application instrumentation management: • Instrumentation • inserted into a set of selected functions (configurable) • this set can be changed during execution (insert, remove) • Different snippets: • Event generation (function entry, exit) • Line number information(function call points) • Snippets call tracing library dynamically loaded into a running task

  12. Dynamic tracer Machine N Task1 lib pvm_send (params) { } Tracer entry exit LogEvent (params) { ... } 995364884 524247 1 0 262149 23 1 Trace file Analyzer Event generation: • When - global timestamp • Where - task • What - type of event • Additional information(i.e. function call params,source code line number)

  13. Dynamic tracer Machine 1 Machine 2 Task3 Task2 Task1 lib lib lib Machine 3 Event collector Analyzer Event Collection: • Centralized event collector receives events • Global ordering • Flow of ordered events is passed to the analyzer • To consider, e.g. event buffering, periodical flush

  14. Performance analysis Clue questions: • What should/should not happen during execution (what kind of problems, performance bottlenecks)? • What events are needed (what should be collected to help discover what really happened)? • How to deduce what really happened (finding bottlenecks and causes)? • How to improve performance (finding solutions)?

  15. Performance analysis • Performance model • some fixed model for an application - need to determine desired performance characteristics • model determines what to measure • Analyzing events that come from event collector • Evaluating the model • Giving solutions - the best possible changes that will improve the performance • Sending solutions to tuner WORK IN PROGRESS

  16. Dynamic tuning What can be changed? • parameters: changing parameter value • application contains well-known variables • application is prepared for changes • set of pre-defined snippets • choosing strategy – function replacement • changing order of functions • dynamic snippet creation • it depends on the solutions generated by analyzer • very difficult Tested In progress Future

  17. Dynamic tuning Machine 2 Machine 1 Task3 Task1 Task2 Tuner Tuner Machine 3 Analyzer Where can be changed? • All tuners receives changes (i.e. SPMD) • One concrete tuner that tunes one concrete process (i.e. MW)

  18. Steering loop Machine 1 Machine 2 Apply solutions Task1 Tuner Tuner Task1 Task2 Tracer Tracer lib lib lib pvmd pvmd Change instrumentation Change instrumentation Events Events Machine 3 Event collector Analyzer Event Collection Protocol Performance Tuning Protocol

  19. Content • Our goals - dynamic performance tuning • Steering loop • Dynamic tracing • Performance analysis • Dynamic tuning • DynInst • Application development framework • Conclusions and future work

  20. DynInst Two principal reasons for using DynInst: • instrumentation and tracing • collect information about application behavior • tuning • insert/remove/change code in order to improve application performance

  21. DynInst PVM application foo_1 (...) { pvm_send (); } pvm_send (params) { ... } Tracer ... thread.loadLibrary(“libLog.so”); BPatch_function f = image.findFunction (logEvent); BPatch_funcCallExpr log (f); BPatch_point point (image, pvm_send, BPatch_entry); thread.insertSnippet (log, point); log () { logEvent(format, params) } Snippet logEvent(format,...) { // transfer event } Dynamic library (libLog.so) Instrumentation and tracing:

  22. DynInst for (int i=0;i<100;i++) j = DIM % (nunw); pvm_pkint (x,1,2); pvm_send (...); i = DIM*2/(nunw); DIM = j; pvm_pkint (x,3,1); pvm_send (...); numr = DIM/(nunw); extra = DIM % (nunw); pvm_pkint (x,1,2); pvm_send (...); numr = DIM/(nunw); extra = DIM % (nunw); pvm_pkint (x,1,2); pvm_send (...); numr = DIM/(nunw); extra = DIM % (nunw); pvm_pkint (x,1,2); pvm_send (...); Event line number: logLineNumber PVM application pvm_send(...) { } ...

  23. DynInst Tracer BPatch_function f=image.findFunction (logLineNumber); BPatch_funcCallExpr logLine (f); CallPointList cp (“pvm_send”) for each cp addr = cp.getAddress (); thread.GetLineAndFile (addr); thread.insertSnippetBefore (logLineNumber, cp); PVM application foo_1 (...) { pvm_send (); pvm_recv (); } ... logLine () { logLineNumber (params) } Snippet Event line number:

  24. DynInst ... changeParam (param, value) { send_req (tuner, param, value); } Analyzer recv_req (param, value) { BPatch_variable variable = image.FindVariable (param); variable.writeValue (value); } Tuner main () { int tuneParam; int param; if (param != tuneParam) ... } PVM application Tuning:

  25. DynInst Problems when using DynInst: • Line number • We need to scan all functions to be instrumented and additionally instrument them (before execution) • Insert snippet that logs line number (before function call) • Could be nice to access callee address (call stack??) • Limited number of parameters that can be passed to library function call (max. 5 for Sun Solaris) • Problems with scanning function call points (e.g. C++ mutatee with operator<<, function sleep) • BPatch_function::findPoint (subroutine) • BPatch_point::getCalledFunction ()

  26. Content • Our goals - dynamic performance tuning • Steering loop • Dynamic tracing • Performance analysis • Dynamic tuning • DynInst • Application development framework • Conclusions and future work

  27. Application framework API Parallel pattern Application Framework Perf. model Measure points Tuning points Framework for parallel application development • Based on patters, e.g. master worker, pipeline, divide and conquer • Support for dynamic tuning system: • Measure points, performance model, tuning points • Support for parallel application developer: • hides implementation details • developer provides only specific functionality

  28. Dynamic performance tuning User ApplicationDevelopment Application Problem / Modifications Exe-cution API Solution Application Framework Instrumentation Tracing Tuning Measure points Events Tool Performance analysis Perf. model Suggestions Tuning points

  29. Content • Our goals - dynamic performance tuning • Steering loop • Dynamic tracing • Performance analysis • Dynamic tuning • DynInst • Application development framework • Conclusions and future work

  30. Conclusions • Dynamic tuning • Design of dynamic tuning tool • Characteristics • Steering loop • Design details • Implementation of dynamic tracer • Bases of performance analysis • Tuning methods • How we use DynInst • Parallel application development framework

  31. Future work • Tracer • event buffering, changing instrumentation on demand, performance measurements • Analyzer • performance analysis • Tuner • other solutions that can be applied when changing??? • Find and analyze more examples of what to measure, how to analyze and what to change

  32. Thank you very much

More Related