Interactive and semiautomatic performance evaluation

1. Interactive and semiautomatic performance evaluation W. Funika, B. Baliś M. Bubak, R. Wismueller

2. Outline • Motivation • Tools Environment Architecture • Tools Extensions for GRID • Semiautomatic Analysis • Prediction model for Grid execution • Summary

3. Motivation • Large number of tools, but mainly off-line and non-Grid oriented ones • Highly dynamic character of Grid-bound performance data • Tool development needs a monitoring system • accessible via well-defined interface • with a comprehensive range of possibilities • not only to observe but also to control • Recent initiatives (DAMS – no perf, PARMON – no MP, OMIS) • Re-usability of existing tools • Enhancing the functionality to support new programming models • Interoperability of tools to support each other • When interactive tools are difficult or impossible to apply, (semi)automatic ones are of help

4. Component Structure of Environment

5. feedback mainly from feedback mainly from full Grid testbed local Grid testbed internal integration, testing, refinement internal integration, testing, refinement internal integration, testing, refinement WP 1 WP 1 WP 4 WP 4 2nd development 2.2 - 2.4 2.5 2.5 3rd development 1st development 27 PM 3 6 12 15 33 36 18 24 M2.4 D2.4 D2.6 D2.2 M2.1 M2.3 M2.2 PU CO CO CO internal progress report internal progress report final version D2.3 D2.5 D2.7 D2.1 Design of interfaces Between tool Design of Performance Analysis Tool IR PU PU PU PU Interface to Grid Monitoring Services Performance data Model state-of-the-art GR final demo + report 1st prototype + report 2nd prototype + report X# Task 2.4 - Workflow and Interfaces requirements from WP 1,3,4 2.1

6. Application analysis • Basic blocks of all applications dataflow for input and output • CPU-intensive cores • Parallel tasks/threads • Communication • Basic structures of the (Cross-) Grid • Flow charts, diagrams, basic blocks from the applications • Optional information on application’s design patterns: e.g. SPMD, master/worker, pipeline, divide & conquer

7. Categories of performance evaluation tools Interactive, manual performance analysis Off-line tools • track based (combined with visualization) • profile based (no time reference) • problem: strong influence when fine grained measurements On-line tools • possible definition (restriction) of the measurements at run-time • suitable with cyclic programs: new measurements based to the previous results. => Automation of the bottleneck search is possible Semi-automatic and automatic tools • Batch-oriented use of the computational environment (e.g. Grid) • Basis: Search-model: enables possible refining of measurements

8. Defining new functionality of performance tool • Types of measurements • Types of presentation • Levels of measurement granularity • Measurement scopes: • Program • Procedure • Loop • Function call • Statement • Code region identification • Object types to be handled within an application

9. Definition and design Work • architecture of the tools, based on their functional description • hierarchy and naming policy of objects to be monitored • the tool/monitor interface, based on the expressing of measurement requests in terms of monitoring specification standard services • the filtering and grouping policy for the tools • functions for handling the measurement requests and the modes of their operation • granularity of measurement representation and visualization modes • the modes of delivering performance data for particular measurements

10. Modes of delivering performance data

11. Interoperability of tools ``Capability to run multiple tools concurrently and apply them to the sameapplication''Motivation:- concurrent use of tools for different tasks- combined use can lead to additional benefits- enhanced modularityProblems:Structural conflicts: due to incompatible monitoringmodulesLogical conflicts: e.g. a tool modifies the state ofan object while another tool still keeps outdated informationabout it

12. Semiautomatic Analysis • Why (semi-)automatic on-line performance evaluation? • ease of use - guide programmers to performance problems • Grid: exact performance characteristics of computing resources and network often unknown to user • tool should assess actual performance w.r.t. achievable performance • interactive applications not well suited for tracing • applications run 'all the time' • detailed trace files would be too large • on-line analysis can focus on specific execution phases • detailed information via selective refinement

13. The APART approach • object oriented performance data model • available performance data • different kinds and sources, e.g. profiles, traces, ... • make use of existing monitoring tools • formal specification of performance properties • possible bottlenecks in an application • specific to programming paradigm • APART specification language (ASL) • specification of automatic analysis process

14. APART specification language • specification of performance property has three parts: • CONDITION: when does a property hold? • CONFIDENCE: how sure are we? (depends on data source) (0-1) • SEVERITY: how important is the property? • basis for determining the most important performance problems • specification can combine different types of performance data • data from different hosts => global properties, e.g. load imbalance • templates for simplified specification of related properties

15. Supporting different performance analysis goals • performance analysis tool may be used to • optimize an application (independent of execution platform) • find out how well it runs on a particular Grid configuration • can be supported via different definitions of SEVERITY • e.g.: communication cost • relative amount of execution time spent for communication • relative amount of available bandwidth used for communication • also provides hints why there is a performance problem (resources not well used vs. resources exhausted)

16. Analytical model for predicting performance on GRID • Extract the relationship between the application and execution features, and the actual execution time. • Focus on the relevant kernels in the applications included in WP1. • Assuming message-passing paradigm (in particular MPI).

17. Taking features into a model • HW features : • Networks speeds • CPU speeds • Memory bandwith • Application features: • Matrix and vector sizes • Number of the required coomunications • Size of these communications • Memory access patterns

18. Building a model • Through statistical analysis, a model to predict the influence of several aspects on the execution of the kernels will be extracted. • Then, a particular model for each aspect will be obtained. A linear combination of them will be used to predict the whole execution time. • Every particular model will be a function of the above features. • Aspects to be included in the model: • computations time as a function of the above features • memory access time as a function of the features • communications time as a function of the features • synchronization time as a function of the features

19. X# WP2.4 Tools w.r.t. DataGrid WP3

20. Summary • New requirements for performance tools in Grid • Adaptation of int. performance ev. tool to GRID • New measurements • New dialogue window • New presentations • New objects • Need in semiautomatic performance analysis • Performance properties • APART specification language • Search strategy • Prediction model construction

21. Performance Measurements with PATOP Possible Types of Measurement: • CPU time • Delay in Remote Procedure Calls (system calls executed on front-end) • Delay in send and receive calls • Amount of data sent and received • Time in marked areas (code regions) • Numer of executions of a specific point in the source code Scope of Measurement System Related: • Whole computing system, • Individual nodes, • Individual threads, • Pairs of nodes (communication partners, for send/receive), • Set of nodes specified by a performance condition Program Related: • Whole program, • Individual functions

22. PATOP

23. Performance evaluation tools on top of the OCM

24. On-line Monitoring Interface Specification The interface should provide the following properties: • support for interoperable tools • efficiency (minimal intrusion, scalability) • support for on-line monitoring (new objects, control) • platform-independence (HW, OS, programming library) • usability for any kind of run-time tool (observing/manipulating, interactive/automatic, centralized/distributed)

25. Object based approach to monitoring • observed system is a hierarchical set of objects: • classes: nodes, processes, threads, messages, and message queues • node/process model suitable for DMPs, NOWs, SMPs, and SMP clusters • access via abstract identifiers (tokens) • services observe and manipulate objects • OMIS core services: platform independent • others: platform (HW, OS, environment) specific extensions • tools define their own view of the observed system

26. Classification of overheads • Synchronisation (e.g. barriers and locks) • coordination of accessing data, maintaining consistency • Control of parallelism (e.g. fork/join operations and loop scheduling) • control and manage parallelism of a program (user, compiler) • Additional computation - changes to sequential code to increase paralellism or data locality • e.g. eliminating data dependences • Loss of parallelism – imperfect parallelisation • un- or partially parallelised code, replicated code • Data movement • any data transfer within a process or between processes

27. Interoperability of PATOP and DETOP • PATOP provides a high-level performance measurement and visualisation • DETOP provides a source-code level debugging • Possible scenarios: • Erroneous behaviour observed via PATOP • Suspend application with DETOP, examine source code • Measurement of execution phases • Start/stop measurement at breakpoint • Measurement on dynamic objects • Start measurement at breakpoint when object is created