Accurate Measurement-Based WCET Analysis in the Absence of Source and Binary Code

1. Accurate Measurement-Based WCET Analysis in the Absence of Source and Binary Code presented by Thomas Leveque on behalf of Amine Marref and Adam Betts

2. Introduction • Real-time systems (RTSs) must be both functionally and temporally correct. • The timeliness of a RTS must be proved. • This is achieved via a schedulability analysis. • The schedulability analysis takes as input the periods, deadlines, and worst-case execution times (WCETs) of the tasks in the RTS and decides whether or not the RTS can be scheduled. • The WCET of a task is the longest time that the task will ever exhibit on a particular hardware platform.

3. Introduction • Finding the WCET of a program is the task of WCET analysis. • WCET analysis reduces to the Halting problem and hence the program whose WCET is of interest must have bounded behaviour i.e. loop and recursion upper bounds must exist. • Therefore, the WCET analysis estimates the WCET as opposed to find the WCET. • WCET estimations must be safe (absolutely for hard RTSs, or with appropriate levels of confidence for soft RTSs), and should be tight.

4. Introduction • WCET analysis performs the following steps. • Divides program into units (e.g. basic blocks or bigger segments) and choose structural representation i.e. control-flow graph (CFG) or abstract-syntax tree. • Performs program-flow analysis e.g. find infeasible paths. • Performs processor-behaviour analysis by which a timing-model is derived. • Static modelling of hardware accelerators e.g. pipelines, caches, branch predictors, etc. • Direct measurements on hardware. • Performs calculation that puts together program-flow information and hardware-timing information to produce a WCET estimate.

5. Motivation • WCET analysis requires availability of source/binary code in order to estimate WCET. • A stiff competition between RTS developers means that companies are less willing to give away code for WCET analysis. • They must protect their intellectual property. • How do we estimate the WCET of a program that we do not have? • We require the owner of the program to give us traces of the execution of the program that do not reveal the semantics of the program. • The program-execution traces are used to reconstruct the CFG of the program and devise time constraints used to estimate the WCET in a measurement-based setting.

6. Framework • The task is split between owner of program and us: • Program owner: • Performs test-data generation. • Collects execution traces and format them appropriately. • Gives us the program-execution traces. • Us: • Perform the WCET analysis based on the program-execution traces. • Or: • We could obtain the binary from the program owner, do their work above and never need to reverse-engineer the binary program.

7. Program-Execution Traces • A program-execution trace is a sequence of pairs where each pair is a program-unit identifier and a timestamp. • A program unit can be a basic block, and its identifier is its memory address. • A program-unit identifier must be unique. • When a program unit executes, its memory address and corresponding time are written to the trace. • At the end of trace generation, we obtain a list of program-execution traces. • A program trace does not reveal any information about the semantics of the program under analysis.

8. Test-Data Generation • Test-data generation yields a set of input vectors. • Each input vector executes a program path. • The execution of a program path yields a program-execution trace. • The program-unit identifiers along a single trace represent an execution path. • The set of the program identifiers observed during testing represent the set of nodes of the CFG of the program. • We call it the observed CFG or oG for short. • Each pair of subsequent unit identifiers represents a potential edge in the CFG.

9. Test-Data Generation - Problems • What could go wrong in test-data generation? • Only few paths are exercised during testing. • Structurally-long paths are missing. • Only small execution times are triggered. • Timing-long paths are not covered. • What do we need to ensure? • The full CFG of the program is visited by testing. • Long execution times are exercised.

10. Test-Data Generation – Problems - Inexact oG • The oG might not be identified correctly: • If one of the program units has never been exercised by test-data generation, it will go missing from the oG. • This means that the oG has less nodes than the real CFG of the program. • This causes potential optimism e.g. the missing unit has a large execution time. • The solution is to perform branch-coverage as part of the test-data generation process to ensure all edges and nodes of the program are exercised.

11. Test-Data Generation – Problems - Inexact oG • The oG might not be identified correctly: • If the hardware allows speculative execution, an identifier might be generated for a wrongly-fetched program unit which will be discarded and never executed in the current path. • This means that the oG has two extra edges going into and out of the wrongly-fetched unit. • This causes potential pessimism as this particular execution path of the CFG is augmented by an extra node. • The solution is to ensure that the unit identifiers are written to trace only when their corresponding units commit execution in the pipeline. • Nexus 5001 hardware-debug interface can be used to generate the traces we want and has a self-correcting mechanism which basically discards non-committing program units from the trace.

12. Test-Data generation – Problems – Small Execution Times • If the observed execution times are small, we could underestimate the WCET. • A general problem for measurement-based analysis. • A partial solution is to treat the test-data generation problem as an optimisation problem. • Use evolutionary testing to guide test-data generation towards producing long end-to-end execution times.

13. Deriving Time Constraints • From the traces, we collect the following information: • For each program unit ui, we derive all its execution times cih and how many times xih each execution timecihis observed. • Useful e.g. in separating the execution times of units ui when they miss/hit the cache. • These constraints are linear. • For each program unit ui, derive the set of executing units uj that must execute in order to observe a particular execution time cih. • Reduces to intersecting execution traces together to derive the commonly executing units ujto observe execution time cih. • Enforces more context-sensitivity. • Costly, so we compare the execution traces within a small window and we do it outside loops. • The approach is heuristic, the derived set Ajh of units uj that are common in all the windowed traces where cihis observed, is not guaranteed to be the exact set – it can be a subset or a superset. • These constraints are expressed as implications.

14. Calculating the WCET • We use the implicit-path enumeration technique (IPET). • Because alternatives i.e. path-based and tree-based are either too expensive or too inexpressive. • In addition to the usual structural and flow-preservation constraints, we add the derived time constraints.

15. Evaluation • We compute a WCET estimation in three ways: • Using exhaustive path coverage. • To find the real WCET. • Input space of modified Malardalen benchmark programs has been restricted to make this possible. • Using static analysis. • A combination of Chronos (processor-behaviour analysis) and Sweet (program-flow analysis). • Our own analysis.

16. Evaluation • Our own analysis. • Test-data generation using branch coverage (using Crest) and genetic algorithms. • Trace generation using the cycle-accurate simulator SimpleScalar. • Timestamps generated at the commit stages. • Provides Nexus-like traces. • Is also used by Chronos to derive timing models, so the estimations by static analysis and our own method can be compared. • Collected constraints are added to an IPET model. • Using integer-linear programming. • Only linear constraints can be added. • The model is solved using lp_solve. • Using constraint-logic programming. • In addition to the linear constraints, the implication-based constraints are also added. • The model is solved using eclipse.

17. Results • The estimates using our method are safe. • By comparing to the real WCET. • The estimates are tight. • They are tighter than those provided by static analysis. • Pessimism never exceeded 20%. • Adding the cause/effect constraints always produced further tightness.

18. Conclusions • The presented analysis is solely based on execution traces that do not reveal any program semantics. • The presented method provides safe and tight WCET estimations for the evaluated benchmarks. • It is not possible to argue about safety/tightness of the approach in the general case. • The quality of testing will affect the quality of the derived WCET estimates. • Requires research into WCET-coverage testing techniques.