Qualitative Reverse Engineering

1. Qualitative Reverse Engineering DorianŠuc and Ivan Bratko Artificial Intelligence Laboratory Faculty of Computer and Information Science University of Ljubljana, Slovenia

2. Overview • Qualitative reverse engineering: • generating explanation of engineering designs • in qualitative terms • Case study: industrial crane controller • regression, model trees • qualitative trees • Discussion

3. Motivation • But: incomplete documentation • Why did designer move from one model to another? • Why was a design abandoned in favor of another? • How a designed system works? • Difficulties in re-use of design libraries: • intuitions behind designs are not explained Accumulated knowledge in companies: libraries of designs and corresponding simulation models

4. Context • Automated explanation generation • knowledge sharing: making tacit knowledge available to other people • model re-use: for model to be used by another designer,generate explanation how model works • Clockwork European project: • computer aided engineering design • industrial partners: car and heating industry • libraries of simulation models

5. The problem • Given designed system • Explain: • how the system works? • ideas, intuitions behind the design • Approach: • use ML on traces of system’s behavior • similar to behavioural cloning (Michie et al., 90; Sammut et al., 92)

6. Case studies • Reverse engineering of industrial crane controller, presented in this paper • Reverse engineering of car suspension system

7. Case study: anti-sway crane For transportation of slabs Designed by Czech Technical Univ. In daily use Crane parameters: travel 100m height 15m, width 30m 80-120 tons

8. Anti-sway crane Control force: F State: cX,cV, lX,lV ? Anti-sway controller • control force F • achieves cVDes • minimizes swing desired carriage velocity cVDes

9. Anti-sway crane

10. Matlab model of crane controller

11. Learning problem controller = ? Given simulation model + controller explain how controller works

12. Learning problem

13. Learning problem cVDesF cX cV lX lV 2.0 0 0.0 0.0 0.0 0.0 2.0 24914 0.10.3 0.10.3 2.0 40188 0.4 2.1 0.4 2.1 2.0 -425 1.6 3.1 1.5 2.8 … ….. … … … … ML

14. ML methods used • Linear regression • Regression, model trees • Qualitative trees (QUIN)

15. QUalitative INduction (QUIN) • Learning of qualitative trees • Qualitative tree defines a partition of the state space into areas with common qualitative behavior • Qualitatively constrained functions (QCFs) in leaves: • generalization of monotonicity constraint M+ • similar to qualitative proportionality predicates (QPT-Forbus, 84)

16. Example problem for QUIN • Noisy examples generated from z=x2-y2

17. Qualitative patterns in data x > 0 & y > 0 => z = M+,-(x,y)

18. Induced qualitative tree for functionz=x2-y2 z is monotonically increasing in its dependence on x and monotonically decreasing in its dependence on y z is positively related to x and negatively related to y

19. QUIN algorithm • MDL to learn “most consistent” QCF • Learning qualitative tree (similar to ID3) • for every possible split, split the examples into two subsets, find the “most consistent” QCF for both subsets and select the split minimizing tree-error cost (based on MDL) • QUIN is heuristic improvement of this idea

20. Related work in learning qualitative models • Previous work includes: • Mozetič, 87 (KARDIO), • Coiera, 89; Hau, Coiera, 97 • Bratko, Muggleton, Varšek, 91 • Richards, Kraan, Kuipers, 92 • Say, Kuru, 96 • Kay, Rinner, Kuipers, 2000 • Most of these systems induce QDE models, hard to apply to our problem

21. Reverse engineering of anti-sway crane controller ML from examples of controller’s execution: • Linear regression, regression and model trees (Weka-Witten&Frank,2000; M5-Quinlan, 92) • QUIN

22. Reverse engineering of anti-sway crane controller ML from examples of controller’s execution: • Linear regression, regression and model trees (Weka-Witten&Frank,2000; M5-Quinlan, 92) • QUIN • Given:cVDes=2 • an execution tracecVDes=0 • of anti-sway controller (follow cVDes,minimize swing) • learn control force F = F(lX, lV, cV, cVDes)

23. Evaluation criteria Comparing the induced models: • suitability as explanation • control performance • criterion of controller’s success: mean abs. velocity error and mean abs. swing error bellow errors of anti-sway controller + 10%

24. Regression and model trees Learning F=F (lX, lV, cV, cVDes): • pruning: smallest non-single leaf tree has 34 leaves • control performance: • cannot work: cVDes does not appear in any of induced trees • Helping the learning system: • derived attribute: relative carriage velocity cVRel=cV-cVDes • pruning: smallest non-single leaf tree has 17 leaves • control performance: • does not move(F0 when lX0), or huge swing • Summary: • complex trees, not even close to achieving the task • not suitable as explanation

25. Linear regression Learning F=F (lX, lV, cV, cVDes): • F= 23218 cVDes -18578 cV -20319 lX + 12410 lV -4800 • bad fit (R=0.73, RRE=69.12%) • unsatisfactory control performance • Helping the learning system (using cVRel): • Similar, even slightly worse results • Summary: • simple models • but unsatisfactory control performance and bad fit

26. Experiments with QUIN Using cVRel: accelerate yes no F=M-(lX), lX decreas. => increase F Small swing => follow cVDes (since cVRel=cV-cVDes) Large swing => reduce swing

27. Experiments with QUIN, cont. F = F(lX, lV, cV, cVDes): yes no Equivalent to previous qualitative tree! (given the particular crane dynamics)

28. Control with qualititative trees • By transforming QCFs in leaves into real-valued functions satisfying qualitative constraints M+(X) Randomly generated real-valued increasing functions f+(X) Arbitrary increasing fn. of X • Testing 100 transformed qual. trees for control: • 43 successful (according to criterion of success)

29. Anti-sway clone

30. Reverse engineering ofcar suspension system • Given Quarter Car Model, reverse engineer controller for semi-active car suspension • Matlab/Simulink model provided by CTU and Intec

31. Car suspension model The original controller Black Box The shock absorber characteristics

32. Reverse engineering of car suspension system • Qualitative controller was induced with QUIN • Induced controller applied to control car model • Performance compared with original controller

33. Performance comparison Original controller: Qualitative controller: Comparatively better Comparatively worse

34. Summary of approach • Reverse engineering in qualitative terms, useful to grasp the intuition behind the design • Induction of qualitative trees (QUIN) • Qualitative-to-quantitative transformation of qualitative control strategies (involves optimisation)

35. Qualitative vs.quantitative models • Induced qualitative models superior as explanations (unsurprisingly) • Induced qualitative controllers* also perform better (surprisingly!) *“Qualitative controller” means: qualitative control strategy induced by QUIN, applied as controller after qualitative-to-quantitative transformation

36. Why numerical learning fails? 1. Linear regression alone insufficient - easy to see 2. But, why model trees also fail? 3. Why qualitative trees succeed when model trees fail?

37. Example from crane • M5 model tree contains leaf - 0.0055 < lX =< 0.0055 Corresponding example set is very awkward for linear regression, see visualisation • QUIN finds better split: - 0.223 < lX =< 0.163 This example set exhibits clearer qualitative pattern

38. Example from crane, cont.

39. Example from crane, cont.

40. Tentative explanation • QUIN can detect subtle qualitative patterns, although they are obscured by large quantitative changes • Regression tree learning (M5) does not find good splits, resulting in regression-unfriendly leaves

41. Comparison Qualitative trees more successful: • suitability as explanation • control performance Why regression trees fail? • Important patterns overshadowed by numerically dominant attributes • cVDes does not appear in regression trees! • Still qualitatively visible