Program Obfuscation: A Quantitative Approach

1. Program Obfuscation:A Quantitative Approach • Bertrand Anckaert, MatiasMadou, Bjorn De Sutter, Bruno De Bus,KoenDe Bosschere, and Bart Preneel • Ghent University and K.U.Leuven, Belgium Presented by: MariuszJakubowski Microsoft Research Third Workshop on Quality of Protection October 29th, 2007

2. 0101110 00111001010 00101011001000110001110110010111011011001011101010110100010110111111110001010110110011111001010111001110010111 1 11111111111111110 Obfuscation has many applications

3. There is a large gap between theoretical results + - Positive Results and Techniques for Obfuscation – Lynn et al. (2004) • Towards Realizing Random Oracles: Hash Functions that Hide All Partial Information • Canetti et al. (1997) Large gap Intuitively, obfuscation does help - - On the Impossibility of Obfuscation with Auxiliary Input – Goldwasser et al. (2005) - On the (Im)possibility of Obfuscating Programs – Barak et al. (2001)

4. We need a practical system for evaluating obfuscating transformations • It should be easy to evaluate existing and future transformations => Automated • The evaluation should convey difficulty of reverse-engineering => Build upon experience from complexity metrics

5. Outline • Intro • Metrics • Instruction Count • Cyclomatic Number • Knot Count • (De)Obfuscating transformations

6. Four axes based on typical reverse-engineering scenario Disassemble Flow graph construction Control flow Data flow Code Analyse Data Flow Data Interpret Data

7. Evaluated Complexity Metrics Cyclomatic Number Knot Count Control flow Instruction Count Data flow Code Data Metrics are collected by a run-time instrumentation framework • + No uncertainty about executed code • + Always availabe • - Only about covered part of the code

8. Cyclomatic number and knot count • Cyclomatic number: • #edges – #nodes + 2 • Intuitively: the number of decision points • Knot count: • #crossings • Intuitively: the unstructuredness

9. Outline • Intro • Metrics • (De)Obfuscating transformations • Jump redirection [Linn et al. 2003] • Control flow flattening [Chenxi Wang et al. 2001] • Opaque predicates [Collberg et al. 1998]

10. Jump redirection • Redirect branches to function 1 call branch 1 assumed return site garbage Branch Function Jmp 2 2 2

11. Impact of Jump Redirection

12. Jump redirection - deobfuscation • Identify Branch Function • signature based • run-time behavior • Record (call,return) pairs under debugger • Overwrite calls (1,2) (4,7) (9,5) … 1 call branch jmp 2 assumed return site garbage Branch Function 2

13. Success of De-obfuscation

14. Control flow flattening All original basic blocks have the same predecessor and successor 1 switch 2 3 3 4 1 2 4

15. Control flow flattening significantly increases the complexity metrics

16. Success of De-obfuscation

17. Opaque predicates • Add fake decision statements 1 1 Jmp 2 Jmp if (2==2) 2 fake 2

18. Impact of Opaque Predication

19. Conclusion • A first step towards a unified quantitative evaluation of • obfuscating transformations • deobfuscating transformations • Which leverages experience from the established field of complexity metrics

20. Program Obfuscation:A Quantitative Approach • Bertrand Anckaert, MatiasMadou, Bjorn De Sutter, Bruno De Bus,KoenDe Bosschere, and Bart Preneel • Ghent University and K.U.Leuven, Belgium Presented by: MariuszJakubowski Microsoft Research Third Workshop on Quality of Protection October 29th, 2007