A Binary Agent Technology for COTS Software Integrity

1. A Binary Agent Technology for COTS Software Integrity Anant Agarwal Richard Schooler InCert Software

2. Operating System Input COTS Binary COTS Binary SAP Output The Mission Critical Environment The development environment The deployment environment

3. COTS Binary Objective Operating System Input To improve the integrity of the deployment environment with COTS software in the presence of attacks, bugs SAP Output

4. COTS Binary Assumptions and Scope Operating System Outer security defenses will be breached by attackers Use a practical, systems level approach – execution-time monitoring On COTS program or data corruption, rapidly d-detect problems a- trigger an alarm p- try to protect r- recover Input SAP Output

5. COTS COTS COTS New New New Legacy Legacy Legacy Missing source Missing source Missing source Our Approach: Execution-Time Monitoring of COTS through Binary Instrumentation d- Policy specs for detection d- Heartbeat insertion d- Argument range checks d- Rare code execution/sigs. a- Alarm messages to console p- Defaults for fault tolerance p- Access constraints, redund. r- Logging COTS Binary The development environment The deployment environment

6. Drawbacks of Binary Insertion • Specific to a single platform, needs new technology development for different platform • Challenging to relate low-level observable events back to high-level user actions • hard to detect some types of intrusions that only affect data corruption • hard to protect or correct problems at higher semantic levels

7. Three Major Components in the Prototype,Three Major Tasks • Core technology for customizable agent insertion into PC/NT • Anomaly detection and reporting • Rapid recovery and problem pinpointing

8. Selected Risks/Challenges and Mitigation • Core technology for agent insertion into binary • Dealing with real environments – e.g., multithreading and synchronization, in particular, time syncing and monitoring events in a distributed environment • How to minimize runtime overhead – borrow compiler optimization techniques (e.g., steal registers, in ine code, sampling, multilevel checks) • How to deal with unknown relocations, e.g., for dusty decks – incremental control and dataflow analysis; an integrated static and dynamic method • Anomaly detection – can we catch problems without user help? • Runtime comparison against execution path signatures? • State machines for control flow checks (e.g., Abraham) • Rapid recovery and problem pinpointing technology • Third party problems • Can we get data values? Use dataflow analysis and offline simulation to obtain intermediate data values

9. Measures of Success • Core technology for agent insertion into binary: • Can we handle all binaries, DLLs, even dusty decks? • Target: Performance degradation to be under 1 percent • Anomaly detection • What fraction of injected problems can we detect • Rapid recovery technology • Can we cut recovery time significantly? We will measure recovery time with and without • As a bonus, can we catch problems before system goes down? • Build a prototype system, work with real users, and measure

10. T2 T1 T3 DLL1 DLL2 Realistic Environments HaveMultiple Threads and Modules

11. T2 T1 T3 TS DLL1 TS DLL2 Multiple Threads – Per-DLL Buffer Lock overhead Contention in SMPs TS Thread IDs DLL1 Buffer DLL2 Buffer

12. T2 T1 s T3 s1 s2 DLL1 s4 s5 s3 DLL2 Multiple Threads – Per Thread Buffer Timestamps Sequence counter Ids

13. Multiple Machines? T2, My T1, Mx T3, Mz s s1 s2 s4 DLL s3 How to synchronize efficiently times at a fine grain? How to maintain efficiently a cross-machine counter?

14. Current Progress • Work on NT binary insertion prototype ongoing • Demo of early capability showing • instrumentation • simple recovery log • detecting application has crashed • taking control and • writing out log • user-requested snap-trace for hung or “molasses” mode • information viewer for multithreaded traces • some optimization • Handling multithreading, DLLs imminent – prototyped • needed significant changes to runtime system – leverage shared memory • ongoing thinking on distributed programs • Ongoing thinking on detection capability

15. Summary • A systems approach to COTS Integrity • Approach based on execution-time monitoring using binary insertion • We have an early prototype version of NT binary insertion implemented • We have also successfully instrumented multithreaded programs