Building Defensive Architectures Using Backdoors

1. Building Defensive Architectures Using Backdoors Liviu Iftode Department of Computer Science Rutgers University

3. Frustration Scalability Service.com

4. Internet Attacks Failure 11:00am JST 9:00pm EST 2:00am GMT Planetary-Scale Services • Human operators, phone calls and emails hard to scale • Cost of ownership dramatically exceeds cost of systems

5. Internet Attacks Failure BD BD BD BD BD BD BD BD BD Gateway Gateway Gateway Private Network 9:00pm EST 11:00am JST 2:00am GMT The Dream: A Defensive Architecture

6. Possible Healing Actions • Refresh the state (reboot) • Destructive and Disruptive • Repair the state (continue) • Recover the state (transfer) How to access the memory of the failed system when the OS is “hung”?

7. The Motivating Philosophy • Something is better than nothing • Save application state if possible • Faster is better than slower • Repairing state faster than repairing software • It is hard to corrupt or stop an outsider • Remote healing better than self-healing • Attackers and faults are becoming “smarter” • Try “holistic” approach if nothing else

8. The Backdoor (BD) Backdoor: a hidden software or hardware mechanism, usually created for testing and troubleshooting --American National Standard for Telecommunications

9. Backdoor Design Principles 1. Availability • BD must be highly available (even when OS is not) 2. Non-intrusiveness • BD operations must not involve local OS (zero-overhead monitoring) 3. Integrity • OS cannot alter BD execution or modify the result of a BD operation 4. Responsiveness • A BD operation cannot be delayed indefinitely

10. Possible Backdoor Implementations • A programmable network interface (I-NIC) • Our current prototype is on Myrinet • A virtual machine over a VMM • Work in progress over Xen • IBM’s Remote Supervisor Adapter? • HP’s Remote Management Adapter?

11. Backdoor as building block • Remote Healing Systems • A computer system monitors/repairs/recovers the state of a remote system through the backdoor • Backdoor is controlled by the remote OS • Defensive Architectures • Backdoors are programmed to execute defensive tasks, stand-alone or cooperatively over a private network • Standalone backdoor

12. Outline • Introduction • Backdoor Idea • Remote Healing • Defensive Architectures • Conclusions

13. Remote Healing • Backdoor prototyped on I-NIC (Myrinet) • Remote Repair of OS State • Remote Recovery for Cluster-Based Internet Servers

14. “Front door” CPU Mem NIC I-NIC Backdoor Private Network Backdoor on I-NIC • Backdoor provides an alternative access to system memory without involving local CPU/OS • Private network over a specialized interconnect, VPN, or even over a phone link!

15. Target System Monitor System CPU CPU BD BD Mem Mem I/O I/O A Remote Healing Architecture

16. Backdoors use Remote Memory Communication Target Monitor MONITOR (Remote-Read) Memory Memory CPU CPU Recovery/Repair (Remote-Read/Write) BD BD NIC CPU

17. Remote OS Locking • Implemented by a BD-OS protocol • Two functions • Provides exclusive access to target OS data for state repairing • Enforces fail-stop model in the recovery case to avoid the consequences of false positives in failure detection • Can be avoided? • Yes for monitoring

18. OS Support for Remote Healing • Monitoring and Failure Detection • Sensor Box: system health indicators (sensors) provided by the target OS in its local memory • Sensors: <UniqueID, Type, Threshold , Value> • Repairing • Externalized State: OS state data that the BD can read • Remote Access Hooks: OS control data that the BD can write to perform repairing actions • Recovery • Continuation Box: fine-grain OS and application checkpoint state that the BD can transfer between systems to migrate running applications

19. Sensor Box (SB) • Collection of health indicators (sensors) in the target OS memory • <ID, Type, Threshold, Value>

20. Failure Detection using Sensor Box • Target OS updates progress sensors in SB continuously • Monitoring thread reads SB periodically and checks counters • Failure = counter stalled beyond its deadline • False positive rate vs. detection latency tradeoff Target OS Monitor Sensor Box <Timer interrupts> <Context switches> <NIC interrupts> … Backdoor

21. Remote view Sensor Box Detection Monitoring and Detection Using BD Mem Mem CPU CPU BD BD

22. Diagnosis and Repairig • Diagnosis • Inspect live OS data structures in target’s memory (through the externalized state) • Identify damaged OS state (e.g. resource exhaustion due to memory hogging processes) • Repairing • Modify target OS memory (through remote access hooks) to correct damaged state (e.g. remove memory hogging processes by “injecting” a kill signal in its process control block)

23. Externalized state Fine grained view Diagnosis Diagnosis Using BD Mem Mem CPU CPU BD BD

24. Correct state Repair Hook Repair Repair Using BD Mem Mem CPU CPU BD BD

25. Case Study: Repairing OS State • Damaged OS state : resource exhaustion, corrupted data structures, compromised OS, etc. • Resource exhaustion • Attack, overload, system misconfiguration, programming error • Repairing cannot rely on local resources • Two examples • Fork bomb • Memory hog

26. Case Study : Memory Hog • Program allocates memory in an infinite loop • Both memory and swap space are occupied by the memory hog • System is inaccessible from console or the network • Cannot spawn new processes • Cannot handle interrupts • Local daemons cannot repair system

27. Remote Repairing in case of Memory Hogging • Monitoring • Pressure sensor signals when severe low memory condition is detected • Diagnosis • Target externalizes process table and process memory usage statistics • Monitoring thread identifies the culprit • Repairing • Monitoring thread kills culprit by remotely posting a SIGKILL

28. Prototype • BD implemented on Myrinet LanaiX NIC • Modified firmware and low level GM library • Modified FreeBSD 4.8 kernel • Experimental setup • Dell Poweredge 2600 servers with 2.4 GHz dual Intel Xeon, 1GB RAM, 2GB swap, Myrinet Lanai X NIC • Benchmark: simple counting program with fixed number of iterations

29. Effectiveness of Remote Repairing

30. Repairing Timeline

31. Remote Healing • Backdoor prototype using Myrinet • Remote Repair of OS State • Remote Recovery for Cluster-based Internet Servers

32. Clusters with BD Network M I/O P M I/O P T M BD BD Interconnect M T M T BD BD T M M I/O P M I/O P

33. Cluster-based Internet Services with BD network Client Client Client Server Server Server Monitor Monitor Monitor

34. Cluster-based Internet Services with BD network Client Client Client Server Server Server Monitor Monitor Monitor

35. Continuation Box (CB) • Idea • Define per client-session state (OS and application) • Transfer client sessions from the failed system to other systems in the cluster running the same server application • CB encapsulates the state of a client session associated with a server application (possibly multi-process) • OS state (data in transit through IPC channels) • application-specific state (periodically exported/checkpointed by the application)

36. Recovered State Continuation Box Extraction Continuation Box Memory Memory CPU OS BD BD Recovery machine (healthy) Victim machine (crashed)

37. Client-Session Continuation Boxfor Multi-Process Servers App. state Comm. state Process 1 Process 2 Client 1 TCP/IP IPC CB1 Client 2 CB2

38. Continuation Box API • create_cb for a client session • export application state to CB • associate I/O channel with the CB • open_cb given an I/O channel • import application state from CB

39. Changes to make Server Recoverable while (cid = accept()) { cbid = create_cb(cid) if (import(cbid, &{file_name, offset}) == NULL) { receive(cid, file_name) offset = 0 } fd=open(file_name) seek(fd, offset) while (read(fd, block, size) != EOF) { send(cid, block, size) offset += size export(cbid, {file_name, offset}) } }

40. State Synchronization Problem • Application state (SB_APP) updated only upon export • OS state (SB_IO) updated continuously by the OS kernel • How to synchronize the two components of the CB? Application Application Application A1 A2 A1 RECV export import OS OS OS SB_IO SB_APP SB_IO SB_APP SB_IO SB_APP 3 2 A1 3 2 A1 3 A1 SB SB SB

41. CB-based Recovery • Log-based rollback recovery • restores server state with respect to a client • OS keeps communication logs (send/receive) • 0-copy using the communication buffers • After migration, OS replays send/receive operations from logs • transparent to server and client applications

42. Backdoors Prototype • Myrinet LanaiX NIC as backdoor • in-kernel remote read/write operations • Modified FreeBSD kernel • Sensor Box, Continuation Box • Modified server applications • Apache, Flash, Icecast, JBoss

43. Case Study: A Multi-tier Auction Service Front-End (FE) Apache web server Middle Tier (MT) JBoss app. server Back-End MySQL DB server

44. Recoverable RUBiS

45. Experimental Evaluation • Experimental setup • Dell PowerEdge 2600 servers, 2.4 GHz dual Intel Xeon, 1GB RAM, 1Gb Ethernet • Workload modeled after TPC-W • Fault injection in FE and MT nodes • synthetic freeze, emulated freeze by remote OS locking, bugs inserted in network drivers • Evaluation • Low overhead under load • Recovery is fast

46. Low Overhead under Load

47. Recovery is Fast

48. Outline • Introduction • Backdoor Idea • Remote Healing Experience • Defensive Architectures • Conclusions

49. Autonomous Backdoor BD is programmed to execute defensive tasks, then “sealed”

50. Defensive Architecture Hierarchy • Defensive Computer Architecture (DCA) • Individual computers equipped with BD • BD performs local defensive tasks (e.g. OS state inspection) • Defensive Network Architecture (DNA) • Cluster nodes equipped with BDs connected over high-speed private network • BDs perform defensive tasks cooperatively (e.g. OS integrity checking, continuous remote logging) • Defensive Inter-Network Architectures (DINA) • Loosely coupled DNAs connected over the Internet or other networks • DNA cooperate (e.g. early warnings of virus attacks)