Immunix & Defcon: Defending Vulnerable Code From Intense Attack

1. Immunix & Defcon:Defending Vulnerable Code From Intense Attack Crispin Cowan, Ph.D Seth Arnold, Steve Beattie, Chris Wright WireX and John Viega, Secure Software

2. Talk Outline • About WireX and Immunix • Secure Linux Systems • The Defcon Challenge • Defend vulnerable code against massive attack • Technology Transfer • Commercial products built on this technology

3. Software Security Software security is really simple: • Make sure you only run perfect software Uh-oh :-) Intrusion Prevention: • Systems that detect attack attempts in real time, and reject them • When bugs occur, they are not exploitable • Attacker cannot exploit the bug to gain unintended privileges

4. Immunix Security Technologies • Shipping: StackGuard: stops buffer overflows FormatGuard: stops printf format bugs RaceGuard: stops temp file races SubDomain: contains vulnerable programs

5. Stack Smashing Problem: Weak bounds checking on inputs in C programs Attacker overflows input buffer, corrupting adjacent state to gain control of the program Most common target: function return address on the stack StackGuard: C compiler enhancement Ornaments call stack to detect corruption Very low performance overhead WireX has been shipping fully StackGuard’d system since 1999 StackGuard

6. Format String Problem: Sudden discovery in June 2000 Vulnerability in WU-FTPD Followed by hundreds of similar vulnerabilities Basis: arcane %n printf format string directive Treat corresponding argument as an int * Write back number of items formatted so far Problem: programs that pass un-filtered user input strings direct to printf FormatGuard: Similar to StackGuard compiled defense against printf format string vulnerabilities CPP macro Counts arguments at the call site at compile time Compares that number to the format string presented at run time FormatGuard

7. Temporary File Race Problem: Portable procedure for temporary file creation is non-atomic If attacker gets in the middle, can redirect temporary file creation by privileged programs to corrupt the system RaceGuard: Kernel enhancement to detect race attacks mid-way through Abstract method: detect changes between stat() and open() accesses to the same file name RaceGuard

8. Containment • If your software is vulnerable anyway, you need to contain it so that it runs with the least privilege necessary to perform designated function Chroot: basic isolation for vulnerable programs Immunix SubDomain: flexible confinement for vulnerable programs

9. “Change root”: makes some subdirectory appear to be the root (“/”) directory for the calling process and its children Available as both a shell command and a system call Effect: chroot’d programs cannot affect anything outside the chroot “jail” Limits impact of bugs in program, e.g. chroot BIND Benefits: Standard: Comes with most UNIX’s Compatible: several current programs have been modified to work within a chroot jail Fast:no performance degradation Limitations: Work: must move copies of everything a jailed program needs into the jail Isolation: jailed program cannot interact at all with the rest of the system Containment WithChroot

10. Part of Immunix Kernel Extension: Specify the list of files that a SubDomained program may access Effect: SubDomained programs cannot affect anything they don’t explicitly need access to Limits impact of bugs in program, e.g. SubDomain CGI scripts Benefits: Flexible: SubDomained programs can have controlled interaction with the rest of the system Compatible: SubDomain can confine binary programs without modifications Fast: 1% or less performance overhead Limitations: Work: must specify “shape” of SubDomain Containment WithImmunix SubDomain

11. PHF: infamous vulnerable CGI script legitimate function: database lookup of user information sloppy parsing of CGI input can get PHF to start an xterm on an arbitrary display To SubDomain PHF: Specify all the files that PHF needs Effect: access to all other files is denied Including xterm :-) Place this file in /etc/subdomain.conf/phf /home/httpd/cgi-bin/phf { /bin/sh x , /etc/ld.so.cache r , /etc/nsswitch.conf r , /lib/ld-linux.so.2 r , /lib/libc.so.6 r , /lib/libtermcap.so.2 r , /usr/local/bin/ph ix , } Containing PHF

12. WireX Systems:Immunix Secure OS • Linux system similar to Red Hat Linux • RPM based • All source-available programs compiled with StackGuard and FormatGuard • PointGuard in future releases • Kernel equipped with SubDomain and RaceGuard • All network-accessible daemons SubDomain-profiled

13. Experimentation ... • Some real-world red teaming • Play an Immunix server in the Defcon Capture the Flag (CtF) games • Almost no holds barred: • No flooding • No physical attacks • New gaming rig designed by the Ghettohackers

14. Basic Defcon CtF Rules Player Nodes

15. Basic Defcon CtF Rules If all services found ... Score’bot Polls player nodes, Looking for req. services Player Nodes

16. Basic Defcon CtF Rules If all services found, Score one point for the Flag currently on that node Score’bot Polls player nodes, Looking for req. services Player Nodes

17. Basic Defcon CtF Rules If all services found, Score one point for the Flag currently on that node Score’bot Polls player nodes, Looking for req. services … while each team tries to replace others’ flags Player Nodes

18. No Flooding • DoS attacks are not interesting • Explicit rule against flooding attacks • Game masters will make you stop if you are caught at it • Goal: ensure that all teams are actually able to play • Penalties: • Kicked out for overt DoS attacks • Pay for bandwidth with a point penalty

19. Area View

20. Sporting Event Immunix was white, hence “Weiss Labs” • There was an official bookie :-) • Score broadcast on hotel cable Score obfuscated Teams named funky colors

21. The Catch • The required services are secret • Only a few clues: • They supply us with a VMWare/Linux image reference distribution that provides all required services • It is also riddled with vulnerabilities • The score’bot polls for the required services • But the score’bot stops its poll if it finds something it doesn’t like

22. The Reference Distribution • Red Hat 6.2, unpatched • nmap: shows nearly everything open • finger, POP, IMAP, SMTP, SNMP, Webmin ... • Apache running as root • CGI’s for adduser and deleteuser • Anonymous can create a user login on your node • As any user number, including zero

23. Example Services the Score’bot Wanted • Create a user • Send that user mail • Finger the user • POP in to fetch the mail • Delete the user • Note: no crypto protocols • No proper authentication of the score’bot • Must heuristically distinguish score’bot from attacks using behavior signatures

24. Interesting Challenge • Not just survive severe attack, but also • Protect bad code • A lot of it • Vague functional specification • Rapid deployment • Great new game infrastructure from Ghettohackers • Interesting challenge • Engaging scoreboard

25. Captain’s Meeting • Explain the rules in detail • Hand us the reference distribution

26. Setting Up

27. The Popular Strategy: Human Intrusion Detection • Launch the reference Linux distribution • Ad hoc patch as stuff happens • Defend: • look for logins, I.e. non-score’bot behavior • kill them off ASAP • very labor-intensive

28. The Immunix Strategy: Protect Bad Code with Immunix Tools • Port all plausible services to Immunix 7+ distribution • Use our own fingerd, httpd, etc., up-to-date and compiled with StackGuard and FormatGuard • Run on an Immunix kernel with SubDomain and RaceGuard • Wrap vulnerable services & CGI’s with SubDomain profiles to limit access to least privilege necessary • Launch only when we were reasonably confident that the Immunix machine was configured securely

29. Dealing with Logins: the SubDomain Shim • Change adduser CGI to use a special default shell: /bin/fubush • /bin/fubush is just a hard link to /bin/bash • Restrict /bin/fubush to only the operations needed by the score’bot • Attackers can go ahead and create a login with uid 0 and it still won’t do them any good • They get a root shell, stuck in a tiny sandbox

30. Immunix Team

31. Immunix Team Me Seth Arnold Steve Beattie Chris Wright • Plus 15 volunteers

32. From Our Corner

33. From Our Corner John Viega Me Steve Beattie Seth Arnold Chris Wright

34. Mental Stress • This is a tough game to play • Head-to-head competition with a lot of very smart people • Real-time, continuous • The intensity of qualifying exams • That go on for 22 hours in a 48 hour period • … set in the middle of a rave • Hydrate or die :-)

35. Rave • Loud music • Smoking • Gawkers • Social engineering • Periodic “news breaks”

36. What We Did For first 4 hours No server at all Porting services to Immunix ASAP, based largely on nmap and source inspection Next 4 hours Launch Immunix server It’s secure, but is not making the score’bot happy Cost us massive points Too focused on the science of “can we defend Immunix?” and not enough on the game rules What We Should Have Done Launch reference system immediately Defend ad hoc like everyone else Run network sniffer to determine what the score’bot wants Would have: Put us over the top on points Learned what score’bot wants much faster We eventually did this Our Strategic Error

37. Immunix Server Not Up Yet 6th place

38. Once Immunix Server Up …in the Score’bot’s Opinion :) • Our score quickly rose 2nd place

39. Once Immunix Server Up …in the Score’bot’s Opinion :) Close 2nd place

40. Once Immunix Server Up …in the Score’bot’s Opinion :) 1st place • Stayed there most of Saturday

41. Late Saturday:New Service Requirement • With 4 hours of play to go, the score’bot changed: now it wanted Webmin • Open source web-GUI for Linux administration • Competitor to WireX’s commercial server appliance software • Rather famously vulnerable :) • Took us 2 hours Sunday morning to make the score’bot happy again • Lost our lead

42. Lock Out the Owner Once we root the machine, install a back door Also replace root’s login shell with /sbin/halt Owner can’t log in to their own machine But we can Spam’bot Add user to their server User sends spam mail to all the other teams Costs them penalty points Penalties are per connection Spam’bot sends 1-byte e-mails Some of Our Creative Attacks

43. Final Score: 2nd Place

44. Lesson: Symmetric Red Teaming Solves Rules Issues • Everyone is both an attacker and defender • Bad: everyone needs to learn how to attack • Good: • Everyone should learn how attacks are done :-) • Rule fussing about how hard or easy it is for the attacker apply to all parties -> less fussing • Ghettohackers have designed a great game • Looking for technology transfer to Government

45. Lesson: Mandatory Access Control is Not Enough • telnetd was a required service • WireX never bothered to patch a vulnerability in telnetd for Immunix • Only idiots run telnetd :-) • Someone hacked our telnetd • Didn’t get out of the SubDomain sandbox • Did make our telnetd stop working • Cost us a point that round • General case: MAC protects your system, but not your individual services

46. Lesson: Resource Management is a Security Attribute • SubDomain confined attacker logins to only run prescribed code • Including PERL • Attacker launched a PERL fork bomb • Consumed all of real and virtual memory • While our machine is thrashing, the score’bot passes us by • Costs us a point that round

47. Lesson: Redundancy Helps When You Are Vulnerable • Penetration attacks take a long time to recover • Must clean up state, find & fix vulnerability • DoS attacks take a long time to recover • If machine crashes, must fsck file system; can take 10 minutes • Hot spare can be on-line in seconds • Heterogeneous hot spare keeps attacker from immediately deploying the same attack

48. Lesson: Redundancy is Resource-Constrained • Must have humans on watch to clean up the compromised machine • The hot spare will not protect you for long • Presumption that hot spare prevents repeat attacks assumes resource limit at the attacker’s end • If attacker has lots of exploits/resources, they will hack your heterogeneous server just as quickly • We had a hot spare, but not enough of them

49. Lesson: Immunix was Impenetrable, but not Incorruptible • No one ever “flagged” the Immunix server • Others did plant enemy flags on our reference server (as expected) • But they did hit the Immunix server hard enough to compromise availability • Take out one required service, and the score’bot doesn’t award a point • We missed first place by less than 4 points out of 55

50. Immunix SecureServer Appliances • Base: Immunix Secure OS • Appliances that defend themselves • Turn-key installation: • WireX installer turns PC hardware into server appliance in 5 minutes • No technical skills required • Graphical ease of use • WireX Secure Server Manager Web-GUI management system