SHARK : Architectural Support for Autonomic Protection Against Stealth by Rootkit Exploits

1. Computer Architecture Lab at SHARK: Architectural Support for Autonomic Protection Against Stealth by Rootkit Exploits Vikas R. Vasisht, Hsien-Hsin S. Lee School of Electrical and Computer Engineering Georgia Tech Reading Group, July 21 2009 Based on slides retrieved from http://arch.ece.gatech.edu/present/micro41.pdf

2. Rootkits • Programs used to conceal malware presence • E.g. hide keylogger or network activity • Hard to detect • Anti-rootkit tool executes inside compromised system that tries to hide the intrusion

3. Simple Rootkit System Administrator (e.g., “ps”, “top”) Malware removed from list of running processes User Program User Space API Function Return Library Import Address Table Choose Interrupt Handler from IDT Choose Syscall from SSDT Syscall Function Kernel Space Interrupt Descriptor Table System Service Descriptor Table

4. Existing Anti-Rootkit Tools • Signature-Based • Only works with known rootkits • Behavior-Based • Calculate time and instructions spent on system queries • Unusually high count indicates the O/S is manipulating response • Too many false positives • Cross-level system view • Query system in high and low level • E.g., compare ps response against scheduler lists • The compromised O/S can manipulate all levels

5. Sophisticated (Anti-)Rootkits • Integrity-based detection • Compare snapshot of memory against trusted baseline • Can be fooled • Shadow-Walker • Mark all pages as unavailable • On TLB-i (execute) page fault: present malicious code • On TLB-d (snapshot) page fault: return ‘correct’ page TLB-i View App Page Table TLB-d View AUDIT OK!

6. Even More Sophisticated Rootkits • SubVirt • Install virtual machine monitor • Downgrade host O/S to guest O/S • Run Malicious O/S on the side • Blue Pill • Install tiny malicious hypervisor on-the-fly App1 App2 Host O/S Hardware M1 M2 App1 App2 Mal O/S Host O/S Virtual Machine Monitor Hardware

7. Outline • Motivation • Rootkits • SHARK • High-level View • Page Table Encryption, Update, Decryption • Address Space Protection • Stealth Checker • Evaluation • Security Vulnerabilities • Conclusion

8. Problem • Nearly impossible to detect all rootkits • Especially the virtualization-based ones: guest O/S is clean • All tools depend on the malicious O/S • Proposed solution • Have the processor securely report every application that enjoys hardware resources • Malicious apps running in different VMM will be exposed

9. SHARK: Bird’s Eye View • Hardware should record running processes • Must force O/S to give real PID to CPU • Trusted third party audits system • Compare H/W and O/S reports • Discrepancy raises alarm • O/S cannot hide a malicious app • Tamper-evident address space • O/S blocked from manipulating normal apps • Malware hijacking an app will leave a trail • Idea: Page Table Encryption

10. Page Table Encryption: Key • App’s page table is encrypted using PID • Decrypted on-the-fly during hardware walk using same PID • O/S assigned PIDs are vulnerable • Frequent reuse problematic for security • SHARK provides hardware PID (HPID) • 64-bit counter: won’t overflow, nonce • Only care for memory-resident rootkits, reset will eliminate them • During app creation • O/S request PID and provides an initialized page table • H/W encrypts page table and returns HPID++

11. Page Table Encryption (cont) • Encrypt last level page table entries (counter mode) • Encrypt first & last level valid bits (counter mode) • Force same PID to be used during walk every time • All updates through special H/W instruction • MODPT (modify page table) • Must provide correct PID • Otherwise meaningless 1st level decryption

12. Valid Bit Array Encryption Offset Counter Base (128 bit) HPID||HPID V Translation Array V Translation Array Counter = HPID||HPID 1 0 Pipelined Counter Mode Encryption Engine AES-128 1st 128-bit Valid Bit Block … 1 Counter = (HPID||HPID)+1 1 … 2nd 128-bit Valid Bit Block 0 0 0 … Counter = (HPID||HPID)+N-1 1 1 N 128-bit Valid Bit Block … 0 Page Directory 1 Page Frame Actual Frame stored in memory Hardware Secret Key: Burnt-in, unexposable

13. Page Table: Initial Encryption Counter (128 bit) HPID||HPID V Translation Array V Translation Array Pipelined Counter Mode Encryption Engine AES-128 … 1 ADDR 4 entries concatenated 128-bit ADDR 0 NULL 0 NULL 1 ADDR … Last Level Page Directory 1 Page Frame Actual Frame stored in memory Hardware Secret Key: Burnt-in, unexposable

14. Page Table: Entry Update If other valid entries, decrypt and reencrypt, otherwise just encrypt

15. Page Table Decryption – TLB update Counter (128 bit) HPID||HPID Counter (128 bit) (HPID||HPID)+k V PDE V PDE Normal Page Table Walk V Counter Mode Decryption AES-128 … Counter Mode Decryption AES-128 1 0 ADDR 0 1 … HW Key HW Key Page Directory Last Level Counter (128 bit) HPID||HPID PDE V … 0 Counter Mode Decryption AES-128 0 4 entries encrypted 128-bit 4 entries decrypted 128-bit TLB update ADDR 0 1 … HW Key

16. Address Space Protection • O/S should not tamper with address space of apps • Protect applications from swap-hijacking • Expose malware during auditing • Shouldn’t mark as unavailable and return convenient page • H/W signs pages and checks signatures Mem Page Mem Page Counter Mode Encryption AES-128 Counter Mode Encryption AES-128 4KB 4KB SHA-256 checksum SHA-256 checksum 32B 32B HW Key Tamper-evident swapping Caution: No check during first page mapping =? Disk Disk Counter (128 bit) HPID||HPID Valid/Invalid update

17. Stealth Checking • Firmware called on context switch • Record incoming PID of new process • Every 10 switches, timestamp, encrypt and call O/S to ship OS Alarm Context Switch Hardware Tamper-evident timestamped packet. Lost packets or O/S tampering trigger alarm. HPID Other SHARK H/W PID1 PID2 PID3 Master PID List P I D s …

18. Outline • Motivation • Rootkits • SHARK • High-level View • Page Table Encryption, Update, Decryption • Address Space Protection • Stealth Checker • Evaluation • Security Vulnerabilities • Conclusion

19. Evaluation • Emulation using Bochs • All installed rootkits detected • Timing results using Simics • Close match to Core 2 processors • In-order simulation • First 2 billion instructions of SPEC apps • Most page faults (and overhead) during initialization

20. Evaluation (cont) Expected: More TLB entries, less overhead

21. Outline • Motivation • Rootkits • SHARK • High-level View • Page Table Encryption, Update, Decryption • Address Space Protection • Stealth Checker • Evaluation • Security Vulnerabilities • Conclusion

22. Security Hole #1 • 1st page map cannot be checked • Possible exploit by cleaner-app? Present malicious signature & restore mapping, but freeze auditing All pages marked as invalid, app works through TLBs Mali-cious App’s Page Table Name: xcalc Write Access Audit app launched:Pagefault Note: cleaner-app will leave a PID trace in the stealth checker

23. Security Hole #2 • All page tables but the last (completely) and first (valid bits only) are plaintext • O/S can manipulate internal mappings VALID VALID VALID VALID Level 1 Level 2 Level 3 Data Pages View during execution View during auditing

24. Security Hole #3 • Security 101 • Encryption: to keep something secret, private • Signing: to make something unmodifiable, tamper-evident and bind message pieces together • BAD idea to use one for the other • Creates illusion of protection • Why encrypt the page table? • O/S already knows it, so why keep it a secret? • Signing is necessary, encryption is useless • Paper based on assumption that any tampering will result in meaningless page table

25. Security Hole #3: Exploit Counter Mode Encryption Counter Mode Decryption Counter Counter H/W Key AES 128 AES 128 H/W Key 128-bit plaintext 128-bit Ciphertext ⊕ ⊕ • D(E(A)) = E(A) ⊕ Cloud • = A ⊕ Cloud • ⊕ Cloud = A E(A) = A ⊕ Cloud 128-bit Ciphertext 128-bit plaintext O/S can overwrite E(A) with: (E(A), A, B known to O/S) E(A) ⊕ A ⊕ B = A ⊕ Cloud ⊕ A ⊕ B = B ⊕ Cloud = E(B) H/W will normally decrypt to B

26. Security Hole #3 (cont) • What they intended to say • The entry for virtual address V, for process PID, at level lvl, is physical P (next level pointer or data location) • How it should have been written • P, σΚ(P, V, PID, lvl) (σ= signature) • Bind each page table entry with where it should be located • What was actually written • Some entries are secret; I won’t tell you • Though you can ask me • …And you already know

27. Security Hole #4 • The virtual memory system is used as a fence • (theoretically) tamper-evident page table • O/S can simply bypass the VMM • Issue writes to physical address directly • O/S can purge malicious pages prior to auditing • Similar to cleaner-app, but without any trace

28. Security Vulnerability #1 • Stealth checker employs upgradeable firmware • Rootkit that is stored in disk can upgrade firmware • System reboot will not clean system • (Beyond the scope of this paper) • Compromised O/S may compromise firmware • Enjoy the CPU spying on you • Impossible to detect

29. Security Vulnerability #2 • BAD idea to reuse counter of counter-mode encryption Same for every last-level encryption Counter (128 bit) HPID||HPID V Translation Array V Translation Array Pipelined Counter Mode Encryption Engine AES-128 … 1 ADDR 4 entries concatenated 128-bit ADDR 0 NULL 0 NULL 1 ADDR … Last Level Page Directory 1 Page Frame Actual Frame stored in memory Hardware Secret Key: Burnt-in, unexposable

30. Conclusion • Rootkits are a growing concern • Hardware monitoring is a good idea for protection • Malware cannot cheat hardware that easily • Good idea for rootkits also: BIOS rootkit in the works • Paper’s implementation does more harm than good • Broken Algorithm: signing of each page table level is needed • But O/S can bypass virtual memory manager • Cleaner-app is also a problem • LBA should target O/S protection

31. Questions? SYSTEM PROTECTED