Whispers in the Hyper-space: High-speed Covert Channel Attacks in the Cloud

1. Whispers in the Hyper-space:High-speed Covert Channel Attacksin the Cloud Zhenyu Wu†, Zhang Xu, Haining Wang The College of William and Mary † Now affiliated with NEC Laboratories America Inc.

2. Virtualization and Cloud Computing • Server Virtualization • Consolidates workload • Simplifies resource management • Enabling Utility-based Cloud Computing • Server Virtualization Technologies • Goal: Computing consolidation • Design: Logically-separate but physically-shared • NOT equivalent to physically separated machines • Non-negligible differences expected, e.g. covert channels

3. The Threat of Covert Channels • Covert Channels • Exploit imperfection in the isolation of shared resources • Enable communication bypassing mandatory access controls (i.e. data exfiltration) • On non-virtualized systems • File system objects, network stack, shared processor cache, input device, etc. • On virtualized systems • No practical exploit to date

4. Related Work • Precautionary Research • Hey, you, get off of my cloud (Ristenpart et al., CCS’09) • Bandwidth 0.02bps << 1bps by TCSEC • Follow up research (Xu et al., CCSW’11) • Improved bandwidth 3.2bps (~10% error rate) • Conclusion: can do very limited harm in the cloud • Industry Response • Amazon EC2 Dedicated Instances • Significant extra-cost (Targeting the “paranoid high-end”?)

5. Research Goal • Common belief: • Covert channel is not a serious threat • We say otherwise! • Stimulate serious research on countermeasures

6. Project Outline • Understand Existing Cache Covert Channel • Why it doesn’t work in the Cloud • Design a New Covert Channel • Targeting cross-VM communication • Realistic Evaluation • In-house and on Amazon EC2

7. Struggles of Cache Covert Channel • Classic Cache Channel • A hybrid timing & storage channel 1 0 0 1 0 1 1 0 1 1 0 1 1 0 0 1 0 1 0 1 Receiver Sender Cache

8. Struggles of Cache Covert Channel • Classic Cache Channels • Works very well on hyper-threaded processor • Bandwidth ~400 kilo bytes/sec ~3,000,000 bits/sec • Performs very poorly on virtualize environment • Bandwidth ~1 bit/sec • Reasons for Struggling • Addressing Uncertainty • Scheduling Uncertainty • Cache Physical Limitation

9. Struggle #1: Addressing Uncertainty • L1 Cache • VIVT = Precise Control L1 Cache (VIVT) Process A Process B

10. Struggle #1: Addressing Uncertainty • L2 Cache • VIPT / PIPT – Physical memory address involved L2 Cache (VIPT / PIPT) Physical Address Process A Process B Physical Address

11. Struggle #1: Addressing Uncertainty • Virtualization • Additional Layer of Indirection (or, complication) L2 Cache (VIPT / PIPT) VM 1 VM 2 Process B Guest Physical Address Guest Physical Address Process A Host Physical Address Host Physical Address

12. Struggle #1: Addressing Uncertainty • Result • Percival’s original cache channel – 64 regions • 64 bits per cache pattern • EC2 cache channels by Ristenpart and Xu – 2 regions • 1 bit per cache pattern • Reduction of near 2 orders of magnitude!

13. Struggle #2: Scheduling Uncertainty • Cache “Round-trip” Requirement • Strict Round-Robin Scheduling 1 1 0 1 0 1 1 0 1 Receiver Detect Sender Prepare Receiver Initialize Receiver Receiver Time Sender

14. Struggle #2: Scheduling Uncertainty • Too Many Invalid Scheduling Alternatives Receiver Other Receiver Receiver Receiver Other Receiver Receiver Receiver Receiver Other Receiver Receiver Receiver Receiver Time Time Time Time Time Time Sender Sender Sender Sender Sender Sender

15. Struggle #2: Scheduling Uncertainty • Result • Highly sensitive to non-participating workload • 30% error rate with moderate workload in laboratory setup • Low success in realistic environment • ~10% valid scheduling on Amazon EC2

16. Struggle #3: Physical Limitation • Require stably shared cache, however • Non-participating workload destabilize the cache Core Core Core Core Processor Core Core Core Core Core Core Core Core Core Core L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L2 L2 L2 L2 L3 L3 Receiver Receiver Lily Bob John Sender Emma

17. Struggle #3: Physical Limitation • Require stably shared cache, however • Non-participating workload destabilize the cache • Cores between physical processors do not share cache! Processor Processor Core Core Core Core Core Core Core Core L1 L1 L1 L1 L1 L1 L1 L1 Core Core L2 L2 L1 L1 Receiver Sender

18. Project Outline • Understand Existing Cache Covert Channel • Why it doesn’t work in the Cloud • Design a New Covert Channel • Targeting cross-VM communication • Realistic Evaluation • In-house and on Amazon EC2

19. Redesigning Data Transmission • Change of Data Modulation Technique • Cache region based encoding no longer suitable • State Patterns  Timing Patterns 1 0 0 1 0 0 1 1 1 1 0 1 0 1 0 Cache Miss Latency 1 0 1 0 1 Cache Hit Time

20. Redesigning Data Transmission • Change of Data Modulation Technique • Cache region based encoding no longer suitable • State Patterns  Timing Patterns • Change of Scheduling Requirement • Signal generation and detection are instantaneous Receiver Receiver Time Sender Sender Valid For Data Transmission

21. Demonstration • L2 cache channel using new transmission scheme • Intel Core2, Hyper-V, Windows Guests 39 bits in 200μs  190 kilo bits / sec

22. Progress So Far… • Addressing Uncertainty • Scheduling Uncertainty • Cache Physical Limitation

23. An Insurmountable Limitation • A shared cache is not guaranteed • Virtual processor migration results in frequent interruption Processor Processor Core Core Core Core Core Core Core Core L1 L1 L1 L1 L1 L1 L1 L1 Core Core L2 L2 L1 L1 Receiver Sender

24. The Memory Bus:Fibers of the Hyper-space

25. The Bus Contention Channel • Look beyond the processor cache… … we see the memory bus Processor Processor ! ! ! ! ! ! ! ! Core Core Core Core Core Core Core Core L1 L1 L1 L1 L1 L1 L1 L1 L2 L2 Contention Memory Bus

26. Exploiting Atomic Instructions • Atomic Instructions • Special x86 memory manipulations • Ensures operation atomicity • Very useful for thread synchronization… … and covert channels Core Core Core Core Access Access Atomic Blocked Data Memory

27. Exploiting Atomic Instructions • Theory vs. Reality • Theoretical • Performance impact is localized Core Core Core Core Access Access Access Atomic Data Data Memory

28. Exploiting Atomic Instructions • Theory vs. Reality • Theoretical (Not Exploitable) • Performance impact is localized • Pre-Pentium Pro • Atomicity is implemented using bus lock signal ! ! ! Core Core Core Core Memory Bus Data Memory

29. Exploiting Atomic Instructions • Theory vs. Reality • Theoretical (Not Exploitable) • Performance impact is localized • Pre-Pentium Pro (Exploitable) • Atomicity is implemented using bus lock signal • Pentium Pro ~ Pre-Nehalem • Bus lock is reduced, but not eliminated

30. Exploiting Atomic Instructions • When target data can be put on a cache line • Cache line locking is performed • But when target data spans two cache lines… • Reverts to old bus locking behavior ! ! ! Core Core Core Core Cache Cache Cache Cache Memory Bus Data Memory Data

31. Exploiting Atomic Instructions • Theory vs. Reality • Theoretical (Not Exploitable) • Performance impact is localized • Pre-Pentium Pro (Exploitable) • Atomicity is implemented using bus lock signal • Pentium Pro ~ Pre-Nehalem (Exploitable) • Bus lock is reduced, but not eliminated • Nehalem ~ ? • No shared memory bus!

32. Exploiting Atomic Instructions • When target data can be put on a cache line • Cache line locking is performed • But when target data spans two cache lines… • Coordinated flush of all in-flight memory operations ! Core Core Data Cache Cache ! ! Core Core Data Memory Memory Memory Memory Cache Cache

33. Exploiting Atomic Instructions • Theory vs. Reality • Theoretical (Not Exploitable) • Performance impact is localized • Pre-Pentium Pro (Exploitable) • Atomicity is implemented using bus lock signal • Pentium Pro ~ Pre-Nehalem (Exploitable) • Bus lock is reduced, but not eliminated • Nehalem ~ ? (Still Exploitable) • No shared memory bus!

34. Demonstration • Memory Bus Channels Intel Core2, Hyper-V, Windows VMs Intel Xeon (Nehalem), Xen Linux VMs 39 bits in 1ms  38 kilo bits / sec

35. So Far, So Good… AndThen? • Addressing Uncertainty • Scheduling Uncertainty • Cache Physical Limitation

36. Space Communication is Hard!

37. Reliable Communication • The memory bus channel is quite noisy Contended Contention-Free ~60 bits ~43 bits ~60 bits …

38. Reliable Communication • Receiving Confirmation • No Explicit Send-and-Ack • Simultaneous receiver detection • Leveraging the system-wide observable bus contention • Clock Synchronization • Self-clock coding – Differential Manchester Coding • Standard network coding used in token ring network • Error Correction • Again, no Send-and-Ack • Forward Error Correction – Reed-Solomon Coding • Use for optical discs, satellite communication, etc.

39. Reliable Communication • Data Framing • Send data in small fixed-length packets • Improves resilience to interruptions and errors • Overview of Complete Protocol Data Preamble Data Data Data Data Block Data Parity Parity Data FEC Segments Data Frame Segments DM Encoded Data Frame

40. Project Outline • Understand Existing Cache Covert Channel • Why it doesn’t work in the Cloud • Design a New Covert Channel • Targeting cross-VM communication • Realistic Evaluation • In-house and on Amazon EC2

41. In-house Experiment • Realistic Setup • Intel Xeon (Nehalem), Xen, Linux Guests • Spawn two VMs, single vCPU, 512MB memory • Sender and receiver runs as unprivileged user programs • Protocol Parameters • 32 bit data frame (8 bit preamble, 24 bit payload) • FEC Parity : Data = 4 : 4 • Channel Capacity • Bandwidth: 781.6 ± 13.6 bps • Error rate: 0.27%

42. Amazon EC2 Experiments • Preparation • Spawn two physical co-resident VMs • /proc/cpuinfo: Intel Xeon (Yorkfield) (share memory bus) • Initial Trial: • Protocol Parameters • 24 bit data frame (8 bit preamble, 16 bit payload) • FEC Parity : Data = 4 : 4 • Best Channel Capacity • Bandwidth: 343.5 ± 66.1 bps • Error rate: 0.39% • But degrades over time in an interesting way

43. Amazon EC2 Experiments • Three Staged Performance • Due to Xen scheduler preemption

44. Amazon EC2 Experiments • Adapting to “Offensive” scheduling: • Protocol Parameters • Slow down transmission rate by 20% • Increase FEC Parity : Data = 4 : 2 • Stable Channel Capacity • Bandwidth: 107.9 ± 39.9 bps • Error rate: 0.75%

45. Amazon EC2 Experiments • Protocol Versatility • Automatic speed adjustment according to channel quality

46. How to Mitigate the Problem?

47. Mitigation Avenues • Tenants • Limited Options – Performance Anomaly Dentations • Defender continuously monitors memory access latency • High overhead, high false positives • Cloud Providers • Preventative Measures • Dedicated Instance (Amazon EC2) – Effective but costly • Alternative Low-Cost Placement Strategies • Controlled and deterministic sharing • Makes arbitrary attack difficult

48. Mitigation Avenues • Cloud Providers • Detective Measures • Leverage access to Hypervisor, and rich server resource • Discover performance anomalies by hardware counters • Low overhead, low false positive penalties • Hardware Manufactures • Isolation Improvements (Preventative Measures) • Trap instead of handling exotic conditions • Tag request by VM, and limit performance impact scope • Costly (one-time), but effective and efficient

49. Conclusion • Cover Channel Attacks in the Cloud are Practical • We uncovered a high speed covert channel medium • We designed a reliable transmission protocol • We evaluated the performance in a real cloud environment • Call for Effective and Efficient Mitigations • Three avenues of approaches: • Tenant • Cloud Provider • Hardware Manufacture

50. Q & A Thank you for listening!