Using Branch Correlation to Identify Infeasible Paths for Anomaly Detection

1. Using Branch Correlation to Identify Infeasible Paths for Anomaly Detection Xiaotong Zhuang Tao Zhang Santosh Pande Presented by Qian Zhu

2. Motivation • Complete security is hard to achieve • Buffer overflow attack • Bounds checking is lacking • Traditional attacks based on bounds checking involve code injection • Broader scope  overflow to tamper important data, change program execution behavior, etc. to bypass normal execution of programs • Anomaly detection • Detect anything that makes a program deviates its normal behavior

3. Outline • Background and Motivations • Attack Model and Machine Model • Branch Correlations • Approach • Experimental Evaluation • Conclusion

4. Attack: Privilege Escalation w/o Code Insertion 1. char str[SIZE], user[SIZE]; 2. 3. verify_user(user) 4. if (strncmp (user, "admin", 5)) { 5. ... 6. } else { 7. ... 8. } 9. strcpy (str, someinput); 10. if (strncmp (user, "admin", 5)) { 11. //superuser privilege 12. ... 13. } else { 14. ... 15. }

5. Attack: Privilege Escalation w/o Code Insertion 1. char str[SIZE], user[SIZE]; 2. 3. verify_user(user) 4. if (strncmp (user, "admin", 5)) { 5. ... 6. } else { 7. ... 8. } 9. strcpy (str, someinput); 10. if (strncmp (user, "admin", 5)) { 11. //superuser privilege 12. ... 13. } else { 14. ... 15. }

6. Attack: Privilege Escalation w/o Code Insertion 1. char str[SIZE], user[SIZE]; 2. 3. verify_user(user) 4. if (strncmp (user, "admin", 5)) { 5. ... 6. } else { 7. ... 8. } 9. strcpy (str, someinput); 10. if (strncmp (user, "admin", 5)) { 11. //superuser privilege 12. ... 13. } else { 14. ... 15. }

7. Outline • Background and Motivations • Attack Model and Machine Model • Branch Correlations • Approach • Experimental Evaluation • Conclusion

8. Attack Model • Interaction with the system through input channels, such as keyboards, network to launch attacks. • Multiple processes running on the same system, and some of them are corrupted and malicious.

9. Assumptions • We introduce hardware components for security inside the processor. • Process context is protected and is considered secure during context switch. • Read-only memory can be enforced by the processor. • Code accessing the critical data are available for compiler analysis.

10. Outline • Background and Motivations • Attack Model and Machine Model • Branch Correlations • Approach • Experimental Evaluation • Conclusion

11. Infeasible Paths • Branches could be correlated. • During normal execution, such relationship should be preserved. • Violation of branch correlations indicates anomaly

12. Branch Correlation—Example BB1 If y<5 N Y BB2 If x<10 Y N BB3 BB4 x=w y=z BB5 If y<10 N Y

13. Branch Correlation—Example BB1 If y<5 N Y BB2 If x<10 Y N BB3 BB4 x=w y=z BB5 If y<10 N Y

14. Observations • Violation of branch correlations indicates path anomaly • Values might be changed unexpectedly • Execution might be redirected • Efficiently detect branch violations • Branch correlations are complicated, lots of information • Branches are frequent • Checking must be fast and low cost

15. Outline • Background and Motivations • Attack Model and Machine Model • Branch Correlations • Approach • Experimental Evaluation • Conclusion

16. Branch Status Vector • Branch Status Vector (BSV) [Dynamically Updated] • Indicate the expected outcome of branches that will be executed next time • Used for detecting normal vs. abnormal branch outcome • Updates to the BSV internal to the hardware – not exposed to the user program • Branch Action Table (BAT) [Read Only] • A lookup table • Outcome of one branch tells us how to update expected outcome of (one or more) correlated branches

17. init BB5 BB1 BB3 BB2 UN T T T T UN UN T UN UN T T T UN T Example for Branch Status Vector BAT Lookup BB1 BR1 BR2 BR5 If y<5 N Y BB2 If x<10 Y N BB3 BB4 x=w y=z BB5 If y<10 N Y

18. BB3 BB5 BB1 BB1 init BB2 BB2 BB5 BB4 UN UN T T T T T UN T UN NT NT UN UN T UN NT UN NT T UN T T T T T UN Example for Branch Status Vector BB1 BR1 BR2 BR5 If y<5 N Y BB2 If x<10 Y N BB3 BB4 x=w y=z BB5 If y<10 N Y

19. Verify against the expected outcome in BSV 2 Look up in the BAT and update other branches in BSV 3 Operations w/ Branch Status Vector Get the outcome once a branch executes 1

20. Construct the Branch Action Table • A compiler algorithm • First step: alias analysis, identify memory resident variables • Establish subsumption relationship among branch conditions • Optimizations • Branch Checking Vector(BCV) • Only need entry for each correlated branch • Speedup common cases

21. Infeasible Path Detection System filter by branch checking vector(BCV) branch ID branch status vector (BSV) stack requestqueue verify lookup update branch action table (BAT) stack

22. Outline • Background and Motivations • Attack Model and Machine Model • Branch Correlations • Approach • Experimental Evaluation • Conclusion

23. Experimental Evaluation • Two parts: precision and performance • Compiler side: SUIF/MachSUIF • 10 server programs with known vulnerabilities • Arbitrary memory location could be tampered, 100 independent attacks – worst case assumed

24. Detection Rate • 29.3% of the tampering can be detected • 59.3% of the tampering that change control flow can be detected

25. Average Table Sizes • Average BSV size: 34 bits, average BCV size: 17 bits • Average BAT size: 393 bits

26. Parameters for Performance Evaluation • Simplescalar, 2 billion instructions

27. Normalized Performance • 0.79% slowdown

28. Outline • Background and Motivations • Attack Model and Machine Model • Branch Correlations • Approach • Experimental Evaluation • Conclusion

29. Conclusion • Anomaly detection helps prevent unknown attacks • We observe: branches are sometimes correlated  infeasible path detection  anomalous behavior • A low overhead compiler/architectural approach • No false positives, good detection rate

30. Questions & Answers