Backtracking Algorithmic Complexity Attacks Against a NIDS

1. Backtracking Algorithmic Complexity Attacks Against a NIDS Randy Smith, Cristian Estan, Somesh Jha University of Wisconsin–Madison

2. Algorithmic Complexity Attacks • Vulnerable algorithm: algorithm whose worst case differs from typical case. The larger the difference, the more vulnerable the algorithm. • Examples:

3. Algorithmic Complexity Attacks • Algorithmic Complexity Attack – an attacker induces worst-case behavior in a vulnerable algorithm. • Common observable effect is denial of service. • Crosby and Wallach: induced worst-case behavior in hash function implementations. • “Algorithms are now part of the attack surface” (Crosby and Wallach, 2003)

4. Are NIDS vulnerable? • NIDS and IPS are ubiquitous, but… • Do they contain vulnerable algorithms? Can they be exploited? • YES! Only need 1 packet every 3 seconds.

5. Evading a NIDS • Attacker’s Goal: Evade NIDS • Two attack vectors in an evasion attempt: 1st—alg. complexity attack targeting the NIDS 2nd—true attack targeting the network • Effect of an algorithmic complexity attack: • (NIDS) Packets enter network unexamined • (fail-closed IPS) Packets are dropped

6. Main results • In Snort, vulnerability in rule-matching • worst-case vs. typical case: 6 orders of magnitude. • “Backtracking Attack” • Easily exploitable through packet payloads • Improved rule-matching algorithm limits running time differences to within 1 order of magnitude.

7. Outline • Snort rule matching • Inducing backtracking attacks • Countermeasures • Measurement results • Conclusion

8. Snort Rule Matching alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678)

9. Snort Rule Matching alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678) Rule matches! fmt=acc player=default fmt=mp3 rate=14kbps player=cmd.exe?overflow#@!%

10. P2 P2 P3 P4 P5 Rule matches! Matching the packet P1 alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678) fmt=acc player=default fmt=mp3 rate=14kbps player=cmd.exe?overflow#@!%

11. P1,P2,P3,P4 match in 3 positions each P5 never matches Inducing Backtracking attacks alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”ReelAudio jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678) • Leads to excessive packet traversals! fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe fmt=acc player=default fmt=mp3 rate=14kbps player=cmd.exe?overflow#@!%

12. P1 P2 P2 alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 P3 P3 content:”player=”; //P3 P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 P4 P4 P4 P4 P4 P4 P4 sid:5678) P4 P4 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 Matching the malicious packet P1 P2 fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe

13. Are real rules vulnerable?

14. Memoization: maintain a table of subproblem “answers”; never evaluate a predicate twice at the same starting payload offset Safer backtracking alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678) • Identify constrained predicate sequences • Monotone memoization: don’t re-evaluate monotone predicates that have been evaluated at lower offsets

15. 18 4 P2 P2 11 P3 P3 7 14 21 P4 P4 P4 P4 P4 P4 28 35 42 28 35 42 28 35 42 P4 46 46 46 54 54 54 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 50 50 50 P5 P5 P5 Reductions in processing cost P1 P2 P3 P4 P4 P5 P5 P5 P5 P5 P5 fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe

16. Outline • Snort rule matching • Inducing backtracking attacks • Protecting against backtracking attacks • Measurement results • Conclusion

17. Measurement results

18. Live experiment topology Background Traffic AC Attack True Attack

19. Live experiment • Background Traffic @ 10Mbps • AC Attack • Targets Snort SMTP rule 3682 • Directed at sendmail server • True Attack: NIMDA • 300 exploit attempts, sent 1 byte per second. • New exploit started every second.

20. Live experiment results

21. Conclusions • NIDS operation is complex. Many opportunities for vulnerable algorithms. • In Snort, rule-matching is vulnerable and can be exploited by an attacker. • Memoization, along with other semantics-preserving operations, significantly reduces vulnerability. • Other vulnerable algoritms exist.

22. Backtracking Algorithmic Complexity Attacks Against a NIDS Thank you.