A Formal Approach to Robustness Testing of Network Protocol

1. A Formal Approach to Robustness Testing of Network Protocol Chuanming Jing1,2, Zhiliang Wang1,3, Xia Yin1,2, Jianping Wu1,2,3 1 Tsinghua National Laboratory for Information Science and Technology 2 Department of Computer Science & Technology, Tsinghua University 3 Network Research Center of Tsinghua University NPC 2008

2. Outline • Motivation and previous works • Our proposed method • Conclusions

3. Outline • Motivation and previous works • Our proposed method • Conclusions

4. Motivation • Among 5925 holes, 60％～80％ were caused by Inputs • Input Validation: 23％ • Boundary Condition:21％ • Exceptional Conditon:11％ • Access Validation: 10％ • Design Error:18％ Bugtrap Statistic in 2002(http://www.securityfocus.com)

5. Related works • Robustness testing (IEEE STD 610.12) • test to verify whether Implementation Under Test(IUT) can function correctly in the presence of invalid inputs or stressful environmental conditions • aims to detect vulnerabilities of protocol implementations • vulnerabilities of malformed message parsing • vulnerabilities of state transitions • hole of buffer overflow

6. Related works • Model-based robustness testing • Fuzz testing • Limitations: • lacks guidance of theory • verdict mechanism needs improvement • test system is not generic to other protocols • readability, extensibility and maintainability of test suite are not good • Highly desirable and critical to have a formal approach to robustness testing

7. Outline • Motivation and previous works • Formal Model • Test Generation • Extension of TTCN-3 and its Systems • Test practice • Conclusions

8. Formal Model: NPEFSM • Existing Models • FSM: Finite State Machine • EFSM: FSM + data -- protocol variables and operations • PEFSM: EFSM + parameters • Robustness testing • requires injecting many invalid messages • state transitions after these invalid injections are often nondeterministic • Our Model • NPEFSM: Nondeterministic Parameterized EFSM • covers more detailed and precise nondeterministic features i/o s1 s2

9. Formal Model: NPEFSM

10. Formal Model: NPEFSM • Transitions after injecting invalid inputs • Tdeter • Tnondeter-spec • Tnondeter-unspec

11. Outline • Motivation and previous works • Formal Model • Test Generation • Extension of TTCN-3 and its System • Test practice • Conclusions

12. Structure of Robustness Testing • Conformance testing • <State Leading Sequence, Executing Sequence, State Verification Sequence> • Robustness testing • Anomalous Test Case • <State Leading Sequence, Invalid PDU Inputting, Normal-Verification Sequence>

13. Normal-Verification Sequence • Requirements of Robustness testing • keep in the normal state • continue normal operations conforming to protocol specification • Construct Normal-Verification Sequence • tTdeter • Normal-Verification Sequence=State Verification Sequence • UIO Seq: Unique Input Output • tTnondeter • Normal-Verification Sequence=State Identification Sequence • Use Forced transition in test practice Tdeter State Verification s1 s2 Forced Transition Tnondeter State Verification s1 s2 S*

14. Compound Anomalous Test Case • Why compound anomalous test case? • Simplify the test sequence • Inject a large number of invalid inputs

15. Invalid Message Generation • Invalid inputs generation • Check one or more fields of a PDU • Normal PDU  invalid PDU (mutation) • Single-field • Multi-field • pairwise algorithm

16. Outline • Motivation and previous works • Formal Model • Test Generation • Extension of TTCN-3 and its System • Test Practice • Conclusions

17. Why we use TTCN-3 • TTCN-3 • Test and Testing Control Notations • ETSI: European Telecommunications Standards Institute • A standard testing language • Has many advantages and been widely used • Extension • Not good for mutation operation • Difficult for test case description

18. Extension of TTCN-3

19. Test System based on TTCN-3

20. Outline • Motivation and previous works • Formal Model • Test Generation • Extension of TTCN-3 and its Systems • Test Practice • Conclusions

21. Test practice • Tester：PITSv3 • IUT：Zebra-0.94

22. Test suite of compound anomalous test cases for OSPFv2

23. Test results multi-field single-field Zebra: cannot parse invalid messages with mutated “length” field in OSPF header with robustness

24. Outline • Motivation and previous works • Formal Model • Test Generation • Extension of TTCN-3 and its System • Test Practice • Conclusions

25. Conclusion and Future work • Conclusion • A formal approach to robustness testing • NPEFSM • TTCN-3 • Future work • application layer protocols • test real-time distributed systems • semantics of protocol

26. Thank you! Q&A wzl@cernet.edu.cn http://netarchlab.tsinghua.edu.cn/~wzl

27. Backup Slides

28. Related works • Model-based robustness testing • Difficult to guide test practice • Fuzz testing • Deliver semi-valid data to the target • Widely used in software testing • manual, not efficient • not generic

29. Formal Model: NPEFSM • Forced Transition：(sS')sj

30. Formal Model: NPEFSM • APart of NPEFSM for OSPFv2 Neighbor State Machine: Link State Database Exchange

31. Invalid Message Generation • Field value mutation rules • Boundary value • Input partition value • Field values mismatch • Format error • Length,Checksum and Encapsulation error • Field mutation rules • Removal and Addition • Overflow • Permutation

32. Compound Anomalous Test Case-2

33. Compound Anomalous Test Case Generation

34. Invalid Message Generation • Multi-field mutation • Pairwise algorithm: cover any pair of any two fields

35. Invalid Message Generation • Invalid inputs generation • Check one or more fields of a PDU • Normal PDU  invalid PDU (mutation) • Single-field and Multi-field (pairwise algorithm) • Field value mutation rules • Boundary value • Input partition value • Field mutation rules • Removal and Addition • Overflow • Permutation