Penetration testing demystified

1. Penetration testing demystified James A. Whittaker Security ArchitectMicrosoft

2. Types of Testing • There are lots of different types of testing: • Performance • Reliability • Availability • Upgrade testing • Internationalization testing • … • The one thing each of these has in common is that they involve testing under realistic user scenarios • Security testing is the one glaring exception • Problem: where do you draw testing cases from?

3. Information Sources • Functional testing • The spec • User documentation • “Use cases” • What you know about how the system is supposed to work • Security testing • The interface • Error messages/ outputs • “Abuse cases” • What you know about how the system is NOT supposed to work

4. Information Sources (1) • The interface • The information on the screen is directed at legitimate users • Hackers are users too! • And they can learn a lot from what they see • Consider the following screenshot … can you spot the hack opportunity?

8. Information Sources (2) • Error messages • Are a special concern, they deal with bad input data • Usability v. information disclosure • Being helpful to users is fine, but don’t give hackers something they can use • Review all error message for their strategic value to users and adversaries

10. Another Example: Message Comparison

11. Compare this …

12. …to…

13. …this!

15. // TODO - Test this function Function.prototype.implementsInterface = function(interfaceType) { var interfaces = this._interfaces; if (interfaces) { if (interfaces.contains(interfaceType)) { return true; } } else if (this.bases) { for (vari=0; i < this.bases.length; i++) {   if(this.bases[i].implementsInterface(interfaceType)) { return true; } } } return false; }

20. Information Sources (3) • Abuse cases • Can be gleaned from historical information • MSRC cases against similar products • Bugtraq and CERT advisories • The black hat literature • Threat models • Enumerate entry points and describe threats

21. Information Sources (4) • What you know about how the system is not supposed to work • Inputs map to outputs … so what is the universe of dangerous inputs and unacceptable outputs? • It helps to understand more about inputs and outputs

22. Which Inputs? • Think clearly about where an input is coming from OS/RTE UI app external resource file system

23. Which Inputs? • Think clearly about where an input is coming from • What is the expected behavior when these inputs arrive? • How might things go wrong? OS/RTE UI app external resource file system

24. Which Inputs? • Think clearly about where an input is going to • Can it do any damage at its destination? • How might the app try to mitigate this damage? OS/RTE UI app external resource file system

25. SELECT * FROM <table> WHERE Account = ‘usersupplied’ AND Pin = ‘usersupplied’ SELECT * FROM <table> WHERE Account = ‘usersupplied’ AND Pin = ‘usersupplied’ SELECT * FROM <table> WHERE Account = ‘sam’ --

26. Categories of Dangerous Inputs* * This list subject to change without notice • Long strings • Format strings • Numeric boundaries • Scripts • Code • OS commands • ‘Control’ chars • Error codes • Return values OS/RTE UI app external resource file system

27. Automating Security Tests • Input generation and application • Lists, lists and more lists • Not often shared since they are business assets • Not often cross-applicable anyway • Plan to create your own • Applying inputs is easy, verifying results is the hard part • Some interfaces lend themselves well to automation • Network and file system specifically • Commercial web testing tools get historically bad reviews • Automation can and should focus on the above lists • Monitor and modify: fault injection and fuzz testing are low cost, high payoff techniques

28. Defect Severity • Ever heard “a user would never do that”? • A user might not but an adversary will • The team who refuses a bug should be able to say (with a straight face): • “Over the lifetime of this product (which may be ten years), I predict that no adversary will be clever enough to find a way to exploit that bug” • “And this is true for all environments in which the code will run now and in the future”

29. The Source of the Problem Where bad things come from

30. The Origin of Bad Things • They are in the environment in which our application executes • They arrive as inputs to our application • They are embedded within the logic our application

31. The Origin of Bad Things • They are in the environment in which our application executes • They arrive as inputs to our application • They are embedded within the logic our application

32. Environment Issues • Software does not execute in isolation • It relies on binaries and other code-equivalent (e.g., script) files • It uses configuration files and the registry • It uses databases and other local or remote services • These entities may provide an attacker a route to exploit our app! • A system is only as secure as its weakest link • The issue is: how much can we trust our environment?

33. Environment Issues • DLLs that are faulty or have been replaced or intentionally modified • Binaries/files that are incorrectly ACL’d • Other apps that access shared data • Sensitive data stored in the registry or in temporary files • System stress • Slow network, low memory, etc.

34. Example: Update Expert

35. Example: Update Expert

36. Example: Update Expert

37. Example: Update Expert

38. Example: Update Expert

39. Input Issues • Inputs originate from outside the system in which our app runs and arrive via entry points • Communication paths • Communication protocols, sockets • Exposed remote functionality (DCOM, RPC, Web Services) • Files • “Control” files (scripts, HTML, XML, controls …) • Data files, temp files, configuration files • UI • Logon screens, web front-end, …

40. Input Issues • Dangerous input entered in UI controls • Special characters, encoded input, script fragments, format strings, escape sequences • Long strings embedded in files • Corrupt packets in protocol streams • Localization of defenses (input filters) against bad input

41. Data and Logic Issues • Some faults are embedded in an application’s internals (data and logic): • Code path variation, not all code paths are secure • Information disclosure that gives advantage to an attacker • Hard coded accounts or test APIs not removed when the product ships • Authenticating on easily faked credentials • Most of these failures can be revealed with legitimate input without rigging the environment

43. Takeaways are next, then questions The end

44. Takeaways • As security testers, we: • Depend on a different set of documents and requirements • Threats • Interfaces (inputs and outputs) • It’s about thinking on your feet and thinking like an attacker

45. Takeaways • As security testers, we: • Depend on a different set attacks points • It’s not about the UI • It’s the invisible interfaces that we must worry about • It’s about getting used to looking for the invisible

46. Takeaways • As security testers, we: • Must do our homework • The past must be part of our present • MSRC, CERT, bugtraq • Keep current…what are the bad guys in your application space doing lately? • Extrapolate from this…what’s next? • It’s about being prepared and expecting the expected and unexpected

47. Takeaways • As security testers, we: • Must build an arsenal • Attack tools • Lists of dangerous inputs • Attack database/wiki/SharePoint site … • It’s about staying at the front of the arms race

48. Takeaways • As security testers, we: • Must understand that the threat is constantly changing • We cannot rely on yesterday’s exploits • We must try to stay a few steps ahead • It’s about what happened yesterday, today and what might happen tomorrow

49. Questions? • Recommended companion courses • Threat modeling • Fuzz testing • Recommended reading