Erbil YILMAZ Area Survey Exam Summer 2003

Survey of Intrusion Detection and Attack Description LanguagesBased on Monitoring Program Behavior Erbil YILMAZ Area Survey Exam Summer 2003

Intrusion Detection Approaches • Anomaly Detection • Detect deviations from normal behavior • Misuse Detection • Looks for pattern or predefined sequences of events defining a known attack • Specification Based • Describes the intended behavior of programs

Monitoring Program Behavior • Regardless of the nature of an attack, damage can ultimately be affected only via system calls made by processes running on the target system • By monitoring and restricting system calls, possible attacks can be prevented • Increase efficiency of IDS • System call interception is usually achieved by wrappers

Early Systems Using Program Behavior Patterns • JANUS • Restricts access of untrusted helper applications to the operating system • “Sense of Self” for Unix Processes • Normal behavior is defined by short-range correlations in a process’s system calls • DPEM • Describes a security-critical application’s intended behavior through specifications

JANUS • Netscape and ghostview example • Uses “/proc virtual file system” • Modular approach: framework + dynamic policy modules • First configuration file is read, which contains lines like: • path allow read, write /tmp/*

JANUS Modules • Basic module: Supplies defaults for system calls that are easiest to analyze and takes no configuration parameters. • Putenv module: Allows one to specify environment variable settings for the traced application via its parameters. The ones that are not mentioned are unset. • Tcpconnect module: Allows restricting TCP connections by host and port. • Path module: Allows or denies path access according to patterns.

Sense of Self for Unix Processes • Based on an analogy between the human immune system and intrusion detection • Normal behavior is defined by short-range correlations in a process’s system calls • Unix’s strace is used to intercept system calls • A file of short sequences of system calls is created

Short Sequence of System Calls Creation Procedure • Application runs and all its system calls are recorded • These sequences then decomposed into short sequences using sliding window • Unique ones are stored in a database and called N-grams • Comparatively small collection of short sequences derived

DPEM • Describes a security-critical application’s intended behavior through specifications • Monitors its behavior for violations of these specifications also called trace policies • Audit trails are used to check for violations of these trace policies

Security Relevant Characteristics of Programs • Access: The set of system objects being accessed through the program’s execution. • Sequencing: The order of operations performed by a program. • Synchronization: Refers to improper synchronizations of programs. • Race Conditions: A special case of the synchronization problem.

Example • 1. SPEC <(?, binmail, U, H)> • 2. ENV int CREATTMP = 0; • 3. ENV int PID = getpid(); • 4. SE: <binmail> || <other> • 5. <binmail> -> <init> <mktemp> <rest>. • 6. <init> -> <not_mktemp> <init> | Nil. • 7. <rest> -> any_op <rest> | Nil. • 8. <mktemp> -> open_tmpfile-PID { CREATTMP = 1; }. • 9. <not_mktemp> -> not_open_tmpfile-PID • 10. <other> -> <vop, CREATTMP> <other> | Nil. • 11. <vop, 0> -> not_chgtmp. • 12. <vop, 1> -> any_op. • 13. END;

Attack Description and Specification Languages • STATL • Extensible, domain-independent, state-transition-based attack description language • ASL • Specifies normal and abnormal behaviors of a process as logical assertions about an application program’s sequence of system calls and their argument values • BMSL • Monitors incoming network packets • Also monitors system calls requested by executing processes

STATL

Transition Types and Example • Non-consuming transition • Consuming transition • Unwinding transition

ASL • Supports automatic defensive actions • Normal and abnormal behaviors of a process as logical assertions • Local correctness assertions: execve(f) | f != “usr/ucb/finger” -> exit(-1) • Non-interference assertions: nonatomic(f) in (access(f, mode) .. open(f)->exit(-1))

ASL

BMSL • Efficient real-time intrusion detection • Supports temporally related event sequences

Some Related Software Engineering Concepts • Threat Model • A way to introduce security into design • Helps list vulnerabilities of a software system • Helps choose appropriate solutions • Aspect-Oriented Programming • Isolates different aspects of programming • Helps ease of comprehension and maintenance

Threat Modeling Process • Brainstorm the known threats to the system. • Rank the threats by decreasing risk. • Choose how to respond to the threats. • Choose techniques to mitigate the threats. • Choose the appropriate technologies from the identified techniques. • STRIDE Threat Model • Spoofing identity, Tampering with data, Repudiation, Information disclosure, Denial of service, Elevation of privileges

Aspect Oriented Programming • Some aspects of a program’s code • Data Storage • Security • Performance • Components • Connectors • Tangling of the code (aspect weavers)

Conclusions • Monitoring system calls is effective • Which system call sequence is normal? • Program behavior spec’s and monitoring system are highly coupled • Integrate software engineering practices to get better specifications • Specifications should be a part of dev. effort • Automated generation of monitoring system based on spec’s

Erbil YILMAZ Area Survey Exam Summer 2003