Memory Forensics

1. Memory Forensics Son Dinh, Chris Gill, Brian Kocoloski CSE 522S – Advanced Operating Systems Washington University in St. Louis St. Louis, MO 63130

2. Dynamic Memory Allocation Heap allocator gives memory to a process • malloc() returns the memory’s address • At least one thread must store that address • Threads in same process can read/write it • Not true across processes unless shared Process B Process A 20 bytes 40 bytes 48 bytes 16 bytes CSE 522S – Advanced Operating Systems

3. Mistakes with Dynamic Memory • Failing to keep at least one pointer to it • C/C++ are not garbage collected, so it’s lost • “Double free”: freeing the same memory twice • “Use after free” -- reading/writing memory that’s been freed • Bad pointer arithmetic, buffer overruns, etc. Process A Process B 20 bytes Freed memory Leaked memory 16 bytes CSE 522S – Advanced Operating Systems

4. Linux cgroups Features Filesystem entries mapped for different resources • Memory (today’s focus), CPUs, I/O, network E.g., /sys/fs/cgroup/memory/ • Allows memory use limits to be set A process can watch memory limits for others • Via filesystem descriptors, event multiplexing, etc. • Uses “everything is a file” abstraction once again CSE 522S – Advanced Operating Systems

5. Studio Exercises Today Use mallinfo() to gather and print statistics • Showing changes in memory state when a program allocates or deallocates memory dynamically • Comparing the same sequence of memory operations with larger or smaller memory allocation sizes Use mallopt() or MALLOC_CHECK to detect errors • Double freeing, freeing or reallocating invalid addresses • Writing into memory that has been freed or that doesn’t belong to the program stack or its dynamically allocated memory Use the cgroups facility to set and detect memory limits • First for a single program, via the filesystem • Then for multiple programs, using event multiplexing CSE 522S – Advanced Operating Systems