Introduction & Machine -Level Programming

Introduction & Machine-Level Programming Haibo Chen Institute of Parallel and Distributed Systems (IPADS) Shanghai Jiao Tong University http://ipads.se.sjtu.edu.cn/haibo_chen

Outline • Why Choose This Course? • Course information • Intro to assembly language

Carnegie Mellon Abstraction Is Good: But Don’t Forget Reality Many courses emphasize abstraction Abstract data types Asymptotic analysis These abstractions have limits Especially in the presence of bugs Need to understand details of underlying implementations Useful outcomes from taking from this course Become more effective programmers Able to find and eliminate bugs efficiently Able to understand and tune for program performance Understand design and implementation of real complex systems Try to build one on your own (like a turing-award level OS)

Great Reality #1: Ints are not Integers, Floats are not Reals • Example 1: Is x2 ≥ 0? • Float’s: Yes! • Int’s: • 40000 * 40000 1600000000 • 50000 * 50000 ?? • Example 2: Is (x + y) + z = x + (y + z)? • Unsigned & Signed Int’s: Yes! • Float’s: • (1e20 + -1e20) + 3.14 3.14 • 1e20 + (-1e20 + 3.14) ?? Source: xkcd.com/571

Carnegie Mellon Great Reality #2: You’ve Got to Know Assembly Chances are, you’ll never write programs in assembly Compilers are much better & more patient than you are But: Understanding assembly is key to machine-level execution model Behavior of programs in presence of bugs High-level language models break down Tuning program performance Understand optimizations done / not done by the compiler Understanding sources of program inefficiency Implementing system software Compiler has machine code as target Operating systems must manage process state Creating / fighting malware x86 assembly is the language of choice!

Great Reality #3: Memory MattersRandom Access Memory Is an Unphysical Abstraction Memory is not unbounded It must be allocated and managed Many applications are memory dominated Memory referencing bugs especially pernicious Effects are distant in both time and space Memory performance is not uniform Cache and virtual memory effects can greatly affect program performance Adapting program to characteristics of memory system can lead to major speed improvements

Carnegie Mellon Memory Referencing Bug Example double fun(int i) { volatile double d[1] = {3.14}; volatile long int a[2]; a[i] = 1073741824; /* Possibly out of bounds */ return d[0]; } Result is architecture specific fun(0)  3.14 fun(1)  3.14 fun(2)  3.1399998664856 fun(3)  2.00000061035156 fun(4)  3.14, then segmentation fault

Carnegie Mellon Memory Referencing Bug Example double fun(int i) { volatile double d[1] = {3.14}; volatile long int a[2]; a[i] = 1073741824; /* Possibly out of bounds */ return d[0]; } fun(0)  3.14 fun(1)  3.14 fun(2)  3.1399998664856 fun(3)  2.00000061035156 fun(4)  3.14, then segmentation fault Explanation: Location accessed by fun(i)

Carnegie Mellon Memory Referencing Errors C and C++ do not provide any memory protection Out of bounds array references Invalid pointer values Abuses of malloc/free Can lead to nasty bugs Whether or not bug has any effect depends on system and compiler Action at a distance Corrupted object logically unrelated to one being accessed Effect of bug may be first observed long after it is generated How can I deal with this? Program in Java, Ruby or ML Understand what possible interactions may occur Use or develop tools to detect referencing errors (e.g. Valgrind)

Why Build OS? Operating systems are a maturing field Hard to get people to switch operating systems Then why? Many things are OS issues High-performance Resource consumption Battery life, radio spectrum Security needs a solid foundation New “smart” devices need new Oses And … Cloud computing & big data

Is “simple” really simple? A simple program? How does it get executed? How does it got translated to machine code? How does the program get executed in detail? Link, load, execute, finish? #include <stdio.h> int main() { printf(“hello world\n”); return 0; }

Carnegie Mellon Course Perspective Our Course is Programmer-Centric Purpose is to show that by knowing more about the underlying system, one can be more effective as a programmer Enable you to Write programs that are more reliable and efficient Incorporate features that require hooks into OS E.g., concurrency, signal handlers Build an operating system! Cover material in this course that you won’t see elsewhere Not just a course for dedicated hackers

Outline Why we study OS? Course information A tale of OS OS structures

Faculty Information Instructor CHEN Haibo 陈海波CHEN Rong陈榕 Office: Room 3-402, Software Building Office hour: 9:00am~10:00am Friday Contact: 13701830272 haibochen@sjtu.edu.cn 13661816826 rongchen@sjtu.edu.cn TA Wenhao Li利文浩

Tentative Course Schedule

Reference Books • xv6: a simple, Unix-like teaching operating system. By Russ Cox FransKaashoek Robert Morris. Online • Computer Systems: A Programmer's Perspective (2nd Edition) Addison Wesley. By Randal E. Bryant / David R. O'Hallaron

Labs Understand assembly programming Lab1: Binary Bombing Lab2: Buffer Overflow Implement JOS: adopted from MIT 6.828 course Lab3: Booting a PC Lab4: Memory Management Lab5: User Environments Lab6: Preemptive Multitasking

Lab 1 Binary Bomb Familiar with the assembly language Defuse interesting bombs in the binary code

Lab2 Buffer bomb Attack program by stack overflow

Now we start building our own OS

Lab 3 Boot a PC Familiar with PC boot process

Lab4 Memory Management Setup virtual memory map for JOS

Lab 5 User environment Environment in JOS = Process in Linux In this lab, only an environment on a single core

Lab6 Preemptive Multitasking Multicore Preemptive multitasking

Build your OS Supporting code will be provided Still a hard but interesting job Hints: Start to work early and work hard

Before the lab Generate your private/public key $ ssh-keygen -t dsa Send a email to TA (liwenhaosuper@gmail.com) Your name Your public key

Lab Rules Individual project Don’t copy any code from others Can read but don’t copy other OSes (Linux, Open/FreeBSD, etc.) Cite any code that inspired your code Please don’t post any “攻略”or code in BBS/forum

Grading General (Tentative) Lab 1~6: 45% Final Exam：40% Labs will be reflected in exam Homework: 10% Help you to understand OS/Labs Course performance (5%) Q&A in classroom, Presentation Lab vs. Grade The formula of finishing lab(L) with 60% core with the maximum grade Max(Grade) = ‘D’ – L (L=1,2,3,4) So you directly fails if no lab is done with 60% score

I am not that tough as you would think

Intro to Assembly

IA32 Processors Totally Dominate Computer Market Evolutionary Design Starting in 1978 with 8086 Added more features as time goes on Still support old features, although obsolete Complex Instruction Set Computer (CISC) Many different instructions with many different formats But, only small subset encountered with Linux programs Hard to match performance of Reduced Instruction Set Computers (RISC) But, Intel has done just that!

Assembly Programmer’s View CPU Memory Addresses Registers • Programmer-Visible State • PC Program Counter • Address of next instruction • Called “EIP” (IA32) or “RIP” (x86-64) • Register File • Heavily used program data • Condition Codes • Store status information about most recent arithmetic operation • Used for conditional branching • Memory • Byte addressable array • Code, user data, (some) OS data • Includes stack used to support procedures P C Object Code Program Data OS Data Data Condition Codes Instructions Stack

Turning C into Object Code • Code in files p1.c p2.c • Compile with command: gcc -O p1.c p2.c -o p • Use optimizations (-O) • Put resulting binary in file p C program (p1.c p2.c) text Compiler (gcc -S) Asm program (p1.s p2.s) text Assembler (gcc or as) Object program (p1.o p2.o) Static libraries (.a) binary Linker (gcc orld) binary Executable program (p)

Some compilers use single instruction “leave” Compiling Into Assembly Generated IA32 Assembly sum: pushl %ebp movl %esp,%ebp movl 12(%ebp),%eax addl 8(%ebp),%eax movl %ebp,%esp popl %ebp ret int sum(int x, int y) { int t = x+y; return t; } • C Code • Obtain with command • gcc -O -S code.c • Produces file code.s

Assembly Characteristics Minimal Data Types “Integer” data of 1, 2, or 4 bytes Data values Addresses (untyped pointers) Floating point data of 4, 8, or 10 bytes No aggregate types such as arrays or structures Just contiguously allocated bytes in memory Primitive Operations Perform arithmetic function on register or memory data Transfer data between memory and register Load data from memory into register Store register data into memory Transfer control Unconditional jumps to/from procedures Conditional branches

Object Code Code for sum 0x401040 <sum>: 0x55 0x89 0xe5 0x8b 0x45 0x0c 0x03 0x45 0x08 0x89 0xec 0x5d 0xc3 • Assembler • Translates .s into .o • Binary encoding of each instruction • Nearly-complete image of executable code • Missing linkages between code in different files • Linker • Resolves references between files • Combines with static run-time libraries • E.g., code for malloc, printf • Some libraries are dynamically linked • Linking occurs when program begins execution • Total of 13 bytes • Each instruction 1, 2, or 3 bytes • Starts at address 0x401040

Machine Instruction Example int t = x+y; • C Code • Add two signed integers • Assembly • Add 2 4-byte integers • “Long” words in GCC parlance • Same instruction whether signed or unsigned • Operands: x: Register %eax y: Memory M[%ebp+8] t: Register %eax • Return function value in %eax • Object Code • 3-byte instruction • Stored at address 0x401046 addl 8(%ebp),%eax Similar to expression: x += y Or int eax; int *ebp; eax += ebp[2] 0x401046: 03 45 08

Disassembling Object Code Disassembled 00401040 <_sum>: 0: 55 push %ebp 1: 89 e5 mov %esp,%ebp 3: 8b 45 0c mov 0xc(%ebp),%eax 6: 03 45 08 add 0x8(%ebp),%eax 9: 89 ec mov %ebp,%esp b: 5d pop %ebp c: c3 ret d: 8d 76 00 lea 0x0(%esi),%esi • Disassembler objdump -d p • Useful tool for examining object code • Analyzes bit pattern of series of instructions • Produces approximate rendition of assembly code • Can be run on either a.out (complete executable) or .o file

Alternate Disassembly Disassembled Object 0x401040 <sum>: push %ebp 0x401041 <sum+1>: mov %esp,%ebp 0x401043 <sum+3>: mov 0xc(%ebp),%eax 0x401046 <sum+6>: add 0x8(%ebp),%eax 0x401049 <sum+9>: mov %ebp,%esp 0x40104b <sum+11>: pop %ebp 0x40104c <sum+12>: ret 0x40104d <sum+13>: lea 0x0(%esi),%esi 0x401040: 0x55 0x89 0xe5 0x8b 0x45 0x0c 0x03 0x45 0x08 0x89 0xec 0x5d 0xc3 • Within gdb Debugger gdb p disassemble sum • Disassemble procedure x/13b sum • Examine the 13 bytes starting at sum

What Can be Disassembled? % objdump -d WINWORD.EXE WINWORD.EXE: file format pei-i386 No symbols in "WINWORD.EXE". Disassembly of section .text: 30001000 <.text>: 30001000: 55 push %ebp 30001001: 8b ec mov %esp,%ebp 30001003: 6a ff push $0xffffffff 30001005: 68 90 10 00 30 push $0x30001090 3000100a: 68 91 dc 4c 30 push $0x304cdc91 • Anything that can be interpreted as executable code • Disassembler examines bytes and reconstructs assembly source

%eax %edx %ecx %ebx %esi %edi %esp %ebp Moving Data: IA32 • Moving Data movlSource, Dest: • Move 4-byte (“long”) word • Lots of these in typical code • Operand Types • Immediate: Constant integer data • Like C constant, but prefixed with ‘$’ • E.g., $0x400, $-533 • Encoded with 1, 2, or 4 bytes • Register: One of 8 integer registers • But %esp and %ebp reserved for special use • Others have special uses for particular instructions • Memory: 4 consecutive bytes of memory • Various “address modes”

movl $0x4,%eax temp = 0x4; movl $-147,(%eax) *p = -147; movl %eax,%edx temp2 = temp1; movl %eax,(%edx) *p = temp; movl (%eax),%edx temp = *p; movl Operand Combinations Source Dest Src,Dest C Analog • Cannot do memory-memory transfer with a single instruction Reg Imm Mem Reg movl Reg Mem Mem Reg

Simple Addressing Modes Normal (R) Mem[Reg[R]] Register R specifies memory addressmovl (%ecx),%eax Displacement D(R) Mem[Reg[R]+D] Register R specifies start of memory region Constant displacement D specifies offsetmovl 8(%ebp),%edx

Using Simple Addressing Modes swap: pushl %ebp movl %esp,%ebp pushl %ebx movl 12(%ebp),%ecx movl 8(%ebp),%edx movl (%ecx),%eax movl (%edx),%ebx movl %eax,(%edx) movl %ebx,(%ecx) movl -4(%ebp),%ebx movl %ebp,%esp popl %ebp ret Set Up void swap(int *xp, int *yp) { int t0 = *xp; int t1 = *yp; *xp = t1; *yp = t0; } Body Finish

• • • Offset 12 yp 8 xp 4 Rtn adr %ebp 0 Old %ebp -4 Old %ebx Understanding Swap void swap(int *xp, int *yp) { int t0 = *xp; int t1 = *yp; *xp = t1; *yp = t0; } Stack Register Variable %ecx yp %edx xp %eax t1 %ebx t0 movl 12(%ebp),%ecx # ecx = yp movl 8(%ebp),%edx # edx = xp movl (%ecx),%eax # eax = *yp (t1) movl (%edx),%ebx # ebx = *xp (t0) movl %eax,(%edx) # *xp = eax movl %ebx,(%ecx) # *yp = ebx

Address %eax %edx %ecx %ebx %esi %edi %esp %ebp 0x104 Understanding Swap 123 0x124 456 0x120 0x11c 0x118 Offset 0x114 yp 12 0x120 0x110 xp 8 0x124 0x10c 4 Rtn adr 0x108 0 %ebp 0x104 -4 0x100 movl 12(%ebp),%ecx # ecx = yp movl 8(%ebp),%edx # edx = xp movl (%ecx),%eax # eax = *yp (t1) movl (%edx),%ebx # ebx = *xp (t0) movl %eax,(%edx) # *xp = eax movl %ebx,(%ecx) # *yp = ebx

Address %eax %edx %ecx %ebx %esi %edi %esp %ebp 0x104 Understanding Swap 123 0x124 456 0x120 0x11c 0x118 Offset 0x114 yp 12 0x120 0x120 0x110 0x120 xp 8 0x124 0x10c 4 Rtn adr 0x108 0 %ebp 0x104 -4 0x100 movl 12(%ebp),%ecx # ecx = yp movl 8(%ebp),%edx # edx = xp movl (%ecx),%eax # eax = *yp (t1) movl (%edx),%ebx # ebx = *xp (t0) movl %eax,(%edx) # *xp = eax movl %ebx,(%ecx) # *yp = ebx

Address %eax %edx %ecx %ebx %esi %edi %esp %ebp Understanding Swap 123 0x124 456 0x120 0x11c 0x118 Offset 0x124 0x114 yp 12 0x120 0x110 0x120 xp 8 0x124 0x124 0x10c 4 Rtn adr 0x108 0 %ebp 0x104 -4 0x100 movl 12(%ebp),%ecx # ecx = yp movl 8(%ebp),%edx # edx = xp movl (%ecx),%eax # eax = *yp (t1) movl (%edx),%ebx # ebx = *xp (t0) movl %eax,(%edx) # *xp = eax movl %ebx,(%ecx) # *yp = ebx 0x104

Address %eax %edx %ecx %ebx %esi %edi %esp %ebp Understanding Swap 123 0x124 456 456 0x120 0x11c 456 0x118 Offset 0x124 0x114 yp 12 0x120 0x110 0x120 xp 8 0x124 0x10c 4 Rtn adr 0x108 0 %ebp 0x104 -4 0x100 movl 12(%ebp),%ecx # ecx = yp movl 8(%ebp),%edx # edx = xp movl (%ecx),%eax # eax = *yp (t1) movl (%edx),%ebx # ebx = *xp (t0) movl %eax,(%edx) # *xp = eax movl %ebx,(%ecx) # *yp = ebx 0x104

Introduction & Machine -Level Programming