Unix Kernel Compilation and 64bit computing

Unix Kernel Compilationand 64bit computing Guntis Barzdins Girts Folkmanis

Kernel Overview

Kernel source • Download source fromwww.kernel.orgor Mirrors. • Unpack: • cd /usr/src • tar xzvf linux-<Version>.tar.gz • ln –s linux-<Version>.tar.gz linux • Source-Root: • /usr/src/linux

Linux Source Tree Layout /usr/src/linux scripts Documentation ipc kernel init net arch mm lib fs drivers include 802 appletalk atm ax25 bridge core decnet econet ethernet ipv4 ipv6 ipx irda khttpd lapb … acorn atm block cdrom char dio fc4 i2c i2o ide ieee1394 isdn macintosh misc net … adfs affs autofs autofs4 bfs code cramfs devfs devpts efs ext2 fat hfs hpfs … asm-alpha asm-arm asm-generic asm-i386 asm-ia64 asm-m68k asm-mips asm-mips64 linux math-emu net pcmcia scsi video … adfs affs autofs autofs4 bfs code cramfs devfs devpts efs ext2 fat hfs hpfs … alpha arm i386 ia64 m68k mips mips64 ppc s390 sh sparc sparc64

Sizes (linux-2.4.0-test2) size directory entries files loc 90M /usr/src/linux/ 19 7645 2.6M 4.5M Documentation 97 380 na 16.5M arch 12 1685 466K 54M drivers 31 2256 1.5M 5.6M fs 70 489 150K 14.2M include 19 2262 285K 28K init 2 2 1K 120K ipc 6 6 4.5K 332K kernel 25 25 12K 80K lib 8 8 2K 356K mm 19 19 12K 5.8M net 33 453 162K 400K scripts 26 42 12K

linux/arch • Subdirectories for each current port. • Each contains kernel, lib, mm, boot and other directories whose contents override code stubs in architecture independent code. • lib contains highly-optimized common utility routines such as memcpy, checksums, etc. • arch as of 2.4: • alpha, arm, i386, ia64, m68k, mips, mips64. • ppc, s390, sh, sparc, sparc64.

linux/drivers • Largest amount of code in the kernel tree (~1.5M). • device, bus, platform and general directories. • drivers/char – n_tty.c is the default line discipline. • drivers/block – elevator.c, genhd.c, linear.c, ll_rw_blk.c, raidN.c. • drivers/net –specific drivers and general routines Space.c and net_init.c. • drivers/scsi – scsi_*.c files are generic; sd.c (disk), sr.c (CD-ROM), st.c (tape), sg.c (generic). • General: • cdrom, ide, isdn, parport, pcmcia, pnp, sound, telephony, video. • Buses – fc4, i2c, nubus, pci, sbus, tc, usb. • Platforms – acorn, macintosh, s390, sgi.

linux/fs • Contains: • virtual filesystem (VFS) framework. • subdirectories for actual filesystems. • vfs-related files: • exec.c, binfmt_*.c - files for mapping new process images. • devices.c, blk_dev.c – device registration, block device support. • super.c, filesystems.c. • inode.c, dcache.c, namei.c, buffer.c, file_table.c. • open.c, read_write.c, select.c, pipe.c, fifo.c. • fcntl.c, ioctl.c, locks.c, dquot.c, stat.c.

linux/include • include/asm-*: • Architecture-dependent include subdirectories. • include/linux: • Header info needed both by the kernel and user apps. • Usually linked to /usr/include/linux. • Kernel-only portions guarded by #ifdefs • #ifdef __KERNEL__ • /* kernel stuff */ • #endif • Other directories: • math-emu, net, pcmcia, scsi, video.

linux/init • Just two files: version.c, main.c. • version.c – contains the version banner that prints at boot. • main.c – architecture-independent boot code. • start_kernel is the primary entry point.

linux/ipc • System V IPC facilities. • If disabled at compile-time, util.c exports stubs that simply return –ENOSYS. • One file for each facility: • sem.c – semaphores. • shm.c – shared memory. • msg.c – message queues.

linux/kernel • The core kernel code. • sched.c – “the main kernel file”: • scheduler, wait queues, timers, alarms, task queues. • Process control: • fork.c, exec.c, signal.c, exit.c etc… • Kernel module support: • kmod.c, ksyms.c, module.c. • Other operations: • time.c, resource.c, dma.c, softirq.c, itimer.c. • printk.c, info.c, panic.c, sysctl.c, sys.c.

linux/lib • kernel code cannot call standard C library routines. • Files: • brlock.c – “Big Reader” spinlocks. • cmdline.c – kernel command line parsing routines. • errno.c – global definition of errno. • inflate.c – “gunzip” part of gzip.c used during boot. • string.c – portable string code. • Usually replaced by optimized, architecture-dependent routines. • vsprintf.c – libc replacement.

linux/mm • Paging and swapping: • swap.c, swapfile.c (paging devices), swap_state.c (cache). • vmscan.c – paging policies, kswapd. • page_io.c – low-level page transfer. • Allocation and deallocation: • slab.c – slab allocator. • page_alloc.c – page-based allocator. • vmalloc.c – kernel virtual-memory allocator. • Memory mapping: • memory.c – paging, fault-handling, page table code. • filemap.c – file mapping. • mmap.c, mremap.c, mlock.c, mprotect.c.

linux/scripts • Scripts for: • Menu-based kernel configuration. • Kernel patching. • Generating kernel documentation.

Linux Kernel Configuration • Download the source code and extract it under /usr/src • Customize kernel configuration • make config • make menuconfig • make xconfig • Two menu items: Networking options and Network device support • Device support • Three options: y, m, n

Kernel Config • Get sources from kernel.org • Unpack in your home directory gzip -cd linux-2.4.XX.tar.gz | tar xvf - • make menuconfig • make dep • make bzimage

Building the Kernel • cd /usr/src/linux-2.4.20-8 • make xconfig • This command runs a X-based configuration tool that asks you specific question about every kernel configuration.

Building and Installing Kernel • Compiling the kernel. • make dep • [ make clean ] • make bzImage • make modules • Installing the kernel. • make modules_install • [ make install ] • Kernel stored in /usr/src/linux-2.4/arch/i386/boot/bzImage and copied to /boot • cp arch/i386/boot/bzImage /boot/ • Edit your lilo.conf and run /sbin/lilo • Reboot.

Building the Kernel (Cont’d) • make xconfig • Most kernel features have three compilation options: Y (compiling the option directly into the kernel), N (not compiling the option at all), and M (compiling the option as a kernel module. • After saving the selection, the configuration file /user/src/linux-2.4.20-8/.config is created.

Building the Kernel (Cont’d) • make dep • Creates dependency information, so that the compiler knows each component’s dependencies and can compile components as appropriate. • make clean • Cleans up some miscellaneous object files. • make bzImage • Compiles the Linux kernel properly. • The result is a kernel file called bzImage and located in /user/src/linux-2.4.20-8/arch/i386/boot • make modules • Compiles the kernel modules files

Building the Kernel (Cont’d) • make modules_install • Installs the kernel modules into the directory path /lib/modules/2.4.20-8/kernel/drivers. • make install • Copies the new kernel and its associated files to /boot directory. • Builds a new initrd image and adds new entries to the boot loader configuration file. • Use the command ls -l /boot to make sure the initrd-2.4.20-8.img file was created. • Confirm that the file /boot/grub/grub.conf contains a title section with the same version as the kernel package just installed

initrd • Initial RAM disk called “initrd” : a single file to make the work ofboot loader simple

unix boot # pwd; ls -lRp /boot .: total 1582 lrwxrwxrwx 1 root root 1 Sep 23 14:11 boot -> ./ drwxr-xr-x 2 root root 1024 Sep 23 15:34 grub/ -rw-r--r-- 1 root root 458622 Sep 23 14:58 initrd-2.4.26-gentoo-r9 -rw-r--r-- 1 root root 1137878 Sep 23 14:50 kernel-2.4.26-gentoo-r9 drwx------ 2 root root 12288 Sep 23 13:49 lost+found/ ./grub: total 846 -rw-r--r-- 1 root root 30 Sep 23 15:34 device.map -rw-r--r-- 1 root root 11264 Sep 23 15:34 e2fs_stage1_5 -rw-r--r-- 1 root root 10256 Sep 23 15:34 fat_stage1_5 -rw-r--r-- 1 root root 9216 Sep 23 15:34 ffs_stage1_5 -rw-r--r-- 1 root root 245 Sep 23 15:34 grub.conf -rw-r--r-- 1 root root 1495 Sep 23 15:32 grub.conf.sample -rw-r--r-- 1 root root 11456 Sep 23 15:34 jfs_stage1_5 lrwxrwxrwx 1 root root 9 Sep 23 15:32 menu.lst -> grub.conf -rw-r--r-- 1 root root 9600 Sep 23 15:34 minix_stage1_5 -rwxr-xr-x 1 root root 196836 Sep 23 15:32 nbgrub -rwxr-xr-x 1 root root 197860 Sep 23 15:32 pxegrub -rw-r--r-- 1 root root 12864 Sep 23 15:34 reiserfs_stage1_5 -rw-r--r-- 1 root root 33856 Sep 23 15:32 splash.xpm.gz -rw-r--r-- 1 root root 512 Sep 23 15:34 stage1 -rw-r--r-- 1 root root 135148 Sep 23 15:34 stage2 -rwxr-xr-x 1 root root 196900 Sep 23 15:32 stage2.netboot -rw-r--r-- 1 root root 8896 Sep 23 15:34 vstafs_stage1_5 -rw-r--r-- 1 root root 12840 Sep 23 15:34 xfs_stage1_5 ./lost+found: total 0 unix boot # /boot unix grub # more grub.conf default 0 # How many seconds to wait before the default listing is booted. timeout 5 title=gentoo root (hd0,0) kernel /kernel-2.4.26-gentoo-r9 root=/dev/ram0 init=/linuxrc ramdisk=8192 real_root=/dev/hda2 initrd /initrd-2.4.26-gentoo-r9 title GNU/Linux (2.4.25)root (hd0,4)kernel (hd0,4)/boot/kernel-2.4.25_pre6-gss root=/dev/ram0 init=/linuxrc real_root=/dev/hda5 vga=0x317 splash=verboseinitrd (hd0,4)/boot/initrd-2.4.25_pre6-gss unix grub #

GRUB # /etc/grub.conf generated by anaconda timeout=10 splashimage=(hd0,1)/grub/splash.xpm.gz password --md5 $1$ÕpîÁÜdþï$J08sMAcfyWW.C3soZpHkh. title Red Hat Linux (2.4.18-3custom) root (hd0,1) kernel /vmlinuz-2.4.18-3custom ro root=/dev/hda5 initrd /initrd-2.4.18-3.img title Red Hat Linux (2.4.18-3) Emergency kernel (no afs) root (hd0,1) kernel /vmlinuz-2.4.18-3 ro root=/dev/hda5 initrd /initrd-2.4.18-3.img title Windows 2000 Professional rootnoverify (hd0,0) chainloader +1

Linux Loader (LILO) # sample /etc/lilo.conf boot = /dev/hda delay = 40 password=SOME_PASSWORD_HERE default=vmlinuz-stable vga = normal root = /dev/hda1 image = vmlinuz-2.5.99 label = net test kernel restricted image = vmlinuz-stable label = stable kernel restricted other = /dev/hda3 label = Windows 2000 Professional restricted table = /dev/hda

Build kernel for other platform • unix linux # make menuconfig ARCH=x86_64 • /usr/src/linux/arch • alpha cris ia64 mips parisc ppc64 s390x sh64 sparc64arm i386 m68k mips64 ppc s390 sh sparc x86_64 • Cross-compile • make HOSTCC="gcc -m32" ARCH="x86_64" bzImage • get a gcc wrapper in order to crosscompile on a i386 host • http://www.jukie.net/~bart/debian/amd64/scripts/gcc.bart

BSD

Source Code Control • The entire source code for FreeBSD is stored in a CVS repository • The logs, and individual changes for each file can be traced back to 1994. • The source tree can be checked out at any state, or corresponding to any release • CDs are available taking the history back a further 20 years

Building World • The entire operating system, including all libraries and utilities can be built with a single command : “make world” • The kernel of the operating system is built separately with a single command: “make kernel” • The source code for the system is placed in /usr/src during installation.

Building Releases • You can even build a complete release of FreeBSD, including FTP install directories, floppy images, and ISO images for CDROMs with one command. • “make release” is used by many large companies to produce special versions of FreeBSD with special patches or additional software installed by default. • It is also the well documented way in which the release engineering team makes all official releases of FreeBSD.

Release Engineering • “make release” makes it much easier to deploy thousands of systems pre-configured for a specific environment. • The release engineering team for FreeBSD publishes schedules, identifies QA issues that must be resolved before release, and publishes documents to help other people build FreeBSD based products. • See release(7) and www.freebsd.org/releng

Running a FreeBSD binary • Code like fd = open(“/etc/passwd”, O_RDONLY); • Becomes syscall(5, ...) • Kernel knows it’s a FreeBSD binary, uses freebsd_syscalls[] array freebsd_syscalls[5] = freebsd_open(…); • File is opened

Running a Linux binary • Code like fd = open(“/etc/passwd”, O_RDONLY); • Becomes syscall(5, ...) • Kernel knows it’s a Linux binary, uses linux_syscalls[] array linux_syscalls[5] = linux_open(…); • File is opened • All Linux file operations redirected to /compat/linux first

Linux Modules

Linux modules • driver modules in /lib/module [root@dafinn net]# pwd; ls /lib/modules/2.4.22-1.2166.nptlsmp/kernel/drivers/net 3c509.o b44.o eepro100.o netconsole.o pppox.o tg3.o 3c59x.o bonding epic100.o ns83820.o ppp_synctty.o tlan.o 8139cp.o de4x5.o ethertap.o pcmcia r8169.o tulip 8139too.o dl2k.o fealnx.o pcnet32.o sis900.o tun.o 82596.o dmfe.o irda ppp_async.o sk98lin typhoon.o 8390.o dummy.o mii.o ppp_deflate.o slhc.o via-rhine.o acenic.o e100 natsemi.o ppp_generic.o smc9194.o wireless amd8111e.o e1000 ne2k-pci.o pppoe.o starfire.o • Recompiling the kernel • Make the kernel smaller • Add a new device • Modify a system parameter

Hello World ! #define MODULE #include <linux/module.h> int init_module() { printk("<1>Hello, world\n"); return 0; } void cleanup_module() { printk("<1>Goodbye !\n"); }

Loaded and Unloaded root#gcc -c hello.c root#insmod ./hello.o Hello, world root#rmmod hello Goodbye ! root# • MUST be root to load/unload module • Check /var/log/messages if nothing shown • Can use module_init(my_init) & module_exit(my_cleanup) in Linux 2.2 and later

Linux modules • List installed modules • lsmod • Module dependencies • The meaning of (autoclean) • Load the module mannually: • insmod [–k] 3c509 • modprobe smc-ultra • Generate the dependency • depmod –a • Remove module: rmmod • Device driver can be dynamically loaded to compiled into the kernel.

Function pointers

LKM Utilities cmd • insmod • Insert an LKM into the kernel. • rmmod • Remove an LKM from the kernel. • depmod • Determine interdependencies between LKMs. • kerneld • Kerneld daemon program • ksyms • Display symbols that are exported by the kernel for use by new LKMs. • lsmod • List currently loaded LKMs. • modinfo • Display contents of .modinfo section in an LKM object file. • modprobe • Insert or remove an LKM or set of LKMs intelligently. For example, if you must load A before loading B, Modprobe will automatically load A when you tell it to load B.

Two ways for loading a module • manual way : using ‘insmod’ command. • automatic way : the kernel discovers the need for loading a module (for example, user mounts a file system) => requests ‘kerneld’ daemon to load the needed module. => ‘kerneld’ loads module using insmod.

The ‘insmod’ mechanism for loading a module • reads the module into its virtual memory • fixes up unresolved references to kernel routines and resources using the exported symbols from the kernel. • requests the kernel for enough space to hold the new kernel • the kernel allocates a new module data structure and enough kernel memory to hold the new module and puts it at the end of the kernel modules list

The ‘insmod’ mechanism for loading a module • insmod copies the module into the allocated space and relocates it so that it will run from the kernel address that it has been allocated • the new module exports sysmbols to the kernel and insmod builds a table of these exported symbols • if the new module depends on another module, that module has the reference of the new module. • the kernel calls the modules initialization routine and carries on installing the module

Unloadig a module • Two ways for unloading a module • Manual way : uses ‘rmmod’ command • automatic way : when idle timer expires, the kerneld calls the service routines for all unused loaded modules • the mechanism of unloading • if the module can be unloaded, its cleanup routine is called to free up the kernel resources that it has allocated • the module data structure is unlinked from the list of kernel modules • all of the kernel memory that the module needed is deallocated.

User Space Driver • Advantages • Full C library can be linked in • Run in conventional debugger • Problem in driver unlikely to hang entire system • User memory is swapping (more to use) • Still allow concurrent access to a device for well-designed driver • Disadvantages • Interrupts are not available in user space • Direct access to memory by mmapping /dev/mem (only privileged user) • Access to I/O ports after calling ioperm or iopl (not all platform support), and access to /dev/port can be too slow (only privileged user) • Response time is slower (context switch, driver swapped to disk) • Most important devices can’t be handled in user space (block, network)

64bit support

Single Instruction, Multiple Data (SIMD)

Unix Kernel Compilation and 64bit computing