Basic System Administration

1. Basic System Administration Your daily commands as root

2. Becoming root • Avoid login as root over network (denied by default) • Use "/bin/su -" command from your regular account - “-” runs user LOGIN scripts - root can su to any userid without password. - Note utility for NOLOGIN ID’s. • $HOME is sometimes not “/root” but instead “/”; so watch what you delete! • Remove the current working directory (".") from your PATH • Never execute any regular user's program as root (possible Trojan Horse) • Use SSH, not TELNET over network to avoid sniffers

3. root access - sudo • visudo (as root) creates /etc/sudoers file in the following format: usernames/group servername = (usernames command can be run as) command • To implement: sudo <command> • Groups are the same as user groups and are differentiated from regular users by a % at the beginning. The Linux user group "users" would be represented by %users. • You can have multiple usernames per line separated by commas. • Multiple commands also can be separated by commas. Spaces are considered part of the command. • The keyword ALL can mean all usernames, groups, commands and servers. • If you run out of space on a line, you can end it with a back slash (\) and continue on the next line. • sudo assumes that the sudoers file will be used network wide, and therefore offers the option to specify the names of servers which will be using it in the servername position in Table 9-1. In most cases, the file is used by only one server and the keyword ALL suffices for the server name. • The NOPASSWD keyword provides access without prompting for your password. • Same concept as the SUID bit in permissions (4000).

4. sudo examples • sudocommand … to run the command as USER Examples: - user1 localhost=/sbin/halt … user1 can halt local system [user1@student1]$ sudo /sbin/halt password: [user1@student1]$ System going down now! - user2ALL= NOPASSWD: /sbin/halt … user2 can halt any system w/o password - user3 instructor = /usr/sbin/* … user 3 can run any command in /usr/sbin

5. System Administration tools • man: Sections - 1 commands, 2 system calls, 3 C library routines, 4 devices and networks, 5 file formats, 6 games and demos, 7 miscellaneous, 8 system administration • info – textinfo man page • vi editor (front-end to a lot of utilities) • su, sudo • df/du, mount • dump/restore, dd, cpio, tar, rmt, find, rsync • ps, at, batch, crontab, anacron, watch, kill, nice, nohup, killall, • useradd, usermod, userdel • groupadd, groupmod, groupdel • who, whoami • syslog • system configuration files – /etc

6. System information • hostname • uname –a • dmesg • who, whoami • last (reboot) • which, where cmd • hwclock • date • ulimit (user limits) • sysctl (system limits/settings) • cgroups • /etc/sysconfig • /etc/security • /proc • ps, pstree,

7. System monitoring • sar • pmap • vmstat, • mpstat • iostat • nstat (network), • pidstat • free • lsof • top, ntop, iftop, latencytop • ulimit –a (view), ulimit –n (set) ulimits –Hn, ulimits -Sn … per user limits, /etc/security/limits.* • See /etc/security/limits.conf: sysctl –q (view), sysctl –w (set) … system limits, /etc/sysctl.conf • strace (debugging)

8. /etc/sysconfig • The /etc/sysconfig directory is where many of the files that control the system configuration are stored for daemon processes or system services like networking. Contents vary depending on products installed. /etc/sysconfig is usually “sourced” (. /etc/sysconfig) in SYSV startup scripts. • Files in the /etc/sysconfig/ Directory amd , apmd ‘ authconfig , clock , desktop , devlabel, dhcpd, firstboot, gpm, harddisks, hwconf, i18n, init, ip6tables-config , iptables-config, irda, keyboard , kudzu, mouse, named, netdump, network, network-scripts, ifup-xxxx, ntpd, pcmcia, radvd, rawdevices, selinux, logrotate, samba , sendmail, spamassassin, squid , tux , vncservers, xinetd

9. /proc • /proc is a virtual filesystem. It's sometimes referred to as a process information pseudo-file system. It doesn't contain 'real' files but runtime system information (e.g. system memory, devices mounted, hardware configuration, etc) for all processes started by init including PID and startup commands. /proc was developed as a LINUX extension to keep track of all the complex processes started in the system • For this reason it can be regarded as a control and information centre for the kernel. In fact, quite a lot of system utilities are simply calls to files in this directory. For example, 'lsmod' is the same as 'cat /proc/modules' while 'lspci' is a synonym for 'cat /proc/pci'. By altering files located in this directory you can even read/change kernel parameters (sysctl) while the system is running. • The most distinctive thing about files in this directory is the fact that all of them have a file size of 0, with the exception of kcore, mtrr and self.

10. /etc/security • Central directory for system defaults • The limits.conf  file defines process resource limits for users.  (see ulimit) • opasswd - Store old passwords. • access.conf used to allow or restrict access to the system.  • chroot.conf used to restrict users to there home directories • console.apps contains files which are same as service names. • console.perms and console.perms.d directory determine the permissions that will be given to • The rest are PAM (Program Authentication Module) related.

11. sysctl – system limits • sysctl –q, sysctl –w, sysctl –p file, sysctl -A • /etc/sysctl.conf For network: # Enable IP spoofing protection • net.ipv4.conf.all.rp_filter=1 # Disable IP source routing • net.ipv4.conf.all.accept_source_route=0 # Ignoring broadcasts request • net.ipv4.icmp_echo_ignore_broadcasts=1 net.ipv4.icmp_ignore_bogus_error_messages=1 • # Make sure spoofed packets get logged net.ipv4.conf.all.log_martians = 1 • # disable IPv6 net.ipv6.conf.all.disable_ipv6=1 Kernel isolation (test carefully on test system): • Turn on execshield kernel.exec-shield=1 kernel.randomize_va_space=1

12. ulimit – user limits • ulimit- set user limits • -c maximum core file size (in 512-byte blocks) • -d maximum size of data segment or heap (in kbytes) • -f maximum file size (in 512-byte blocks) • -n maximum file descriptor plus 1 • -s maximum size of stack segment (in kbytes) • -t maximum CPU time (in seconds) • -v maximum size of virtual memory (in kbytes) • -S soft limit • -H hard limit • /etc/security/limits.conf

13. c(ontrol)groups • Cgroups allow you to allocate resources—such as CPU time, system memory, network bandwidth, or combinations of these resources—among user-defined groups of tasks (processes) running on a system. • A *cgroup* associates a set of tasks with a set of parameters for one or more subsystems. A *subsystem* is a module that makes use of the task grouping facilities provided by cgroups to treat groups of tasks in particular ways. A subsystem is typically a "resource controller“ in a hierarchy of processes. • A cgroup is mounted as a virtual filesystem and can be modified to re-alllocate kernel resources. Each cgroup is represented by a directory in the cgroup file system containing the following files describing that cgroup: - tasks: list of tasks (by pid) attached to that cgroup - releasable flag: cgroup currently removeable? - notify_on_release flag: run the release agent on exit? - release_agent: the path to use for release notifications (this file exists in the top cgroup only) Other subsystems such as cpusets may add additional files in each cgroup dir.

14. PAM • Pluggable Authentication Module • Centralized authentication mechanism • “Plug in” different authentication methods • Different services can have different authentication policies • Highly secure systems can require multiple passwords to authenticate

15. PAM Framework Applications ftp login ssh PAM Library libpam conf Modules pam_unix pam_ldap pam_securetty

16. Process info: ps –ef (System V) • Common options: -e print all processes -f print full listing -l long listing (more info than -f) • pstree • Meaning of full listing columns: S state PRI priority SZ total size (in 4096 byte pages) of the process RSS total resident size (in pages) of the processTIME starting time TIME cumulative execution time

17. PAM Stack • Modules are stacked (order is important) • Sample PAM configuration in /etc/pam.d: interface control flag module name auth required pam_nologin.so auth required pam_securetty.so auth sufficient pam_unix.so auth required pam_ldap.so

18. Security Enhanced LINUX • Kernel level security included since 2.60 kernel • Not an application interface. But sends access return codes to applications. • Can be combined with ACLs • /etc/selinux directory • Old gui system-config-selinux, new gui: policycureutils-gui • BE CAREFUL with changes. Especially deleting files. • See “enforcing=0” or “selinux=0” on Grub edit menu • Protects, files, processes, applications • Based on security “context” An SE Linux security context is comprised of three parts: an "identity", a "role", and a "type" for users and files or "domain" for processes. Default context for root, the role is sysadm_r, and the domain is sysadm_t. • Configuration directory: /etc/selinux directory

19. SELINUX Modes/Types • Enforcing:  enable and enforce the SELinux security policy on the system, denying access and logging actions in /var/log/audit/audit.log • Permissive:  enabled but will not enforce the security policy, only warn and log actions. Used for troubleshooting SELinux issues • Disabled:SELinux is turned off • Targeted: Specific processes • MLS/STRICT: VERY secure - systemwide

20. SELINUX Policy • Policy: a set of rules the SELinux security engine that defines types for file objects and domains for processes, and user defined (identities) roles to limit the domains that can be entered. • Strict - minimum access • Targeted – specific processes • Unconfined – not under SELINUX control, outside SELINUX context

21. SELINUX Access Control • Type Enforcement (TE): Type Enforcement is the primary mechanism of access control used in the targeted policy • Role-Based Access Control (RBAC): Based around SELinux users (not necessarily the same as the Linux user), but not used in the default targeted policy • Multi-Level Security (MLS): Not commonly used and often hidden in the default targeted policy. • Shown with –Z option (ls –Z, ps –Z etc)

22. SELINUX Commands • sestatus: show SELINUX status • getenforce: show SELINIX status • setenforce: set SELINUX status • semanage: command line policy management • chcon: change SELINUX context • restorecon: restore default SELINIX context • audit2allow: Generate SELINUX policy from /var/log/audit/audit.log • sealert: troubleshooting tool • ls –Z, ps –Z: show SELINUX context for files, processes • id: show the current user id context.

23. Process info: ps -aux(BSD) • Common options: -a print all processes involving terminals -e print environment and arguments -l long listing -u print user information -xi nclude processes with no terminals • Meaning of user information columns: %CPU percentage use of CPU SZ total size (in 1024 byte pages) of the process RSS total resident size (in pages) of the process STAT state of the process TIME time, including both user and system time

24. Process Management • at – schedule onetime batch job (scripts or commands). Example: at now +1 minutes –f somecommandfile.txt “batch” interactive at command on some systems. atq, atrm etc • anacron – (Linux) workstation scheduler. See /etc/anacrontab • watch - execute a program periodically, display results fullscreen • cron – scheduler routine must be a started process. See /etc/crontab/, /etc/cron.d, /etc/cron.hourly, /etc/cron.daily, /etc/cron.weekly - crontab [-e] [-l] [-r] [–u user] [filename]. Creates cron table “-e”uses vi syntax. - Each line contains: mm(0-59) hh(0-23) dd(1-31) mm(1-12) day (0-6, 0=Sunday**) command * - Is treated as a wild card. Meaning any possible value. */5 - Is treated as ever 5 minutes, hours, days, or months. 2,4,6 - Treated as an OR, if placed in the hours, this could mean at 2, 4, or 6 o-clock. 9-17 - Treats for any value between 9 and 17. If placed in day of month, days 9 -17. ** Note 1-7 on some UNIX systems Example */5 * * * * echo “hi there” > /dev/tty2 2>&1

25. Process Management • killpid– stop a process. -9 kills absolutely. killall • nice pid or command – set process priority • nohupcommand – run a process after logging off. Nohup.out contains job output. • “sighup” option on some commands for forcing process to reinitialize. • “zombie” processes – “kill”ed or abended processes with no parent. Usually requires a reboot to reclaim resources. Can cause system instability. See also: Ctrl Z, Ctrl C, fg, bg, &

26. Backup/Restore • dump/restore - backs up file systems, has interactive mode, can do incremental backups, maintains "sparse files", is most commonly used utility • cpio - can back up individual files/directories, handles special files, packs data tighter than tar, skips bad spots on media on restore, use with find (some versions of find have -cpio option for this purpose) • tar - backs up directory trees, does not back up special files, poor error handling with media errors, does not pack blocks (GNU tar solves some of these problems). Some LINUX/UNIX systems have built-in compress with –z flag. • dd - copies/converts files, can go from one medium to another, processes whole entity or select blocks, can swap bytes and do ASCII/EBCDIC conversions. Performs physical backup of raw devices. • rmt - used for remote tape operations. Varies by OS. • rsync – used for directory synchronization, e.g. “hot” folders

27. Backup Strategy • Physical (dd, cpio) – usually devices (as root) dd if=devicefile if=outputfile bs=blocksize count=#blocks • Logical (rmt, tar, dump/restore). Backup marker. create: tar –cvf tarfilename.tar[directory list] | compress list: tar –tvf tarfilename.tar extract: tar –xvf tarfilename.tar • Can use logical backups in conjunction with find command –exec option (next panel) for differential or incremental backups • Backup types: Full (everything) Incremental (Difference since last backup) Differential (Difference since last full backup) Full + Incremental or Differential = Backup set

28. find • Syntax: find starting-dir(s) matching-criteria-and-actions Matching criteria -atime n file was accessed n days ago -mtime n file was modified n days ago -size n file is exactly n 512-byte blocks -type c file type (e.g., f=plain, d=dir) -name nam file name (e.g., `*.c') -user usr file's owner is usr -perm p file's access mode is p -print display pathname -exec cmd execute command ({} expands to file) • find examples find . -name \*.c -print find / -size +1000 -mtime +30 \ -exec ls -l {} \; find / \( -name a.out -o -name core \ -o -name' \) -type f -atime +14 \ -exec rm -f {} \; find / \( -perm 2000 -o -perm 4000 \) \ -print | diff - files.secure

29. Disk management • df • mount / umount • du | sort –rn | more • find / -name core -exec rm -f {} \; • Filesystems: /home, /var, /tmp (noexec), / (never full!) • mkdev, mkfs, fdisk

30. User management • Set system account parameters (e.g., password aging, account expiration, quotas, login scripts - /etc/profile, /etc/bashrc etc) • Determine login name, user ID (UID), group ID (GID) • Assign password (passwd) • /etc/passwd - logname:passwd:uid:gid:user info:home:shell • Passwords stored in /etc/shadow (pwconv) • Commands: useradd, usermod, userdel, chage, passwd • /etc/group: group:passwd:gid:members • Commands: groupadd, groupmod, groupdel

31. User Security • ALWAYS use /etc/shadow (pwconv command) • Password aging: get: chage -l userid set: chage -M 60 -m 7 -W 7 userid • Lock/unlock and account passwd –l userid passwd –u userid • Limit password reuse: vi /etc/pam.d/system-auth (RHEL/Fedora) vi /etc/pam.d/common-password (Ubuntu) Add: password sufficient pam_unix.so use_authtok md5 shadow remember=10 • Verify root ID’s: awk -F: '($3 == "0") {print}' /etc/passwd • Verify no password ID’s: awk -F: '($2 == "") {print $1 }' /etc/shadow | grep /etc/passwd Make sure they are /bin/nologin • No ROOT LOGIN, use su or sudo. • Disable ROOT login under FTP., SSH, GUI (procedures vary) • Configure password policy (LINUX: see pam_cracklib.so, other vary). • See defaults in /etc/login.defs for /etc/shadow values.

32. SYSLOG • (r)syslog is a utility for tracking and logging all manner of system messages from the merely informational to the extremely critical. Sysogs stored in • In LINUX, system logs are stored in /var/log. System messages are recorded in /var/log/messages. Other OS’es may use different files in different directories (e.g. /var/adm). • Each system message sent to the syslog server has two descriptive labels associated with it that makes the message easier to handle. - The first describes the function (facility) of the application that generated it. For example, applications such as mail and cron generate messages with easily identifiable facilities named mail and cron. - The second describes the degree of severity of the message.

33. SYSLOG • Severity: Level: Keyword: Description • 0 emergencies System unusable • 1 alerts Immediate action required • 2 critical Critical condition • 3 errors Error conditions • 4 warnings Warning conditions • 5 notifications Normal but significant conditions • 6 informational Informational messages • 7 debugging Debugging messages

34. SYSLOG • configuration file /etc/rsyslog.conf or/etc/syslog.conf. • File consists of two columns. - First lists the facilities and severities of messages to expect - Second lists the files to which they should be logged. - LINUX default directory is /var/log • Example: *.info;mail.none;authpriv.none;cron.none /var/log/messages • Note other services may record messages in other files (e.g. sendmail) • Syslog is also a network service. A common implementation is to forward system info to a common syslog server. TCP or UDP can be used. • Logs compressed, ,stored and optionally e-mailed by the logrotate function. Definitions stored in /etc/logrotate.conf and /etc/logrotate.d

35. System shutdown • Shutdown will run SysV K* scripts. • shutdown {–h|-r} {time in minutes|now} • See also wall command • Other commands: halt, reboot, Ctrl-Alt-Del may bypass some processing. Not recommended for production systems. • Reboots recorded in /var/log/wtmp or utmp • last (reboot) displays info