Introduction to UNIX UNIX & LINUX Fundamentals

1. Introduction to UNIXUNIX & LINUX Fundamentals

2. Purpose of This Lecture • Fundamentals of using UNIX and UNIX-like systems • History of UNIX & Linux • Basic system commands • Data Management • Constructing basic shell scripts

3. History of UNIX • "...the number of UNIX installations has grown to 10, with more expected..." • Dennis Ritchie and Ken Thompson, June 1972 • "... When BTL withdrew from the project, they needed to rewrite an operating system (OS) in order to play space war on another smaller machine (a DEC PDP-7 [Programmed Data Processor] with 4K memory for user programs). The result was a system which a punning colleague called UNICS (UNiplexed Information and Computing Service)--an 'emasculated Multics'; no one recalls whose idea the change to UNIX was"

4. Early Movers and Shakers • Dennis Ritchie & Ken Thompson • PDP11

5. And then there was C… • in 1972, Ritchie rewrote B and called the new language C; Thompson created the pipe--a "Write programs that do one thing and do it well. Write programs to work together. Write programs that handle text streams, because that is a universal interface."

6. Bringing UNIX to the desktop • Unix was very expensive • Microsoft DOS was the mainstream OS • MINIX, tried but was not a full port • An Open Source solution was needed!

7. Linux 0.02 – October 5, 1991 • “Do you pine for the nice days of minix-1.1, when men were men and wrote their own device drivers? Are you without a nice project and just dying to cut your teeth on a OS you can try to modify for your needs? Are you finding it frustrating when everything works on minix? No more all-nighters to get a nifty program working? Then this post might be just for you :-)” - Linus Benedict Torvalds • "I still maintain the point that designing a monolithic kernel in 1991 is a fundamental error. Be thankful you are not my student. You would not get a high grade for such a design :-)" (Andrew Tanenbaum to Linus Torvalds)

8. 1990’s Movers and Shakers • Richard Stallman, father of the GNU Project • Linus Torvalds

9. Why UNIX/LINUX is Still Used • 30+ years of development • Many academic, scientific, and system tools • Open Source • System Stability • Lightweight • Easy Development

10. The Basics • Command-Line • Interaction with UNIX & LINUX is based on entering commands to a text terminal • Oftentimes there are no ‘warnings’ with commands, no ‘undo’ • The Shell • The user environment that enables interaction with the kernel, or lower-system OS. • Windows Explorer would be a shell for Microsoft Windows.

11. Common Shells • sh – the original UNIX shell; still located in /bin/sh • Bash is a Unix shell written for the GNU Project and is installed on most Linux systems. • csh – C Shell, modeled after the C programming language used by UNIX systems • tcsh – C Shell with modern improvements such as filename completion • echo $SHELL – displays what shell your account is using • chsh – change your shell

12. Before we go further… • Read the Manual. • man command • man [section] command • man –k keyword (search all manuals based on keyword) • Most commands have a built-in UNIX manual, even the man command! • Commands without manuals have help too, with –h, --help, or /? option.

13. The Manual • The manual pages are divided into eight sections depending on type of command. • 1 commands and applications • 2 system calls • 3 C library functions • 4 special files • 5 file formats • 6 games • 7 misc. • 8 system administration utilities

14. Conventions for this lecture • This lecture is too short to give you all of the options, so look at the manual for specific syntax for commands. • Commands will be in bold, options will be in italics. • command-arguments • Output will be shown in its own bordered table

15. Command Conventions • In help files and manuals, commands will have required input and option input • cp[OPTION] source destination • Optional arguments are in brackets, required arguments are not. • cp –R or cp - -recursive • Short options ‘-’, long options ‘- -’

16. Who’s Path is it Anyway? • UNIX treats the directory structure as a hierarchy of individual paths / dev home usr bhecker bin

17. Directories • What is a working directory? • The directory your shell is currently associated with. At anytime in the system your login is associated with a directory • pwd – view the path of your working directory • ls – view your working directory

18. Finding your home • Each user has a home directory which can be found with: • cd • cd~bhecker • cd$HOME • The tilde character ‘~’ will tell the shell to auto-complete the path statement for the cd command • $HOME refers to an environment variable which contains the path for home.

19. More file commands • cddirectory - change your current working directory to the new path • ls–a – Show hidden files • Hidden files are files that begin with a period in the filename ‘.’ • mv • moves one file to another • cp • copies files or directories • rm • remove files & directories • rm –rf • remove everything with no warnings • rm–rf * • most dangerous command you can run!

20. Recursive Directories • Often times a manual will refer to ‘recursive’ actions on directories. This means to perform an action on the given directory and recursively to all subdirectories. • cp–R source destination • Copy recursively all directories under source to destination

21. Relative vs. Absolute Path • Commands expect you to give them a path to a file. Most commands will let you provide a file with a relative path, or a path relative to your working directory. • ../directory - the ‘..’ refers to looking at our previous directory first • ./executable - ‘.’ says this directory, or our working directory • Absolute, or Full paths are complete. An easy way to know if a path is absolute is does it contain the ‘/’ character at the beginning? • /home/user/directory/executable - a full path to file executable

22. Poking around in $home • How much space do I have? • Quota • command to see all quotas for your directories are, if any. • How much space am I taking up? • du • command to find out how much space a folder or directory uses. • df • display space information for the entire system

23. Helpful Hints on Space • Almost all commands that deal with file space will display information in Kilobytes, or Bytes. Nobody finds this useful. • Many commands will support a ‘-h’ option for “Human Readable” formatting. • ls –lh • displays the working directory files with a long listing format, using “human readable” notation for space

24. Representing Space • Bit – either a 1 or 0 • Byte – 8 bits “0000 1111”, or “x0F”, or 16 • KB, Kilobyte - 1024 Bytes • MB, Megabyte – 1024KB • GB, Gigabyte – 1024MB • TB, Terabyte – 1024 GB • PB, Petabyte – 1024 TB

25. Permissions • The *NIX systems are multi-user environments where many users run programs and share data. Files and directories have three levels of permissions: World, Group, and User. • The types of permissions a file can contain are:

26. File permissions are arranged in three groups of three characters. In this example the owner can read & write a file, while others have read access Permissions Cont.

27. Changing Permissions • chmod – change permissions on a file or directory • chown – change file ownership to another user • Both options support ‘-R’ for recursion.

28. All About Me • Every user id corresponds to a unique user or system process • whoami – returns the use rid of the current user • passwd – change password • What is my group? – support!

29. What everyone else is up to • top – show a detailed, refreshed, description of running processes on a system. • uptime – show the system load and how long the system has been up. • ‘load’ is a number based on utility of the cpu’s of the system. A load of 1 indicates full load for one cpu.

30. Working With Programs • Commands or programs on the system are identified by their filename and by a process ID which is a unique identifier. • ps • Display process information on the system • killpid • Terminates the process id • ^c (control+c) terminates the running program • ^d (control+d) terminates your session. • Only you and the superuser (root) has permissions to kill processes you own.

31. Advanced Program Options • Oftentimes we must run a command in the background with the ampersand ‘&’ character • command–options & • Runs command in background, prompt returns immediately • Match zero or more characters wildcard ‘*’ • cp* destination • Copy everything to destination • This option can get you into trouble if misused

32. Editing Files • emacs vs vi • Among the largest ‘nerd battle’ in history. emacs relies heavily on key-cords (multiple key strokes), while vi is mode based. (editor mode vs command mode) • vi users tend to enter and exit the editor repeatedly, and use the Unix shell for complex tasks, whereas emacs users usually remain within the editor and use emacs itself for complex tasks • pico (nano) • editor originally used for the email client pine, simple no-frills editor which resembles notepad for windows.

33. Input and Output • Programs and commands can contain an input and output. These are called ‘streams’. UNIX programming is oftentimes stream based. • Programs also have an error output. We will see later how to catch the error output. • STDIN – ‘standard input,’ or input from the keyboard • SDTOUT – ‘standard output,’ or output to the screen • STDERR – ‘standard error,’ error output which is sent to the screen.

34. File Redirection • Oftentimes we want to save output (stdout) from a program to a file. This can be done with the ‘redirection’ operator. • myprogram> myfile – using the ‘>’ operator we redirect the output from myprogram to file myfile • Similarly, we can append the output to a file instead of rewriting it with a double ‘>>’ • myprogram>> myfile – using the ‘>’ operator we append the output from myprogram to file myfile

35. Input Redirection • Input can also be given to a command from a file instead of typing it to the screen, which would be impractical. • catprograminput > mycommand - This command series starts with the command ‘cat’ which prints a file to a screen. programinput is printed to stdout, which is redirected to a command mycommand.

36. Redirecting stderr • Performing a normal redirection will not redirect sdterr. In Bash, this can be accomplished with ‘2>’ • command2> file1 • Or, one can merge stderr to stdout (most popular) with ‘2>&1’ • command> file 2>&1

37. Pipes • Using a pipe operator ‘|’ commands can be linked together. The pipe will link the standard output from one command to the standard input of another. • Very helpful for searching files

38. Searching • A large majority of activity on UNIX systems involve searching for files and information. • find – utility to find files • grep – the best utility ever written for UNIX, searches for patterns inside files and will return the line, if found

39. Packing Files • When creating backups of files, or transferring to other hosts, files must be packed into larger files. This is needed for ease of manipulation, transfer speeds, and file management. • tar – create or extract a packed file. tar stands for ‘tape archive’.

40. Compressing Files • Compressing files can gain file space at the expense of cpu time to compress and decompress files. • Compression works well for text files, but not as well for binary files with random data such as float values. • Compression algorithms & commands • gzip, gunzip, bzip2, bunzip2

41. Using tar to create compressed files • Tar will create compressed files for you • tar–czvf mytarfile.tar.gz directory • creates a compressed file named mytarfile.tar.gz containing all of the files in the directory directory • tar–xzvf mytarfile.tar.gz • uncompresses all directories and files inside the file mytarfile.tar.gz into the working directory

42. Connecting to Another Machine • Secure Shell vs. Restricted Shell • ssh is an encrypted remote login program that is ‘secure’ to trust across non secure networks. • ssh userid@hostname

43. Copying Files to Remote Hosts • copy local file lfile to rfile on remote machine rsys • scp lfile rsys:rfile • -p preserves modification time, access time and mode from original • scp -p lfile rsys:rfile • copy rfile from remote machine rsys tolocal file lfile • scp -p rsys:rfile lfile

44. Running Commands on a Remote Host • Commands can be executed on a remote host with ssh • sshuserid@hostname “ls” • Run ls on remote host hostname

45. My Environment • View all system variables by the command ‘env’ • Depending on shell, startup commands can be managed with the files .profile for bash and .cshrc with c shell

46. Basic Shell Scripts • Many times it is helpful to create a ‘script’ of commands to run instead of typing them in individually. Scripts can be made to aid in post-processing, system administration, and automate menial tasks • #!/bin/bash • First statement inside a script, will list which shell to run this script in • # - says what will follow is a comment and not to execute

47. Variables • By convention system variables are capitalized • HOME – location of the home directory • OLDPWD – location of the previous working directory • PATH – locations to look inside for executable files • Setting system variables differs by shell. bash uses export, csh uses setenv • User defined variables in scripts are lower-case by convention • myvariable=10 • sets myvariable to 10 • echo $myvariable • prints myvariable

48. Conditionals if condition then condition is zero (true - 0) execute all commands up to else statement else if condition is not true then execute all commands up to fi fi

49. Performing Loops • Loops are statements that are repeated until the conditions are met. for { variable name } in { list } do execute one for each item in the list until the list is not finished (And repeat all statement between do and done) done for i in 1 2 3 4 5 doecho "Welcome $i times“ done

50. Putting it Together #!/bin/bash #my first script #scp replacement remotefile=labresults myserver=saguaro.fulton.asu.edu mylsinfo=`ssh $myserver ls $remotefile 2>&1` ismissing=`echo $mylsinfo | grep ERROR` if [ "$ismissing" ] then echo "$remotefile not found! Exiting!" else ssh -n "cat < $remotefile" > $localfile Fi