Working under your qandc account

1. Working under your qandc account

2. Connection from PC to qandc server • In order to get into the qandc servee, we need a secure shell application that facilitates the connection. • One simple software that is free and it is also downloaded in our computers at CSUSM computer labs is called putty. • PuTTY is a free implementation of Telnet and SSH (secure shell) for Windows and Unix platforms, along with an Xterm terminal that allows us to interactively with Unix operating system. • You can download putty to your home computer. The site is: • http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

3. Downloading Putty • Once you enter the web site, click on the following link that shows below:

4. Downloading Putty • Save the application anywhere in your computer. I usually put it in my desktop.

5. Installing Putty • Now double click on the downloaded putty application to get connected to the qandc server • Once you double click you should see the following. Click on the run button to start the application.

6. Connection to qandc server using Putty • Make sure the category is session. The port is 22, and connection type is SSH. Then type qandc.csusm.edu for the Host Name.

7. Connection to qandc server using Putty • Type your user id and password and you enter the UNIX environment

8. Connection to qandc server using Putty • Type “ls” at the prompt line to see the content of the current directory. • You may not have any file under your home directory at this time. • In mine, my files and folders are shown in different colors.

9. Connection to qandc server using Putty • You can remove the default colors setting, as follows. Right click on the header of the terminal and select “Change Setting…” from the menu.

10. The following window appears. • Select “Colours” from the Category section as shown below and uncheck the “Allow terminal to specify ANSI colours” option. • Then click on “Apply” button to save the setting • This gives simple black and whiter terminal.

11. Logging out from Unix • We will discuss the UNIX commands later. • For now, just type “logout” to exit from Unix and close the terminal. • In conclusion, putty is very simple to install and work with; however, it is limited to simple connection to a particular system such as qandc or cocco or bioinfo servers. • We all want to be able to download our files and place them under our qandc account and. We need to be able to transfer files into and from qandc server. • Thus, we need to have a secure shell that has the capability of transferring files as well.

12. Windows SSH Secure Shell Initial Connection and Transferring Files. • First we need to download this application. For now, you can download this under your H drive to test.. Then you can download it on your PC at home. • Go to: http://www.colorado.edu/its/docs/authenticate/printouts/win_ssh.html • Click on the link that is shown below:

13. Installing SSH Secure Shell • Save the application. You can place it under the H drive. • Double-click on the application and press “run” button to start installation. • Click “next to continue:

14. Installing SSH Secure Shell • Accept the License agreement, then click browse and under path type H:\SSH. This installs the application under your H drive . • Next follow the rest of the download procedure until it is complete. • You should be able to see the following icon, which is “SSH Secure File Transfer Client” under your H drive.

15. Running SSH Secure Shell • To run the program and getting connected to the qandc server simply double click on the application • The following window opens. • Click on “Quick connect” to start the connection to qandc server

16. Connecting to qandc server using SSH • Type “qandc.csusm.edu” for the Host Name and your username for the User Name section. Make sure the port is 22. • Click “Connect” and it should ask you for the password. • Note that since this is our private server, the password is permanent and we do not require our users to change it every few months. It is up to you what to set as your password and when you prefer to change it.

17. Connecting to qandc server using SSH • Once you are successfully connected, you should see the content of your PC on the left side of the window and the content of your qandc server on the right side of the window, as shown below • The same way you navigate through your folders in your PC, you can navigate through your folders in your qandc account Your PC directories your qandc server directories

18. Navigating through your directories • Click on the “home” icon of your PC section and the “home” icon of your qandc section. • The PC section shows your drives. You can go to any folder or any drive in your PC. • As you see the default directory of your qandc account is /home/userid

19. File Transfer between your PC and your qandc • Transferring a file between the your PC and your qandc account is as simple of drag and drop • Look for a text file of your choice under any of your drives on your PC section and drag and drop it to your /home/username folder of your qandc. • For example, I have chosen the file “StrainB39_Plot1.xls” file and drag it into the “/home/ahadaegh/ folder of my qandc account

20. File Transfer between your PC and your qandc • Note that the selected file that I dragged into my qandc account is now successfully copied and it shown under my home folder of my qandc account • Further, the status of all transferred files are also shown in the status window section of the SSH application

21. File Transfer between your PC and your qandc • Similarly you can take any file from your qandc account and transfer it to any folder in your PC. • You can also transfer the whole directory from/to your PC or your qandc account. • When you transfer the whole directory, files under that directory are queued and they get transferred one by one. • Select a directory on your PC and drag and drop it to your qandc server. Make sure the directory is not very big in size; otherwise, it may take a while before all the files are transferred.

22. File Transfer between your PC and your qandc • You can discover about all the available buttons in the SSH application yourself . For example, one of the button that may become handy is the hide/show folder button. • Click the “hide/show local folder” button of your PC section and your qandc section. An explore folder structure appears which allows you to navigate through the folders a lot simpler.

23. File Transfer between your PC and your qandc • Another option that is quite handy is the “refresh” button. • Occasionally, when you transfer files from/to your PC or qandc account you may not see the file immediately. • What you can do is to click on the refresh button to refresh the directory content. 23

24. Creating Terminal and working with UNIX

25. Opening a session • You can click on the “New Terminal Window” icon to open a session. • Now you can use Unix commands to communicate with the system interactively. • In this section we introduce simple Unix commands and show how each command correspond to the ones in window. This is your terminal

26. What is UNIX? • UNIX is an operating system which was first developed in the 1960s, and has been under constant development ever since. • By operating system, we mean the suite of programs which make the computer work. • It is a stable, multi-user, multi-tasking system for servers, desktops and laptops. • UNIX systems also have a graphical user interface (GUI) similar to Microsoft Windows which provides an easy to use environment. • However, knowledge of UNIX is required for operations which aren't covered by a graphical program, or for when there is no windows interface available • There are many different versions of UNIX, although they share common similarities. The most popular varieties of UNIX are Sun Solaris, GNU/Linux, and MacOS X.

27. The UNIX operating system • The UNIX operating system is made up of three parts; the kernel, the shell and the programs. • The kernel • The kernel of UNIX is the hub of the operating system: it allocates time and memory to programs and handles the file store and communications in response to system calls. • As an illustration of the way that the shell and the kernel work together, suppose a user types rm myfile (which has the effect of removing the file myfile). • The shell searches the filestore for the file containing the program rm, and then requests the kernel, through system calls, to execute the program rm on myfile. • When the process rm myfile has finished running, the shell then returns the UNIX prompt [ahadaegh@qandc ~]$ to the user, indicating that it is waiting for further commands

28. The shell • The shell acts as an interface between the user and the kernel. • When a user logs in, the login program checks the username and password, and then starts another program called the shell. • The shell is a command line interpreter (CLI). It interprets the commands the user types in and arranges for them to be carried out. The commands are themselves programs: when they terminate, the shell gives the user another prompt ([ahadaegh@qandc ~]$ on our systems). • The adept user can customize his/her own shell, and users can use different shells on the same machine. The shell that we use is called bash.

29. Files and processes • Everything in UNIX is either a file or a process. • A process is an executing program identified by a unique PID (process identifier). • A file is a collection of data. They are created by users using text editors, running compilers etc. • Examples of files: • a document (report, essay etc.) • the text of a program written in some high-level programming language instructions comprehensible directly to the machine and incomprehensible to a casual user, for example, a collection of binary digits (an executable or binary file); • a directory, containing information about its contents, which may be a mixture of other directories (subdirectories) and ordinary files.

30. All the files are grouped together in the directory structure. • The file-system is arranged in a hierarchical structure, like an inverted tree. The top of the hierarchy is traditionally called root (written as a slash /) / home work bin …… …… usr ahadaegh bread dgarcia cdeleone tcrouch …… …… shizer Strain1516_Plot1.xls CPP NewEhux(June3,2010) public_html …… me.cpp …… HIV_Protease Bioinfo ESTWebByMohammd(April23) DataFiles Document QuickSearchCode.txt SQLTables CodeForFeb18.txt all_databases.sql

31. Listing files and directories • ls (list) • When you first login, your current working directory is your home directory. • Your home directory has the same name as your user-name, for example, jsmith, and it is where your personal files and subdirectories are saved. • To find out what is in your home directory, type • $ ls • The ls command ( lowercase L and lowercase S ) lists the contents of your current working directory.

32. Making Directories • mkdir (make directory) • We will now make a subdirectory in your home directory to hold the files you will be creating and using in the course of this tutorial. • To make a subdirectory called unixstuff in your current working directory type: • $ mkdir unixstuff • To see the directory you have just created, type: • $ ls

33. cd (change directory) • The command cd directory means change the current working directory to 'directory'. • The current working directory may be thought of as the directory you are in, i.e. your current position in the file-system tree. • To change to the directory you have just made, type • $ cd unixstuff • Type ls to see the contents (which should be empty) • Now create a subdirectory called “backup” under the unixstuff directory • $ mkdir backup • To see the directory you have just created, type • $ ls

34. Exercise 1 • On your PC create 6 simple text files called home1.txt home2.txt, unix1.txt, unix2.txt, backup1.txt, backup2.txt • Use SSH to transfer these files from your PC to appropriate directories: • Place home1.txt, and home2.txt under your home directory, unix1.txt and unix2.txt under your unixstuff directory and backup1.txt and backup2.txt under your backup directory

35. The directories . and .. • From you terminal Type • $ cd /home/unixstuff • to get back to your unixstuff directory, and then type • $ ls -a • As you can see, in the unixstuff directory (and in all other directories), there are two special directories called (.) and (..) • In UNIX, (.) means the current directory, so typing • $ cd . • means stay where you are (the unixstuff directory). Note that there is a space between cd and the dot • This may not seem very useful at first, but using (.) as the name of the current directory will save a lot of typing, as we shall see later. • (..) means the parent of the current directory, so typing • $ cd .. • will take you one directory up the hierarchy (back to your home directory). • Note: typing cd with no argument always returns you to your home directory. This is very useful if you are lost in the file system.

36. Pathnames • pwd (print working directory) • Pathnames enable you to work out where you are in relation to the whole file-system. • For example, to find out the absolute pathname of your home-directory, type cd to get back to your home-directory and then type • $ pwd • The full pathname for the backup directory looks something like this: • /home/ahmadtest/unixstuff/backup • which means that backup is in the sub-directory unixstuff which in turn is located under the ahadaegh sub-directory, which is under the home sub-directory, which is in the top-level root directory called “/”.

37. / home work bin …… …… usr cdeleone tcrouch …… …… shizer ahmadTest dgarcia bread home2.txt unixstuff home1.txt backup unix1.txt unix1.txt backup1.txt backup2.txt

38. Understanding Pathnames • Move to your home directory by typing • $ cd • Now type: • $ lsunixstuff • to list the consents of your unixstuff directory. • Now type: • $ ls backups • You will get a message like this – • backups: No such file or directory • The reason is, backups is not in your current working directory. • To use a command on a file (or directory) not in the current working directory (the directory you are currently in), you must either cd to the correct directory, or specify its full pathname. • To list the contents of your backups directory, you must type • $ lsunixstuff/backups

39. ~ (your home directory) • Home directories can also be referred to by the tilde ~ character. It can be used to specify paths starting at your home directory. So typing: • $ ls ~/unixstuff • will list the contents of your unixstuff directory, no matter where you currently are in the file system. • What do you think the following would list? • $ ls ~ • What do you think the following would list? • $ ls ~/..

40. Summary

41. Exercise 2 • Go to the site http://genome.jgi-psf.org/ • Under Amoebozoa, choose Dictyosteliumpurpureum QSDP1 • Click on the DOWNLOAD to get into the download page • Click on OK to accept the Policy • Download the following files and place them under your desktop • Proteins: Dicpu1_best_proteins.fasta.gz • Transcripts: Dicpu1_best_transcripts.fasta.gz • Genes: Dicpu1_best_genes.gff.gz • EST cluster consensi: Dicpu1_EST_cluster_consensi.fasta.gz • Use either command line or SSH to Create 4 subdirectories called EST, Protein, Gene, and Transcripts under your home directory • Use SSH to transfer protein file into protein directory, Gene file into the gene directory and so on • Create a terminal (If you do not have one opened yet) and use the commands cd, ls and pwd to explore the file system. (Remember, if you get lost, type cd by itself to return to your home-directory)

42. Copying Files • cp (copy) • cp file1 file2 is the command which makes a copy of file1 in the current working directory and calls it file2 • To test this first, under your home directory create a sub directory called testForCopyCmd. • Then type: $ cdtestForCopyCmd • Then at the UNIX prompt, type, • $ cp /home/ahmadtest/unixstuff/unix1.txt . • Note: Don't forget the dot . at the end • Remember, in UNIX, the dot means the current directory. • The above command means copy the file unix1.txt to the current directory, keeping the name the same.

43. Moving files • mv (move) • mvfile1 file2 moves (or renames) file1 to file2 • To test this, first create couple of simple text files called p1.txt and p2.txt and transfer them to your home directory using SSH. • To move a file from one place to another, use the mv command. This has the effect of moving rather than copying the file, so you end up with only one file rather than two. • It can also be used to rename a file, by moving the file to the same directory, but giving it a different name. • Change the directory to your unixstuff directory. Then, under the unixstuff directory, type • $ mv ../p1.txt . • $ mv ../p2.txt . • Now Type $ lsand to see if it has worked.

44. Removing files and directories • rm (remove), rmdir (remove directory) • To delete (remove) a file, use the rm command. As an example, we are going to create a copy of the science.txt file then delete it. • Inside your unixstuff directory, type • $ rm p1.txt • $ ls • You can use the $rmdirdirectoryNamecommand to remove a directory (make sure the directory is empty first). • Try to remove the backups directory. You will not be able to since UNIX will not let you remove a non-empty directory. • Also, the command $ rm * removes all the files but not the the directories under a directory. It does not remove hidden files either • The command $ rm –r directoryNameremoves a directory and all of its subdirectories and files under that directory

45. clear (clear screen) • Before you start the next section, you may like to clear the terminal window of the previous commands so the output of the following commands can be clearly understood. • At the prompt, type • $ clear • This will clear all text and leave you with the % prompt at the top of the window.

46. Displaying the contents of a file on the screen • cat (concatenate) • The command cat can be used to display the contents of a file on the screen. • From your desktop, create a text file called science.txt. • Go the following site: • http://cropsci.illinois.edu/academics/grad/bioinformatics.cfm • and copy the article and paste it into the science.txt file • Then use SSH to transfer the file under your home directory. • Now from the terminal, go to your home directory and type • cat science.txt • As you can see, the file is longer than the size of the window, so it scrolls past making it unreadable

47. Less • The command “less” writes the contents of a file onto the screen a page at a time. Type: • $ less science.txt • Press the [space-bar] if you want to see another page, and type [q] if you want to quit reading. • As you can see, “less” is used in preference to cat for long files.

48. Head • The head command writes the first ten lines of a file to the screen. • First clear the screen then type • $ head science.txt • Then type: • $ head -5 science.txt • What difference did the -5 do to the head command? • tail • The tail command writes the last ten lines of a file to the screen. • Clear the screen and type • $ tail science.txt • How can you view the last 15 lines of the file?

49. Simple searching using “less” • Using “less”, you can search though a text file for a keyword (pattern). For example, to search through home1.txt for the word 'science', type • $ less science.txt • then, still in “less”, type a forward slash [/] followed by the word to search for the word “science” as follows: • /science • As you can see, “less” finds and highlights the keyword. Type “n” to search for the next occurrence of the word “science”. • To search the previous word, you can hold the shift key down and type “n”.

50. “grep” command for searching words • grep is one of many standard UNIX utilities. It searches files for specified words or patterns. First clear the screen, then type • $ grep science science.txt • As you can see, grep has printed out each line containing the word science. • Try typing • $ grep Science science.txt • The grep command is case sensitive; it distinguishes between “Science” and “science”. • To ignore upper/lower case distinctions, use the -i option, i.e. type: • $ grep -i science science.txt