CS162B: Pipes

1. CS162B: Pipes Jacob T. Chan

2. Pipes • These allow output of one process to be the input of another process • One of the oldest and most basic forms of Inter-Process Communications • We shall discuss this first before going on to semaphores. Both actually deal with IPC, but piping will be discussed so that the basics will be tackled • This is useful when two processes run different programs and there is a need for both of them to communicate • This is simple enough to understand, but it looks harder than it seems • Process management issues

3. Inter-Process Communication (IPC) • Exchange of data among multiple threads in one or more processes (according to Wikipedia) • Issues include • Read-write to shared data • Synchronization • Memory sharing • Remote Procedure Calls (RPC) aka execution of subroutine in another address space • In other words, it’s how processes communicate with one another (hence, the name) • Processes need to communicate too! • Example: ALL processes depend on the mother of all processes known as the SCHEDULER

4. Pipes in Unix • In bash, what we do is to run a piped process. Example: ls | wc –l • Running this command will result into the output of the first command being the input of the second command • So, ls will run first. Then whatever its output, it will be the input of the second command wc –l

5. Invoking Pipe • #include <unistd.h> (This will allow the use of libraries like pipe() ) • int pipe(intfiledes[2]) • Creates data pipe • filedes[2]are your file descriptors, with filedes[0] opened for READING ONLY and filedes[1] opened for WRITING ONLY • Note: reading data on filedes[0] will access data written by filedes[1] • Output: • 0 if successful • -1 if not successful • The variable errno will indicate what type of error that caused the pipe to fail • Example of pipe posted in Moodle (Piping Basic Example)

6. Piping Features • When two processes are piped, they are unaware of it • They continue writing to and reading from standard file descriptors • Whatever the output of the first process, the second process will use that REGARDLESS of conditions (in short, second process of pipe is dependent on first one) • Pipes are byte streams! • Process reading from pipes is free to read any size of data blocks • Implication: not everything written by the writing process is read by the reading process • You cannot lseek pipes! Because data is read in FIFO fashion

7. Piping Features • If the file descriptor to the write end is closed (as a result of EOF), all reads will return 0 or EOF once everything is read • PIPES ARE UNIDIRECTIONAL (one end for reading, one end for writing ONLY) • Each pipe write is atomic • Writing to the same pipe will not be mixed if they write at most X bytes at a time (that’s the function of having a specific file descriptor) • When writing to full pipe (meaning max space for writing is reached), operation will block UNTIL space becomes available on pipe • Reading from empty pipe will be blocked until new data is written to the pipe

8. Piping Features • Analysis: Producer-Consumer (which will be like your lab) • Producer (writing side) will keep on producing the characters until there is nothing left to write • Consumer will then read what the producer produced until everything is read • In cases where there is nothing to read, the consumer “sleeps” (or gets BLOCKED) until there is something to read • In cases where pipe is full, producer will stop producing in the pipe! • Will wait until consumer consumes something. Only then will producer will write output to the pipe

9. Pipe Sample Code Snippet (more examples at Moodle) intfiledes[2]; if(pipe(filedes) == -1) exit(1); switch(fork()) { case 0: if(close(filedes[1]) == -1) errExit(“close”); break; default: if(close(filedes[0]) == -1) errExit(“close”) break; }

10. Piping Issues • Rarely do two or more processes read from the same pipe • If both processes read at the same time, there will be RACE CONDITIONS (and there will be no way to guarantee will be reading data over the other) • Any two processes (not just parents and children) that are “related” to one another should communicate! • Running the previous command ls | wc –l should work because they communicate via shell (which is their parent that created the pipe) • If the processes being piped belong to different parents, then the pipe will not work (in most cases, since they do not share similar data)

11. Piping Issues • Implementing parent-child communication is simple enough, but connecting two programs reading from stdin and writing to stdout might be trickier • This is an issue of shared memory, semaphores, socketing (which will be discussed in the coming weeks) • Redirection of stdout to a pipe and stdin being pointed to the same pipe (which is technically socketing) • In piping, we can use the function dup2()

12. Pipes: int dup2(intoldfd, intnewfd) • This duplicates of file descriptor oldfd and assigns it the new number newfd • If newfd corresponds to open descriptor at time of calling, the file descriptor is CLOSED first (to ensure that no blocking will occur in this time) • This allows read and write descriptors to get their corresponding descriptor numbers • Returns the new file descriptor number; otherwise returns -1 on error • Example posted in Moodle

13. Pipes: FILE* popen(char* command, char* type) • A simpler way to creating shell-like piping • Pipes are usually used to read from output or write to input in the context of shell commands • This command creates a pipe and forks child process that execs a shell • This creates child process to execute shell command (which is first parameter) • Type is a string that indicates whether the process will write to (noted by “w”) or read from (noted by “r”) the pipe • For using popen, instead of fclose, use pclose(FILE *stream) instead to close the pipe itself • Example posted in Moodle

14. Pipes: FILE* popen(char* command, char* type)

15. Pipes • There are many more techniques to piping, and what was shown were just one of them • We only handled the basic ones. • If you want more examples on piping, I posted some on Moodle for your reference • Or, you can consult your man pages in your terminal.

16. Lab 9: pcpipe.c • Create a program pcpipe.c that run as follows: • Create a pipe in your program • Fork a child (this will be the producer named “prod”) • Fork another child in the PARENT part (this will be the consumer named “cons”) • Make output of prod the same as the read end of pipe using dup2 • Make input of cons the same as write end of pipe using dup2 • In prod, execve() the producer program, including arguments • In cons, execve() the consumer program, including arguments • Note: popen() is NOT allowed. You have execve()

17. Lab 9: pcpipe.c • The program runs the same way as executing bash commands. For example: ./pcpipeproducerprogprodargs… -consumer consumerprogconsumerargs… should produce the same command as running its bash version: producerprogprodargs… | consumerprogconsumerargs … • Example: ./pcpipels –l –consumer grep “2013-2-2” This program runs similarly as doing the following: ls –l | grep “2013-2-2”

18. Lab 9: pcpipe.c • Other sample input you can use: ./pcpipe cat /etc/passwd -consumer cut -d\: -f1,3,6 ./pcpipels -l /home -consumer grep "$LOGNAME“ This should produce the same expected results as its bash version

19. Lab 9: pcpipe.c • Legal stuff • Don’t forget Certificate of Authorship (with COMPLETE details), as well as your pcpipe.c program. • Filename: CS162B_Lab9_<Section>_<Surname>_<ID Number>.tar • Deadline: Next week • Tuesday for Section A • Thursday for Section B