i/o multiplexing

1. i/o multiplexing On adding a ‘poll()’ method to our character-mode device-driver for an 82573L network controller

2. An application idea • We want to create an application program that would allow two users to conduct an on-line chat-session while working from two separate nodes on our ‘anchor’ cluster • Whatever either user decides to type will be displayed on the both users’ screens • To keep their two input-streams visually separated we want to use two ‘windows’

3. ‘netchat.cpp’ Whatever a user types will be displayed in the lower window on that user’s screen -- and in the upperwindow on the other user’s screen Hello, Susan Hi, Paul Hi, Paul _ Hello, Susan _ anchor01 anchor02

4. Multiplexed input-streams • In order to implement the foregoing idea, it’s necessary for the ‘netchat’ program to accept input from TWO hardware devices: • the keyboard (i.e., ‘standard input’ device) • the ethernet controller (i.e., ‘/dev/nic’) • Such a situation is a fairly common one in UNIX/Linux application programming, and is referred to as ‘i/o multiplexing’

5. Device-driver’s role • Special software support is needed within each of the device-drivers in order for any application-program to do ‘i/o multiplexing’ in a way that avoids wasting of CPU time • (For an excellent discussion of the various approaches that can be taken to deal with this basic programming issue, see Richard Stevens: Advanced Programming in the UNIX Environment, Chapter 12)

6. The basic problem • Normally when an application ‘reads’ from a device-file, that process will ‘sleep’ until some data is available from that device • So if data becomes available on another device, it will not get processed because the application is ‘blocked’ from being given any CPU time by the OS scheduler • This would spoil our ‘netchat’ application

7. ‘read()’ causes ‘blocking’ ‘netchat’ application terminal read write Ethernet controller write read Whichever device the application attempts to read from, it will get ‘blocked’ until that device has some data to deliver

8. Do multiprocessing? • One idea for getting around this ‘blocking’ problem would be to just use the ‘fork()’ system-call to create a separate process for reading from the different device-files • Each process can sleep, and whichever process receives any new data will be awakened and scheduled for execution • No changes needed to device-driver code

9. Different processes do ‘read()’ ‘netchat’ parent- process write read terminal Ethernet controller ‘netchat’ child-process read write Using multiple processes can overcome the ‘blocking-read’ problem, but complicates the code for program termination

10. Non-blocking ‘read’ • It is possible for the application to request ‘non-blocking’ read-operations – i.e., any ‘read()’ calls will immediately return with 0 as return-value in case no data is available • The standard-input device-driver already has support for this non-blocking option, and it can be easily added to the ‘read()’ function in network controller’s driver

11. Code-modification ssize_t my_read( struct file *file, char *buf, size_t len, loff_t *pos ) { static int rxhead = 0; // in case no new data has been received, then either // return immediately if non-blocking mode is in effect // or else sleep until some new data arrives (or until // the user hits <CONTROL>-C to cancel execution) if ( rxhead == ioread32( io + E1000_RDH ) { if ( file->f_flags & O_NONBLOCK ) return 0; if ( wait_event_interruptible( wq_rx, rxring != ioread32( io + E1000_RDH ) ) return –EINTR; } …

12. Uses ‘busy-waiting’ loop ‘netchat’ application terminal read write write Ethernet controller read Using the ‘nonblocking-read’ option overcomes the problem of a sleeping task, but it wastefully consumes the CPU time

13. The ‘elegant’ solution • The ‘select()’ system-call provides a very general scheme for doing i/o-multiplexing in a manner that avoids wasting CPU time or making the program-code complicated • But it does require adding an extra driver ‘method’ – the so-called ‘poll()’ function

14. The ‘select()’ arguments • Using ‘select()’ requires an application to setup an ‘fd_set’ object, which defines the set of file-descriptors whose activity needs to be monitored by the Linux kernel (in our ‘netchat’ application this would be just the two device-files: the console keyboard and the gigabit ethernet network controller) • This ‘fd_set’ object becomes an argument

15. Using ‘select()’ in ‘netchat’ int kbd = STDIN_FILENO; // keyboard ID int aux = open( “/dev/nic”, O_RDWR ); // device-file ID fd_set permset; // create an ‘fd_set’ object FD_ZERO( &permset ); // initialize it to ‘empty’ FD_SET( kbd, &permset ); // add keyboard to set FD_SET( aux, &permset ); // and add the nic to set while (1) { fd_set readset = permset; if ( select( 1+aux, &readset, NULL, NULL, NULL ) < 0 ) break; if ( FD_ISSET( kbd, &readset ) ) { /* process keyboard input */ } if ( FD_ISSET( aux, &readset ) ) { /* process network input */ } }

16. How it works • The ‘readset’ argument to the ‘select()’ system-call lets the kernel know which device-drivers should have their ‘poll()’ method invoked • Then each device-driver’s ‘poll()’ method will perform a test to determine if any new data is ready to be read from that device • So the application calls ‘read()’ only when a device is ready with data immediately

17. struct file_operations • We need to include the function-pointer to our implementation for the ‘poll()’ method: struct file_operations my_fops = { owner: THIS_MODULE, read: my_read, write: my_write, ioctl: my_ioctl, poll: my_poll, };

18. Our driver’s ‘poll()’ method • Linux provides helper-functions to do most of the supporting work for use of ‘select()’ #include <linux/poll.h> // for the ‘poll_wait()’ helper-function unsigned int my_poll( struct file *file, struct poll_table_struct *wait ) { unsigned int mask = 0; poll_wait( file, &wq_recv, wait ); if ( (ioread32( io + E1000_RDH ) != rxhead ) mask |= POLLIN | POLLRDNORM; return mask; }

19. The ‘ncurses’ library • Our ‘netchat’ application uses the Linux implementation for the UNIX ‘curses’ API • This function-library includes routines that not only can manage the console’s cursor, but also implements ‘raw’ keyboard input (for instantaneous keystroke processing) and allows drawing of window-borders

20. Compiling ‘netchat’ • The ‘g++’ compiler can compile-and-link in a single command: • $ g++ netchat.cpp -l ncurses -o netchat The C++ source-file Name of the library Lowercase letter ‘L’ (for library to link with) Name for the compiler’s output-file

21. Try it out • We can try running the ‘netchat’ program on any pair of our anchor-cluster stations • Of course, we will first need to install our ‘nicpoll.ko’ device-driver module on each of those ‘anchor’ machines • Then a user at either machine can type in any messages, and every character typed will be visible immediately on both screens

22. But… • There is a slight problem with using the present version of our ‘nicpoll.c’ module • All of the network packets are ‘broadcast’ to every station on the ‘anchor’ cluster! • So any third-party who is simultaneously trying to use our network will be receiving all the packets that are being broadcast

23. In-class exercise • Based on what you have learned from our past lessons, and from your reading of the Intel “Open Source Software Developer’s Manual” for its PCIe GbE Controllers, can you propose any ideas for using the NIC’s hardware capabilities so as to allow a pair of anchor-stations to use ‘netchat’ without any other anchor-stations being involved?