Implementing 'poll()' Method for Efficient I/O Multiplexing

i/o multiplexing On adding a ‘poll()’ method to our character-mode device-driver for an 82573L network controller

An application idea • We want to create an application program that would allow two users to conduct an on-line chat-session while working from two separate nodes on our ‘anchor’ cluster • Whatever either user decides to type will be displayed on the both users’ screens • To keep their two input-streams visually separated we want to use two ‘windows’

‘netchat.cpp’ Whatever a user types will be displayed in the lower window on that user’s screen -- and in the upperwindow on the other user’s screen Hello, Susan Hi, Paul Hi, Paul _ Hello, Susan _ anchor01 anchor02

Multiplexed input-streams • In order to implement the foregoing idea, it’s necessary for the ‘netchat’ program to accept input from TWO hardware devices: • the keyboard (i.e., ‘standard input’ device) • the ethernet controller (i.e., ‘/dev/nic’) • Such a situation is a fairly common one in UNIX/Linux application programming, and is referred to as ‘i/o multiplexing’

Device-driver’s role • Special software support is needed within each of the device-drivers in order for any application-program to do ‘i/o multiplexing’ in a way that avoids wasting of CPU time • (For an excellent discussion of the various approaches that can be taken to deal with this basic programming issue, see Richard Stevens: Advanced Programming in the UNIX Environment, Chapter 12)

The basic problem • Normally when an application ‘reads’ from a device-file, that process will ‘sleep’ until some data is available from that device • So if data becomes available on another device, it will not get processed because the application is ‘blocked’ from being given any CPU time by the OS scheduler • This would spoil our ‘netchat’ application

‘read()’ causes ‘blocking’ ‘netchat’ application terminal read write Ethernet controller write read Whichever device the application attempts to read from, it will get ‘blocked’ until that device has some data to deliver

Do multiprocessing? • One idea for getting around this ‘blocking’ problem would be to just use the ‘fork()’ system-call to create a separate process for reading from the different device-files • Each process can sleep, and whichever process receives any new data will be awakened and scheduled for execution • No changes needed to device-driver code

Different processes do ‘read()’ ‘netchat’ parent- process write read terminal Ethernet controller ‘netchat’ child-process read write Using multiple processes can overcome the ‘blocking-read’ problem, but complicates the code for program termination

Non-blocking ‘read’ • It is possible for the application to request ‘non-blocking’ read-operations – i.e., any ‘read()’ calls will immediately return with 0 as return-value in case no data is available • The standard-input device-driver already has support for this non-blocking option, and it can be easily added to the ‘read()’ function in network controller’s driver

Code-modification ssize_t my_read( struct file *file, char *buf, size_t len, loff_t *pos ) { static int rxhead = 0; // in case no new data has been received, then either // return immediately if non-blocking mode is in effect // or else sleep until some new data arrives (or until // the user hits <CONTROL>-C to cancel execution) if ( rxhead == ioread32( io + E1000_RDH ) { if ( file->f_flags & O_NONBLOCK ) return 0; if ( wait_event_interruptible( wq_rx, rxring != ioread32( io + E1000_RDH ) ) return –EINTR; } …

Uses ‘busy-waiting’ loop ‘netchat’ application terminal read write write Ethernet controller read Using the ‘nonblocking-read’ option overcomes the problem of a sleeping task, but it wastefully consumes the CPU time

The ‘elegant’ solution • The ‘select()’ system-call provides a very general scheme for doing i/o-multiplexing in a manner that avoids wasting CPU time or making the program-code complicated • But it does require adding an extra driver ‘method’ – the so-called ‘poll()’ function

The ‘select()’ arguments • Using ‘select()’ requires an application to setup an ‘fd_set’ object, which defines the set of file-descriptors whose activity needs to be monitored by the Linux kernel (in our ‘netchat’ application this would be just the two device-files: the console keyboard and the gigabit ethernet network controller) • This ‘fd_set’ object becomes an argument

Using ‘select()’ in ‘netchat’ int kbd = STDIN_FILENO; // keyboard ID int aux = open( “/dev/nic”, O_RDWR ); // device-file ID fd_set permset; // create an ‘fd_set’ object FD_ZERO( &permset ); // initialize it to ‘empty’ FD_SET( kbd, &permset ); // add keyboard to set FD_SET( aux, &permset ); // and add the nic to set while (1) { fd_set readset = permset; if ( select( 1+aux, &readset, NULL, NULL, NULL ) < 0 ) break; if ( FD_ISSET( kbd, &readset ) ) { /* process keyboard input */ } if ( FD_ISSET( aux, &readset ) ) { /* process network input */ } }

How it works • The ‘readset’ argument to the ‘select()’ system-call lets the kernel know which device-drivers should have their ‘poll()’ method invoked • Then each device-driver’s ‘poll()’ method will perform a test to determine if any new data is ready to be read from that device • So the application calls ‘read()’ only when a device is ready with data immediately

struct file_operations • We need to include the function-pointer to our implementation for the ‘poll()’ method: struct file_operations my_fops = { owner: THIS_MODULE, read: my_read, write: my_write, ioctl: my_ioctl, poll: my_poll, };

Our driver’s ‘poll()’ method • Linux provides helper-functions to do most of the supporting work for use of ‘select()’ #include <linux/poll.h> // for the ‘poll_wait()’ helper-function unsigned int my_poll( struct file *file, struct poll_table_struct *wait ) { unsigned int mask = 0; poll_wait( file, &wq_recv, wait ); if ( (ioread32( io + E1000_RDH ) != rxhead ) mask |= POLLIN | POLLRDNORM; return mask; }

The ‘ncurses’ library • Our ‘netchat’ application uses the Linux implementation for the UNIX ‘curses’ API • This function-library includes routines that not only can manage the console’s cursor, but also implements ‘raw’ keyboard input (for instantaneous keystroke processing) and allows drawing of window-borders

Compiling ‘netchat’ • The ‘g++’ compiler can compile-and-link in a single command: • $ g++ netchat.cpp -l ncurses -o netchat The C++ source-file Name of the library Lowercase letter ‘L’ (for library to link with) Name for the compiler’s output-file

Try it out • We can try running the ‘netchat’ program on any pair of our anchor-cluster stations • Of course, we will first need to install our ‘nicpoll.ko’ device-driver module on each of those ‘anchor’ machines • Then a user at either machine can type in any messages, and every character typed will be visible immediately on both screens

But… • There is a slight problem with using the present version of our ‘nicpoll.c’ module • All of the network packets are ‘broadcast’ to every station on the ‘anchor’ cluster! • So any third-party who is simultaneously trying to use our network will be receiving all the packets that are being broadcast

In-class exercise • Based on what you have learned from our past lessons, and from your reading of the Intel “Open Source Software Developer’s Manual” for its PCIe GbE Controllers, can you propose any ideas for using the NIC’s hardware capabilities so as to allow a pair of anchor-stations to use ‘netchat’ without any other anchor-stations being involved?

Implementing 'poll()' Method for Efficient I/O Multiplexing