APIC Tutorial --- Software: Drivers, Libraries and Utilities John DeHart Washington University

1. APIC Tutorial --- Software: Drivers, Libraries and Utilities John DeHart Washington University jdd@arl.wustl.edu http://www.arl.wustl.edu/~jdd

2. Overview • Drivers • NetBSD • Linux • We will NOT talk about the MSR APIC Driver!!! • User Mode Library • APICCell library • Utilities • atm_ifconfig • Traffic Generators

3. Driver Overview • The Big Picture • IP • RATM

4. Driver Overview (Especially NetBSD) User Kernel

5. ipintrq: atmc_output apic_devoutput IPv4 Forwarding in the NetBSD Kernel (APIC) TRANSPORT socket layer tcp/udp_input tcp/udp_output N E T W O R K ip6_input ip6_forward ip6_output ip_forward ip_output ipintr U PPER D A T A atmc_input L OWER L I N K apic_isr apic_intr input packets interrupts output packets

6. User Program Waits in Socket Buffer for User Program to pick up Software Interrupt ipintrq: atmc_output apic_devoutput User IP in the NetBSD Kernel TRANSPORT socket layer tcp/udp_input tcp/udp_output N E T W O R K ip_forward ip_output ipintr U PPER D A T A atmc_input L OWER L I N K apic_isr apic_intr input packets interrupts output packets

7. AAL0 Receive Pool Chain Cluster mbuf Cluster mbuf Cluster mbuf ... D D D Cluster mbuf Host Memory 2048 Two Sets of Receive Descriptors/Buffers AAL5 Receive Pool Chain Cluster mbuf Cluster mbuf Cluster mbuf ... D D D Host Memory AAL0 buffer AAL5 buffer Cluster mbuf - % 48 => - % 56 => 2048 2016 2016 Currently the same size, but not necessarily so

8. APIC Interrupt on Done Driver Reads Which Channel Caused Intr. AAL5 Frame IP Q EOF Interrupt Chan N Done Done Done Host Memory Channel N Notify Register

9. Notify Register APIC Interrupt on Descriptor Read Driver Reads Which Channel Caused Intr. Receive Pool Chain RATM Q Cluster mbuf Cluster mbuf APIC Reads Next Descriptor & Makes it Current Descriptor for Chan N Interrupt Done Chan N Host Memory ... D D Host Memory Channel N Current Descriptor

10. Receive Pools: Two Per-Frame, One Per-Cell • Two Per-Frame Pools: • AAL5 • Interrupt on DONE • size <= requested system buffer (mbuf) size • size % 48 = 0 • Used for IP and RATM AAL5 applications • AAL0 • Interrupt on Descriptor Read • size <= requested system buffer (mbuf) size • size % 56 = 0 • Used for continuous AAL0 data applications • For current cluster mbuf • AAL0_size = AAL5_size = 2016 • Fairly large number of buffers (100’s to 1000’s)

11. Receive Pools (con’t) • AAL0 per-cell Pool • buffer size = 56 bytes • Interrupt on Descriptor Read • Each time APIC reads a new Descriptor, the previous one is available for the Driver to process • Used for per-cell AAL0 applications like GBNSC • Small number of buffers (10’s) • CAVEAT: Deadly if used for high bandwidth channel • Interrupt on EVERYCell • We are using a cluster mbuf (2K Bytes) for each cell!! • Can we use regular mbuf (~200 Bytes)? • Physical address alignment implications?

12. atm_ifconfig • NetBSD ATM VC configuration utility > atm_ifconfig Usage: atm_ifconfig <interface> closeall Usage: atm_ifconfig <interface> getall Usage: atm_ifconfig <interface> <vpivci> [open|close|modify [options...]] Options are: -aal0 | -aal0_cell | -aal5 -llc | -nollc <protocol>: use LLC/SNAP? protocol is inet | ratm | value of ethernet type -loopback -inport [FIBER,RIBBON,SWITCH,LINK] -outports [FIBER,RIBBON,SWITCH,LINK,BOTH] -lowdelay | -paced <rate> | -besteffort -maxqueue <max_descriptors_in_tx_queue> rate is in Kbits/sec

13. NetBSD Device, VC and Route Configuration • ifconfig apic0 inet 192.168.208.2 netmask 0xffffff00 up • Configure VCI 32 for newGBNSC to use (NetBSD): • atm_ifconfig apic0 0x20 open -aal0_cell -nollc ratm -inport FIBER -outports FIBER • Configure VCI 100 for IP usage (NetBSD): • atm_ifconfig apic0 0x64 open –aal5 -nollc inet -inport FIBER -outports FIBER • route add –iface 192.168.208.1 -link apic0:0.0.0.64 Error on handouts

14. Linux Device, VC and Route Configuration • /etc/sysconfig/network-scripts/ifcfg-atm0 • /etc/init.d/atm start • atmsigd • ilmid • atmarp • route add

15. Linux Device, VC and Route Configuration (con’t) # # /etc/sysconfig/network-scripts/ifcfg-atm* # DEVICE=atm0 # device name wuapic0 DEVICE_ADDR=demand4-apic0 # host's hardware address (made up) IPADDR=192.168.10.3 # hosts's IP address on the ATM LIS NETMASK=255.255.255.0 # netmask (e.g. arl uses 8 host bits) NETWORK=192.168.10.0 # network (e.g. arl's experimental ATM LIS) BROADCAST=192.168.10.255 # broadcast address (not really useful) # ARPSRV=lrcsuns-a # ATM address of ATMARP server; omit if local # ARPSRVQOS="ubr:max_pcr=100kbps" # QOS of the ATMARP server VC # ARPDEFQOS="ubr:max_pcr=700kbps" # default QOS for IP over ATM VCs ONBOOT=no # needs to be loaded by ATM-specific procedure MTU=32768 # larger packets ok on tx

16. Linux Device, VC and Route Configuration (con’t) > /etc/init.d/atm start Starting ATM demons: [OK] > ps -auxwww | grep atmsigd root 973 0.0 0.1 1708 628 pts/1 S 09:56 0:00 atmsigd -b -l syslog -D /var/tmp -t 20 root 1019 0.0 0.1 1708 600 pts/1 S 09:56 0:00 grep atmsigd > ps -auxwww | grep ilmid root 984 0.0 0.0 1428 384 pts/1 S 09:56 0:00 ilmid -b -l syslog root 1021 0.0 0.1 1708 600 pts/1 S 09:56 0:00 grep ilmid # YOU MAY WANT TO KILL THESE… • kill 973 984 • atmsigd and ilmid are started in /etc/init.d/atm • If you are SURE you don’t want them on your machines back home, that is the place to edit them out. DO NOT do that here!!!

17. Linux Device, VC and Route Configuration (con’t) > ifconfig atm0 atm0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 inet addr:192.168.10.3 Mask:255.255.255.0 UP RUNNING MTU:32768 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100

18. Linux Device, VC and Route Configuration (con’t) > atmarp -s 192.168.10.1 0.0.50 null # for a network route: > route add -net 192.168.10.0 netmask 255.255.255.0 gw 192.168.10.1 # for a host route: > route add –host 192.168.10.2 gw 192.168.10.1 > atmarp -a ----- Itf 0 (192.168.10.3, netmask 255.255.255.0) ----- Default QOS: ubr,aal5:max_sdu=9188 IP 192.168.10.1, state VALID, addr <none>, flags 0x1004<PERM,NULL> 0.0.50 Send buffer: 65535 ----- Unknown incoming connections ----- ----- Incoming unidirectional connections ----- ----- End of dump -----

19. Linux Device, VC and Route Configuration (con’t) > netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 128.252.153.0 0.0.0.0 255.255.255.0 U 40 0 0 eth0 192.168.10.0 192.168.10.1 255.255.255.0 UG 40 0 0 atm0 192.168.10.0 0.0.0.0 255.255.255.0 U 40 0 0 atm0 192.168.10.0 0.0.0.0 255.255.255.0 U 40 0 0 atm0 127.0.0.0 0.0.0.0 255.0.0.0 U 40 0 0 lo 0.0.0.0 128.252.153.249 0.0.0.0 UG 40 0 0 eth0

20. User Mode Library • User Mode library is Driver functionality exposed to User Space programs. • You get direct manipulation of Descriptors and Buffers • Access to APIC Registers • It has many of the complexities of a Driver. • It has many of the DANGERS of a Driver. • Source: wu_arl/utilities/APIC/ApicLibrary • Binary: wu_arl/utilities/lib/$(OSTYPE)/libAPIC.a • Installed include files: wu_arl/utilities/include/

21. User Mode Library • Following slides contain code from one of our utility programs: wu_arl/utilities/APIC/FileTransfer/FileRead.c • Best way to learn how to use the User Mode library is to start from existing code. • If you want to use the User Mode library: YOU MUST INVEST SOME TIME!!! • Start by using the FileTransfer utility. • Get it to work for you. • Add your own debug statements to it • Figure out what it is doing. • Then use it as an example to do what you want.

22. User Mode library example /* From wu_arl/utilities/APIC/FileTransfer/FileRead.c */ /* Initialize */ i = InitApicLibrary(false); /* try for an apic */ if (i) { fprintf(stdout, "Failed to grab an apic - is the user mode code enabled?\n"); exit(-1); } /* map the Data (main data pool) */ ApicHandle = ApicLibraryMap(0, /* use default apic id */ APIC_LIBRARY_READ5, /* open in read mode */ i, /* initial descr */ FramesReady, /* buffer count */ 4096, /* size of frames */ rvpi, /* VPI connection */ rvci, /* VCI connection */ ENDIANNESS_LITTLE, /* endian (little) */ 0x0000); /* do not pack to page boundary */ if (!ApicHandle) { perror("Failed to init DataPool"); exit(-1); }

23. User Mode library example /* From wu_arl/utilities/APIC/FileTransfer/FileRead.c */ /* Start reading */ AckHandle = ApicHandle; /* save output vci data */ if (ApicHandle->udescr <= 0) { fprintf(stdout, "No AckPool descriptors allocated - see /var/log/messages\n"); exit(-1); } AckFirstDescr = AckHandle->udescr; /* obtain first used descriptor */ AckCount = AckHandle->udescc; AckLastDescr = AckFirstDescr + AckCount - 1; bzero(AckHandle->ubuf, 2 * page_size); i = ApicWriteDescriptor0(AckFirstDescr + 1, NotReady, AckHandle); NextDescr = DataFirstDescr; /* init circular chain */ i = ApicStartAAL5Read(DataHandle); /* always read first */ fprintf(stdout, "Looking for FileSend AAL5 data\n"); SendAckNak(-1, ACKNAK_IDLE1); /* send an I'm Alive */

24. User Mode library example /* From wu_arl/utilities/APIC/FileTransfer/FileRead.c */ /* Processing loop. Process36K() does the work */ while (!kern_start_addr) { int what = Process36K(); if (what < 0) { if (Debug) fprintf(stdout, "failure code %d\n", what); if (LastSequence == -1) { // we still haven't received our first frame. Continuing sending pings SendAckNak(-1, ACKNAK_IDLE1); /* send an I'm Alive */ } else SendAckNak(LastSequence, what); /* ping the other side */ continue; } SendAckNak(LastSequence, ACKNAK_OK); /* say ok to the other side */ }

25. User Mode library example /* From wu_arl/utilities/APIC/FileTransfer/FileRead.c */ /* In Process36K(): Check for a DONE_VALID descriptor */ /* patrol the descriptors for this i/o */ while (!CheckReady(NextDescr)) /* (1:36K, 1:4k) X 8 segments */ { w0 = ApicReadDescriptor0(NextDescr, DataHandle); w1 = ApicReadDescriptor1(NextDescr, DataHandle); if (apic_fval(w0, DESC_SYNC) != DESC_SYNC_DONE_VALID_LINK) { if (Debug) { fprintf(stdout, "Not Valid Next Descr:0x%04x, W0:0x%08x, W1:0x%08x\n", NextDescr, w0, w1); } return retcode; }

26. User Mode library example /* From wu_arl/utilities/APIC/FileTransfer/FileRead.c */ /* In Process36K(): Examine data */ /* we have a valid well aligned frame */ Trailer = (unsigned int *) ((unsigned char *) DataHandle->ubuf + offset * 4096); /* get real address */ LT = (struct load_trailer *) Trailer; /* make a struct of it */ Bytes = LT->DataSize0; Swap = LT->ByteSwap0; /* get byte swap code */ Dest = (unsigned int *) LT->PAddress0; LastSequence = LT->Sequence0;

27. User Mode library example /* From wu_arl/utilities/APIC/FileTransfer/FileRead.c */ /* In Process36K(): Write Data to a file */ if (fd) { int *s1 = (int *) ((unsigned char *) DataHandle->ubuf + ((offset - APIC_AAL5_PAYLOAD_PAGES) * 4096)); int *s2 = s1; int q = 0; for (q = 0; q < 512 * 8; q++) /* swap words for output */ { int x = *s2; *s2 = *(s2 + 1); *(s2 + 1) = x; s2 += 2; } out = write(fd, s1, LT->DataSize0); }

28. AAL5Generator • Uses APIC User Mode Library • Configures one APIC descriptor/buffer pair • Sets descriptor to point to itself and be Read-Only • INFINITE LOOP!! (but it’s a good infinite loop) • APIC will transmit the buffer until descriptor is turned off • After a given length of time (approximate!!) • Turns descriptor off • Uses APIC global pacer and per VC pacing • controls bit/sec rate and packet/sec rate • Intended for high performance test and analysis • Multiple instances can run at same time on same machine • Companion sendCommand utility • to change characteristics on the fly • Need to add control for APIC Batch size • APIC batches cells transmitted. Controlled by an APIC register • easy to add control, just has not been done yet.

29. sendCommand • (Different then the MSR sendcmd you will see tomorrow) • Provides remote control of AAL5Generator • changes the charateristics of a running AAL5Generator • What can be changed: • Packet Rate • Kbit Rate • Destination Address • Run State: • Quit • Stop • Start

30. sendCommand • Sample Commands: • Sends the command to AAL5Generator on tabby which is listening on port 4444 for incoming commands: • Change the packet per second rate to 10000: • sendCommand –p 4444 tabby “Packet Rate 10000” • Change the Kbit per second rate to 10000: • sendCommand –p 4444 tabby “Kbit Rate 10000” • Change the Destination IP Address to 192.168.212.2: • sendCommand –p 4444 tabby “DestAddr 192.168.212.2” • Stop sending packets: • sendCommand –p 4444 tabby “Stop” • Restart sending packets: • sendCommand –p 4444 tabby “Start” • Quit: causes AAL5Generator to Stop sending and exit: • sendCommand –p 4444 tabby “Quit”

31. Source and Executables • source location: • wu_arl/utilities/APIC/AAL5Generator • binary location: • wu_arl/utilities/bin/$OSTYPE/AAL5Generator • wu_arl/utilities/bin/$OSTYPE/sendCommand

32. AAL5Generator Usage Most Used Command Line Options: [-svci #] (VCI for local sending of AAL5 Frames) [-kbits #] (xmit rate using paced link rate) [-frate #] (Frames/sec) [-seconds #] (running time, 0 means run forever) [-packet #] (20 <= X <=1912) pkt size (bytes) includes IP hdr [-port #] (UDP port on which to receive ctrl commands) [-src #] (IP Source address to put in packet IP Header) [-dst #] (IP Destination address to put in packet IP Hdr)

33. AAL5Generator Usage (cont’d) Less Used Command Line Options: [-pacer #] (0:88Mb/S <= X <= 255:3.5Mb/S pacer clock) [-svpi #] (VPI for local sending of AAL5 Frames (0)) [-rate #] (16 bit divisor whole) [-frac #] (8 bit divisor fractional) [-fsize #] (0<=X<=7) (40,136,232,520,760,1192,1480,1912) [-descr #] (0 < X < 64K initial descr) [-debug ] (print debug info) [-swap ] (htonl() byte swap output) [-apic #] (Linux: Apic Id (0 <= X <= 3))

34. APICCell Library API • Provides a cell manager for sending and receiving APIC control cells • Uses the GBNCELL_CellManager • for now it assumes we are talking to a WUGS • thus it is used to talk to SPCs • Provides classes for APIC Registers • Provides member functions for reading and writing: • External Memory • Global Registers • Transmit Registers • Receive Registers

35. APICCell Library Files • Source location: • wu_arl/wugs/src/common_code/APICCELL • Binary: • wu_arl/wugs/lib/$(OSTYPE)/libAPICCELL.a • Sample test code: • wu_arl/wugs/src/common_code/APICCELL/test_code

36. APICCell Library Example // Class defn from wu_arl/wugs/src/common_code/APICCELL/test_code/RWPacingParam.h class RWPacingParameter : public APIC_OperationBase { private: APIC_CellManager* cellManager; Boolean done; Boolean isRead; public: RWPacingParameter(APIC_CellManager* cm) : APIC_OperationBase() { cellManager = cm; done = FALSE; isRead = TRUE; } void readParameter(); void writeParameter(int p); virtual void failed(APIC_ControlCell& c); virtual void receiveCell(APIC_ControlCell& c); Boolean isDone() { return done;} };

37. APICCell Library Example // Snippet of code from // wu_arl/…./APICCELL/test_code/RWPacingParam.C void RWPacingParameter::writeParameter(int p) { isRead = FALSE; done = FALSE; APIC_ControlCell cell; APIC_RegisterData* tmp_data = cell.writeGlobalRegister ( APIC_RegID_PacingParameter, // write this global apic register TRUE, //cell should return NULL, //data to send NULL causes one to be a data // structure to be created and returned to us getCMData()); //and this CMDATA value tmp_data->setPacingParameter(p); cellManager->send(cell); }

38. APICCell Library Example // Snippet of code from // wu_arl/…./APICCELL/test_code/RWPacingParam.C void RWPacingParameter::receiveCell(APIC_ControlCell &cell) { APIC_RegisterData* data = cell.getData(); cout << endl << "\t" << data->getName() ; if (isRead) { cout << "\t = " << data->getPacingParameter() << endl; } else cout << ": \t returned " << endl; done = TRUE; }

39. Notes

40. Notes