Accessing network hardware

1. Accessing network hardware The Network Interface Controllers are part of a larger scheme used in modern PCs for device control

2. Memory-mapped I/O • Non-Intel processor architectures typically will dedicate a set of memory-addresses to performing communication with hardware devices which are installed in the system • Such architectures allow software to read device-status or write device-commands using ordinary CPU instruction-opcodes, such as ‘mov’, ‘test’, ‘add’, ‘xor’, et cetra

3. I/O ports • But Intel’s original x86 architecture used a different approach in which a separate set of addresses and ‘special’ instructions are employed for accessing system devices • The device addresses are called I/O ports, and the opcodes are named ‘in’ and ‘out’: • The ‘in’ instruction reads a device’s ‘status’ • The ‘out’ instruction writes device ‘command’

4. Range of I/O ports • For devices integrated into a PC system’s motherboard, the port-numbers used were 8-bit values and could be specified within an I/O instruction, as ‘immediate’ data • Example: in $0x64, %al # read port 0x64 • For optional devices plugged into ‘slots’ on the motherboard, a port-number could be a 16-bit value, located in the DX register • Example: in %dx, %al # indirect address

5. Two separate address-spaces accessible by using a wide variety of general-purpose arithmetical and logical instructions memory address-space (4GB) accessed only by using the special ‘in’ and ‘out’ instructions I/O address-space (64KB)

6. Early PCs • Peripheral devices in the early PCs used fixed i/o-ports and fixed memory-addresses, e.g.: • Video memory address-range: 0xA0000-0xBFFFF • Programmable timer i/o-ports: 0x40-0x43 • Keyboard and mouse i/o-ports: 0x60-0x64 • Real-Time Clock’s i/o-ports: 0x70-0x71 • Hard Disk controller’s i/o-ports: 0x01F0-01F7 • Graphics controller’s i/o-ports: 0x03C0-0x3CF • Serial-port controller’s i/o-ports: 0x03F8-0x03FF • Parallel-port controller’s i/o-ports: 0x0378-0x037A

7. The PC’s evolution • It became clear in the 1990s that there would be contention among equipment vendors for ‘fixed’ resource-addresses, which of course were in limited supply • Among the goals that motivated the PCI Specification was the creation of a more flexible scheme for allocating addresses that future peripheral devices could use

8. PCI Configuration Space A non-volatile parameter-storage area for each PCI device-function PCI Configuration Space Header (16 doublewords – fixed format) PCI Configuration Space Body (48 doublewords – variable format) 64 doublewords

9. PCI Configuration Header 16 doublewords 31 0 31 0 Dwords Status Register Command Register Device ID Vendor ID 1 - 0 BIST Header Type Latency Timer Cache Line Size Class Code Class/SubClass/ProgIF Revision ID 3 - 2 Base Address 1 Base Address 0 5 - 4 Base Address 3 Base Address 2 7 - 6 Base Address 5 Base Address 4 9 - 8 Subsystem Device ID Subsystem Vendor ID CardBus CIS Pointer 11 - 10 reserved capabilities pointer Expansion ROM Base Address 13 - 12 Maximum Latency Minimum Grant Interrupt Pin Interrupt Line reserved 15 - 14

10. Three x86 address-spaces accessed using a large variety of processor instructions (mov, add, or, shr, push, etc.) and virtual-to-physical address-translation memory space (4GB) accessed only by using the processor’s special ‘in’ and ‘out’ instructions (without any translation of port-addresses) PCI configuration space (16MB) i/o space (64KB) i/o-ports 0x0CF8-0x0CFF dedicated to accessing PCI Configuration Space

11. Interface to PCI Configuration Space PCI Configuration Space Address Port (32-bits) 31 23 16 15 11 10 8 7 2 0 E N reserved bus (8-bits) device (5-bits) function (3-bits) doubleword (6-bits) 00 CONFADD ( 0x0CF8) Enable Configuration Space Mapping (1=yes, 0=no) PCI Configuration Space Data Port (32-bits) 31 0 CONFDAT ( 0x0CFC)

12. Reading PCI Configuration Data • Step one: Output the desired longword’s address (bus, device, function, and dword) with bit 31 set to 1 (to enable access) to the Configuration-Space Address-Port • Step two: Read the designated data from the Configuration-Space Data-Port: # read the PCI Header-Type field (byte 2 of dword 3) for bus=0, device=0, function=0 movl $0x8000000C, %eax # setup address in EAX movw $0x0CF8, %dx # setup port-number in DX outl %eax, %dx # output address to port mov $0x0CFC, %dx # setup port-number in DX inl %dx, %eax # input configuration longword shr $16, %eax # shift word 2 into AL register movb %al, header_type # store Header Type in variable

13. Demo Program • We created a short Linux utility that searches for and reports all of your system’s PCI devices • It’s named “pciprobe.cpp” on our class website • It uses some C++ macros that expand to Intel input/output instructions -- which normally are ‘privileged’ instructions that a Linux application-program is not allowed to execute (segfault!) • But using ‘sudo’ you can escalate the privilege-level at which this utility-program will be run: $ sudo ./pciprobe

14. Example: network interface • We can identify the network interface controllers in our classroom’s PC’s by class-code 0x02 • Then the subclass-code 0x00 is for ‘Ethernet’ • We can identify the NIC from its VENDOR and DEVICE identification-numbers: • VENDOR_ID = 0x8086 // for Intel Corporation • DEVICE_ID = 0x109A // for 82573L controller • Our second demo-program (i.e, ‘82573pci.cpp’) shows this nic’s full PCI Configuration Space

15. Typical NIC main memory packet nic TX FIFO transceiver buffer LAN cable B U S RX FIFO CPU

16. Intel’s Pro1000 nics • These network controllers implement a large number of 32-bit device registers • Two ways are implemented for accessing these numerous registers: • Using a small range of I/O port-addresses • Using a large range of ‘memory-mapped’ register-addresses

17. PCI Base-Address registers • The key to accessing our NIC’s various registers is provided by values we find in certain PCI Configuration Space locations 32-bit Base-Address Register layout for I/O space unused port-address R S V 1 32-bit Base-Address Register layout for Memory space memory-address R S V P R E F 0 00=32-bit decode, 01=reserved, 10=64-bit decode, 11=reserved

18. Output from ‘83573pci.cpp’ executed on ‘anchor01’ BAR0 (at 0x10): 0xE8200000 (Physical memory-address for NIC’s mapped register-bank) BAR1 (at 0x14): 0x00000000 BAR2 (at 0x18): 0x00005001 (I/O port-address for NIC’s ‘multiplexing’ register-bank) BAR3 (at 0x1C): 0x00000000 BAR4 (at 0x20): 0x00000000 BAR5 (at 0x24): 0x00000000

19. ‘mapped’ advantage • For high-performance (and ‘thread-safe’) access to the NIC’s bank of registers, the memory-mapped option offers very clear advantages over multiplexed I/O access: • Access is faster to memory than to I/O ports • Access to memory is ‘atomic’, whereas the multiplexed access via I/O ports is a 2-step operation, giving rise to ‘race conditions’

20. I/O advantage • But during the system’s ‘startup’ process (before the Operating System is loaded), there’s no alternative but to use the I/O access-method, because the memory-mapped registers are at high addresses which the processor cannot yet reach • For example, in cases where an OS needs to be loaded via a BOOTP network-server

21. Clarification • Don’t confuse PCI Configuration Space registers with our NIC’s device registers! CPU’s memory address-space System’s PCI Configuration Space CPU’s I/O-port address-space NIC’s PCI Configuration Space registers BAR2 BAR0 Network Interface Controller NIC device-registers window

22. In-class exercise • Copy our ‘82573pci.cpp’ source-module to your own directory, then rename it as ‘audio.cpp’ • Your assignment is to modify it so that it will display the PCI Configuration Space registers in our classroom workstation’s sound-card (a ‘multimedia’ class device)