Network Interface Card & its Linux programming

1. Network Interface Card & its Linux programming Dr A Sahu Dept of Comp Sc & Engg. IIT Guwahati

2. Outline • PCI Devices • NIC Cards • NIC card architecture • Access to NIC register • PCI access

3. Major Player of NIC Cards • RealTek (Example RTK 8169S) • Broadcom (Example BCM5751) • Intel (Example 82573L NIC) • Download manual from vender site • Or search “Realtek 8169 pdf” in google

4. Detecting PCI Devices : Early PCs • Peripheral devices in the early PCs used fixed i/o-ports and fixed memory-addresses, e.g.: • Video memory address-range: 0xA0000-0xBFFFF • Programmable timer i/o-ports: 0x40-0x43 • Keyboard and mouse i/o-ports: 0x60-0x64 • Real-Time Clock’s i/o-ports: 0x70-0x71 • Hard Disk controller’s i/o-ports: 0x01F0-01F7 • Graphics controller’s i/o-ports: 0x03C0-0x3CF • Serial-port controller’s i/o-ports: 0x03F8-0x03FF • Parallel-port controller’s i/o-ports: 0x0378-0x037A

5. The PC’s evolution • It became clear in the 1990s that there would be contention among equipment vendors for ‘fixed’ resource-addresses, which of course were in limited supply • Among the goals that motivated the PCI Specification was the creation of a more flexible scheme for allocating addresses that future peripheral devices could use

6. PCI Configuration Space A non-volatile parameter-storage area for each PCI device-function PCI Configuration Space Header (16 doublewords – fixed format) PCI Configuration Space Body (48 doublewords – variable format) 64 doublewords

7. PCI Configuration Header 16 doublewords 31 0 31 0 Dwords Status Register Command Register Device ID Vendor ID 1 - 0 BIST Header Type Latency Timer Cache Line Size Class Code Class/SubClass/ProgIF Revision ID 3 - 2 Base Address 1 Base Address 0 5 - 4 Base Address 3 Base Address 2 7 - 6 Base Address 5 Base Address 4 9 - 8 Subsystem Device ID Subsystem Vendor ID CardBus CIS Pointer 11 - 10 reserved capabilities pointer Expansion ROM Base Address 13 - 12 Maximum Latency Minimum Grant Interrupt Pin Interrupt Line reserved 15 - 14

8. Three IA-32 address-spaces accessed using a large variety of processor instructions (mov, add, or, shr, push, etc.) and virtual-to-physical address-translation memory space (4GB) accessed only by using the processor’s special ‘in’ and ‘out’ instructions (without any translation of port-addresses) PCI configuration space (16MB) i/o space (64KB) i/o-ports 0x0CF8-0x0CFF dedicated to accessing PCI Configuration Space

9. Interface to PCI Configuration Space PCI Configuration Space Address Port (32-bits) 31 23 16 15 11 10 8 7 2 0 E N reserved bus (8-bits) device (5-bits) function (3-bits) doubleword (6-bits) 00 CONFADD ( 0x0CF8) Enable Configuration Space Mapping (1=yes, 0=no) PCI Configuration Space Data Port (32-bits) 31 0 CONFDAT ( 0x0CFC)

10. Reading PCI Configuration Data • Step one: Output the desired longword’s address (bus, device, function, and dword) with bit 31 set to 1 (to enable access) to the Configuration-Space Address-Port • Step two: Read the designated data from the Configuration-Space Data-Port • Already discussed PCI-probes pciprobes.c • Lect 29..Showing vram, pciprobe.cpp

11. Example: network interface • We can identify the network interface controller in PC’s by class-code 0x02 • The subclass-code 0x00 is for ‘ethernet’ • We can identify the NIC from its VENDOR and DEVICE identification-numbers: • VENDOR_ID = 0x14E4 • DEVICE_ID = 0x1677 • You can use the ‘grep’ command to search for these numbers in this header-file: </usr/src/linux/include/linux/pci_ids.h>

12. Vendor’s identity example • The VENDOR-ID 0x14E4 belongs to the Broadcom Corporation • Information about this firm may be learned from the corporation’s website • The DEVICE-ID 0x1677 is used to signify Broadcom’s BCM5751 ethernet product

13. Typical NIC main memory packet nic TX FIFO transceiver buffer LAN cable B U S RX FIFO CPU

14. Packet filtering capability • Network Interface’s hardware needs to implement ‘filtering’ of network packets • Otherwise the PC’s memory-usage and processor-time will be wasted handling packets not meant for this PC to receive network packet’s layout Destination-address (6-bytes) Source-address (6-bytes) Each data-packet begins with the 6-byte device-address of the network interface which is intended to receive it

15. Your NIC’s unique address • You can see the Hardware Address of the ethernet controller on your PC by typing: $ /sbin/ifconfig • Look for it in the first line of screen-output that is labeled ‘eth0’, for example: eth0 Link encap: Ethernet HWaddr 00:11:43:C9:50:3A

16. Demo “nic_pci_info.c” • Lets write a kernel module that lets users see certain register-values which pertain to the network interface in your system : • (1) the PCI Configuration Space registers • (2) the Media Access Controller’s address • It also shows your machine’s node-name (in case you want to save the information)

17. How we got the MAC-address • We do not have NIC’s programming datasheet -- but we do have Linux source code for the ‘nic_pci_info.c’ device-driver, which includes a header-file ‘tg3.h’ found here: </usr/src/linux/drivers/net/> • If you scroll through the #define directives you will see the offset where the hardware address is stored in the memory-mapped register-space of the ‘nic_pci_info.c’ interface

18. How we get PCI_NIC registers 16 doublewords 31 0 31 0 Dwords Status Register Command Register DeviceID 0x1677 VendorID 0x14E4 1 - 0 BIST Header Type Latency Timer Cache Line Size Class Code Class/SubClass/ProgIF Revision ID 3 - 2 Base Address 1 Base Address 0 5 - 4 Base Address 3 Base Address 2 7 - 6 Base Address 5 Base Address 4 9 - 8 Subsystem Device ID Subsystem Vendor ID CardBus CIS Pointer 11 - 10 reserved capabilities pointer Expansion ROM Base Address 13 - 12 Maximum Latency Minimum Grant Interrupt Pin Interrupt Line reserved 15 - 14

19. Linux helper-functions #include <linux/pci.h> structpci_dev *devp; unsigned intiomem_base, iomem_size; void *io; devp = pci_get_device( 0x14E4, 0x1677, NULL ); if ( !devp ) return –ENODEV; iomem_base = pci_resource_start( devp, 0 ); iomem_size = pci_resource_len( devp, 0 ); io = ioremap( iomem_base, iomem_size ); if ( !io ) return -EBUSY;

20. Big-Endian to Little-Endian Broadcom network interface storage-addresses 0x0410 0x0411 0x0412 0x0413 0x0414 0x0415 0x0416 0x0417 mac 1 mac 0 mac 5 mac 4 mac 3 mac 2 mac 0 mac 1 mac 2 mac 3 mac 4 mac 5 Intel IA-32 character-array storage

21. Little-Endian to Little-Endian Intel network interface storage-addresses 0x5400 0x5401 0x5402 0x5403 0x5404 0x5405 0x5406 0x5407 mac 0 mac 1 mac 2 mac 3 mac 4 mac 5 mac 0 mac 1 mac 2 mac 3 mac 4 mac 5 Intel IA-32 character-array storage

22. For Intel NICs • For Intel NICs : #define VENDOR_ID 0x8086 // Intel Corp #define DEVICE_ID 0x109A // 82573L NIC • Intel’s filter-register at offset 0x5400 uses the ‘little endian’ storage-convention

23. Ethernet LAN host-1 host-2 host-3 HUB “Collision Domain” host-4 CSMA/CD = “Carrier Sense Multiple Access/Collision Detection”

24. Acronyms • PCI = Peripheral Component Interconnect • MAC = Media Access Controller • Phy = Physical-layer functions • AMT = Active Management Technology • LOM = LAN On Motherboard

25. Intel 82573L NIC: Hardware Features • 32K configurable RX and TX packet FIFO • IEEE 802.3x Flow Control support • Host-Memory Receive Buffers 16K/256K • IEEE 802.3ab Auto-Negotiation • TCP/UDP checksum off-loading • Jumbo-frame support (up to 16KB) • Interrupt-moderation controls

26. External Architecture MDI interface 10/100/1000 PHY GMII/MII interface MDIO interface SM Bus interface LED indicators MAC/Controller EEPROM S/W Defined pins Flash interface PCI/PCI-e Bus

27. Access to PRO/1000 registers • Device registers are hardware mapped to a range of addresses in physical memory • You obtain the location (and the length) of this memory-range from a Base Add register in the nic device’s PCI Configuration Space • Then you request the Linux kernel to setup an I/O ‘remapping’ of this memory-range to ‘virtual’ addresses within kernel-space

28. i/o-memory remapping Local-APIC APIC registers IO-APIC nic registers 1-GB vram nic registers kernel code/data user space vram 3-GB dynamic ram physical address-space ‘virtual’ address-space

29. portability syntax • Linux provides device-driver writers with some macros for accessing i/o-memory: #include <asm/io.h> unsigned int datum; iowrite32( datum, address ); datum = ioread32( address );

30. module_init() #include <linux/pci.h> #include <asm/io.h> #define E1000_STATUS 0x0008 unsigned int iomem_base, iomem_size; void *io; // remap the device’s i/o-memory into kernel space devp = pci_get_device( VENDOR_ID, DEVICE_ID, NULL ); if ( !devp ) return –ENODEV; iomem_base = pci_resource_start( devp, 0 ); iomem_size = pci_resource_len( devp, 0 ); io = ioremap_nocache( iomem_base, iomem_size ); if ( !io ) return –ENOSPC; // read and display the nic’s STATUS register device_status = ioread32( io + E1000_STATUS ); printk( “ Device Status Register = 0x%08X \n”, status );

31. Device Status (0x0008) 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 ? 0 0 0 0 0 0 0 0 0 0 0 GIO Master EN 0 0 0 some undocumented functionality? 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 0 0 0 0 0 PHY reset ASDV I L O S SPEED S L U 0 TX OFF Function ID 0 0 L U F D FD = Full-Duplex LU = Link Up TXOFF = Transmission Paused SPEED (00=10Mbps,01=100Mbps, 10=1000Mbps, 11=reserved) ASDV = Auto-negotiation Speed Detection Value 82573L

32. Thanks