Detecting PCI devices

1. Detecting PCI devices On identifying the peripheral equipment installed in our PC

2. Early PCs • Peripheral devices in the early PCs used fixed i/o-ports and fixed memory-addresses, e.g.: • Video memory address-range: 0xA0000-0xBFFFF • Programmable timer i/o-ports: 0x40-0x43 • Keyboard and mouse i/o-ports: 0x60-0x64 • Real-Time Clock’s i/o-ports: 0x70-0x71 • Hard Disk controller’s i/o-ports: 0x01F0-01F7 • Graphics controller’s i/o-ports: 0x03C0-0x3CF • Serial-port controller’s i/o-ports: 0x03F8-0x03FF • Parallel-port controller’s i/o-ports: 0x0378-0x037A

3. The PC’s evolution • It became clear in the 1990s that there would be contention among equipment vendors for ‘fixed’ resource-addresses, which of course were in limited supply • Among the goals that motivated the PCI Specification was the creation of a more flexible scheme for allocating addresses that future peripheral devices could use

4. PCI Configuration Space A non-volatile parameter-storage area for each PCI device-function PCI Configuration Space Header (16 doublewords – fixed format) PCI Configuration Space Body (48 doublewords – variable format) 64 doublewords

5. PCI Configuration Header 16 doublewords 31 0 31 0 Dwords Status Register Command Register Device ID Vendor ID 1 - 0 BIST Header Type Latency Timer Cache Line Size Class Code Class/SubClass/ProgIF Revision ID 3 - 2 Base Address 1 Base Address 0 5 - 4 Base Address 3 Base Address 2 7 - 6 Base Address 5 Base Address 4 9 - 8 Subsystem Device ID Subsystem Vendor ID CardBus CIS Pointer 11 - 10 reserved capabilities pointer Expansion ROM Base Address 13 - 12 Maximum Latency Minimum Grant Interrupt Pin Interrupt Line reserved 15 - 14

6. Three IA-32 address-spaces accessed using a large variety of processor instructions (mov, add, or, shr, push, etc.) and virtual-to-physical address-translation memory space (4GB) accessed only by using the processor’s special ‘in’ and ‘out’ instructions (without any translation of port-addresses) PCI configuration space (16MB) i/o space (64KB) i/o-ports 0x0CF8-0x0CFF dedicated to accessing PCI Configuration Space

7. Interface to PCI Configuration Space PCI Configuration Space Address Port (32-bits) 31 23 16 15 11 10 8 7 2 0 E N reserved bus (8-bits) device (5-bits) function (3-bits) doubleword (6-bits) 00 CONFADD ( 0x0CF8) Enable Configuration Space Mapping (1=yes, 0=no) PCI Configuration Space Data Port (32-bits) 31 0 CONFDAT ( 0x0CFC)

8. Reading PCI Configuration Data • Step one: Output the desired longword’s address (bus, device, function, and dword) with bit 31 set to 1 (to enable access) to the Configuration-Space Address-Port • Step two: Read the designated data from the Configuration-Space Data-Port: # read the PCI Header-Type field (byte 2 of dword 3) for bus=0, device=0, function=0 movl $0x8000000C, %eax # setup address in EAX movw $0x0CF8, %dx # setup port-number in DX outl %eax, %dx # output address to port mov $0x0CFC, %dx # setup port-number in DX inl %dx, %eax # input configuration longword shr $16, %eax # shift word 2 into AL register movb %al, header_type # store Header Type in variable

9. Demo Program • We created a short Linux utility that searches for and reports all of your system’s PCI devices • It’s named “pciprobe.cpp” on our CS635 website • It uses some C++ macros that expand to Intel input/output instructions -- which normally are ‘privileged’ instructions that a Linux application-program is not allowed to execute (segfault!) • Our system administrator (Alex Fedosov) has created a utility (named “iopl3”) that will allow your command-shell to acquire I/O privileges

10. Example: network interface • We identify the network interface controller in our classroom PC’s by class-code 0x02 • The subclass-code 0x00 is for ‘ethernet’ • We can identify the NIC from its VENDOR and DEVICE identification-numbers: • VENDOR_ID = 0x14E4 • DEVICE_ID = 0x1677 • You can use the ‘grep’ command to search for these numbers in this header-file: </usr/src/linux/include/linux/pci_ids.h>

11. Vendor’s identity • The VENDOR-ID 0x14E4 belongs to the Broadcom Corporation (headquarters in Irvine, California) • Information about this firm may be learned from the corporation’s website: <http://www.broadcom.com> • The DEVICE-ID 0x1677 is used to signify Broadcom’s BCM5751 ethernet product

12. Typical NIC main memory packet nic TX FIFO transceiver buffer LAN cable B U S RX FIFO CPU

13. Packet filtering capability • Network Interface’s hardware needs to implement ‘filtering’ of network packets • Otherwise the PC’s memory-usage and processor-time will be wasted handling packets not meant for this PC to receive network packet’s layout Destination-address (6-bytes) Source-address (6-bytes) Each data-packet begins with the 6-byte device-address of the network interface which is intended to receive it

14. Your NIC’s unique address • You can see the Hardware Address of the ethernet controller on your PC by typing: $ /sbin/ifconfig • Look for it in the first line of screen-output that is labeled ‘eth0’, for example: • (The NIC’s filter-register stores this value) eth0 Link encap: Ethernet HWaddr 00:11:43:C9:50:3A

15. Our ‘tigon3.c demo • We wrote a kernel module that lets users see certain register-values which pertain to the BCM5751 network interface in your classroom workstation: • (1) the PCI Configuration Space registers • (2) the Media Access Controller’s address • It also shows your machine’s node-name (in case you want to save the information)

16. How we got the MAC-address • We do not have Broadcom’s programming datasheet -- but we do have Linux source code for the ‘tigon3’ device-driver, which includes a header-file ‘tg3.h’ found here: </usr/src/linux/drivers/net/> • If you scroll through the #define directives you will see the offset where the hardware address is stored in the memory-mapped register-space of the ‘tigon3’ interface

17. Driver’s authors • The Linux kernel’s open-source driver for the Broadcom ‘tigon3’ network controller was jointly written by David S. Miller (see photo below) and Jeff Garzik David Miller’s announcement in Feb 2002 of their driver’s BETA version is online. It includes his candid comments about the challenge of writing such a driver when the vendor does not make available its device’s programming documentation.

18. How we got tigon3 registers 16 doublewords 31 0 31 0 Dwords Status Register Command Register DeviceID 0x1677 VendorID 0x14E4 1 - 0 BIST Header Type Latency Timer Cache Line Size Class Code Class/SubClass/ProgIF Revision ID 3 - 2 Base Address 1 Base Address 0 5 - 4 Base Address 3 Base Address 2 7 - 6 Base Address 5 Base Address 4 9 - 8 Subsystem Device ID Subsystem Vendor ID CardBus CIS Pointer 11 - 10 reserved capabilities pointer Expansion ROM Base Address 13 - 12 Maximum Latency Minimum Grant Interrupt Pin Interrupt Line reserved 15 - 14

19. Linux helper-functions #include <linux/pci.h> struct pci_dev *devp; unsigned int iomem_base, iomem_size; void *io; devp = pci_get_device( 0x14E4, 0x1677, NULL ); if ( !devp ) return –ENODEV; iomem_base = pci_resource_start( devp, 0 ); iomem_size = pci_resource_len( devp, 0 ); io = ioremap( iomem_base, iomem_size ); if ( !io ) return -EBUSY;

20. Big-Endian to Little-Endian Broadcom network interface storage-addresses 0x0410 0x0411 0x0412 0x0413 0x0414 0x0415 0x0416 0x0417 mac 1 mac 0 mac 5 mac 4 mac 3 mac 2 mac 0 mac 1 mac 2 mac 3 mac 4 mac 5 Intel IA-32 character-array storage

21. In-class exercise • Copy the ‘tigon3.c’ source-module to your own directory, then rename it ‘anchor.c’ • Your assignment is to modify it so that it will show information about the Intel NICs in our ‘anchor’ cluster’s machines: #define VENDOR_ID 0x8086 // Intel Corp #define DEVICE_ID 0x109A // 82573L NIC • Intel’s filter-register at offset 0x5400 uses the ‘little endian’ storage-convention

22. Little-Endian to Little-Endian Intel network interface storage-addresses 0x5400 0x5401 0x5402 0x5403 0x5404 0x5405 0x5406 0x5407 mac 0 mac 1 mac 2 mac 3 mac 4 mac 5 mac 0 mac 1 mac 2 mac 3 mac 4 mac 5 Intel IA-32 character-array storage