1 / 8

Solaris IPoIB (ibd) Implementation

Kanoj Sarcar Sr Staff Engineer Sun Microsystems (presented by Bill Strahm). Solaris IPoIB (ibd) Implementation. Architecture Overview. DLPI driver Only supported for GID0 Interop (TopSpin, openib) Software structure: IP | SDP | uDAPL | ibd | | | ...

adia
Download Presentation

Solaris IPoIB (ibd) Implementation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Kanoj Sarcar Sr Staff Engineer Sun Microsystems (presented by Bill Strahm) Solaris IPoIB (ibd) Implementation

  2. Architecture Overview • DLPI driver • Only supported for GID0 • Interop (TopSpin, openib) • Software structure: IP | SDP | uDAPL | ibd | | | ... |IBTF / IBMF | | HCA driver | • Performance

  3. Tools changes (20 byte MAC) • New SIOC*XARP ioctls and “struct xarpreq” • arp(1M) changes • PPPD proxyarp etc • Undocumented helper function ala ether_ntoa, ether_aton • extern char *_link_ntoa(const unsigned char *, char *, int, int); • extern unsigned char *_link_aton(const char *, int *); • ifconfig(1M), snoop(1M) • RARP • /etc/ethers, NIS assume 6 bytes!

  4. Tools changes (IPoIB specific) • DHCP agent • Usage of BROADCAST bit • Setting IB hardware type in DHCP packet in htype • Client id format • Code Len Type |<---------------- Client-Identifier -------------->| • +-----+-----+-----+-----+-----+-----+-----+-------------------....----+ • | 61 | 21 | 00 | 00 (4 octets) | 16 octet GID | • +-----+-----+-----+-----+-----+-----+-----+-------------------....----+ • DHCP Server • No explicit ARP cache setting with DHCP chaddr • Snoop(1M) parses IPoIB, disallows MAC filters

  5. Tools changes (IPoIB specific)Part II • ifconfig(1M) • hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 • inet 10.6.98.56 netmask ff000000 broadcast 10.255.255.255 • ether 0:3:ba:24:4:df • ibd1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 2044 index 4 • inet 192.168.100.100 netmask ffffff00 broadcast 192.168.100.255 • ipib 0:0:4:a:0:0:0:0:0:0:12:34:0:2:c9:1:9:76:57:11 • SNMP (net-snmp) • ifSpeed consistent with draft-ietf-ipoib-ibif-mib-07.txt • ifPhysAddress as MAC address (to match ARP) • octet/packet counts only for ipoib network interface

  6. IPoIB oddities • DLPI receiver indicates “src addr unavailable” for unicast packets • If transmitter guarantees GID0 usage, SM assisted reverse lookup possible • RARP unsupported • Non persistant MAC • Server requires client's MAC address for 3rd party lookup

  7. IPoIB Configuration • One time discovery • Reconfiguration boot or IO discovery command • Create possible ipoib instance per port/Pkey • SM must be up • Interface initialization • IP “plumb” triggers MCG probe as validity check • SM must be up, fabric admin must create IP MCG • Decisions • End nodes do not create IP MCG • End nodes do not have to know IB parameters • Issues • IP multicast group membership requests • Can not wait, can not spool! • Solaris IP stack joins 224.0.0.1 at interface init

  8. Interesting Driver Details • QPN/CQ/WQE/PathRecord • Tx path copy vs registration • GID/QPN to Pathrecord (hash/cache) • Disallow (DLPI) change MAC address • Async thread for (blocking) SM communication • IB link up/down handling • ULP notification for IPMP failover • MCG create/delete trap handling • 2 MCG membership lists (full/sendonly, non) • MCG pathrecord and membership implication

More Related