OFED 1.2 Status and Contents

1. OFED 1.2 Status and Contents April 07 Tziporet Koren

2. Agenda • What is OFED? • OFED Architecture and Components • OFED 1.2 New Features • OFED 1.2 Status • What’s next?

3. OFED - Open Fabrics Enterprise Distribution • Enterprise Working Group (EWG) within Open Fabrics Alliance (OFA) • Collaborative effort to test & release OFA software • Broader test participation • Multi-vendor interoperability • Ready for OS vendor adoption • Support for many distributions • Components – Kernel & User Space • Add-on components for vendors to differentiate above OFA Reduces deployment complexity and cost

4. Agenda • What is OFED? • OFED Architecture and Components • OFED 1.2 New Features • OFED 1.2 Status • What’s next?

5. OFA Linux Software Stack IP Based App Access Sockets BasedAccess Various MPIs Block Storage Access Clustered DB Access Access to File Systems Application Level Diag Tools Open SM User Level MAD API UDAPL User APIs OpenFabrics User Level Verbs & CMA / API User Space SDP Lib Kernel Space Upper Layer Protocol VNIC IPoIB SDP SRP iSER RDS NFS-RDMA RPC Cluster File Sys Connection Manager Abstraction (CMA) Mid-Layer SA Client MAD SMA Connection Manager ConnectionManager Kernel bypass Kernel bypass OpenFabrics Kernel Level Verbs / API Provider Hardware Specific Driver Hardware Specific Driver Common Apps & AccessMethodsfor usingOF Stack Key Hardware InfiniBand HCA iWARP R-NIC InfiniBand iWARP

6. OFED 1.2 Components OFA development Add on • HCA/NIC Drivers • Mellanox, QLogic, IBM, Chelsio • Core: verbs, MAD, SMA, CMA, SA cache • IPoIB • SDP • SRP, iSER • RDS • VNIC • UDAPL • OSM • Diagnostic tools • Bonding module • MPI Components: • MVAPICH • Open MPI • MVAPICH2 • MPI tests:OSU benchmarks, Intel MPI benchmarks, Presta New in 1.2

7. Agenda • What is OFED? • OFED Architecture and Components • OFED 1.2 New Features • Kernel • High Availability • User Level • Management • iWARP • MPI • OFED 1.2 Status • What’s next?

8. Main New Features - Kernel • GA level for the EDC market: • SDP, RDS, High Availability, Storage (iSER and SRP) • Stability improvements • Performance improvements: • New - IPoIB Connected Mode (~1000 MB/Sec) • New - RDS for Oracle • SDP message BW: • 10X for small messages • 5X for medium messages • Scalability: • SDP memory consumption limit

9. High Availability - IPoIB • Fails over from one interface to another on carrier off • Two solutions: • User space: A script that detects carrier off/on events, reconfigures interfaces, sends rarp to notify remote side • Status: GA • Kernel module: Bonding • Covered in a separate talk • Status: Beta

10. High Availability - SRP • Failover between two ports/HCAs • SRP HA is built of two parts: • srp_daemon – discovery and sets up for all possible paths to SRP targets on the fabric • Multipath tool – switches to a different path when a path fails. Current version uses Device Mapper multi-path • Device Mapper (DM) driver from Linux kernel • The persistent binding and HA are provided by user space apps (dm-multipath & dm-multipathd) • Solution works for RedHat EL4 and SLES10 • Status: Beta

11. High Availability – RDS & DSP • RDS: • RDS implementation fully support HA • Requires IPoIB HA to work • SDP: • Does not support HA • Can be implemented for same HCA using APM • For multi-HCA support requires protocol change

12. Main New Features - User Level • libibverbs 1.1: • Fork support (requires apps change) • Better low-level driver handling, including multiple drivers linked in statically • Documentation: man pages • librdmacm (uCMA) 1.0: • Multicast joining from user space • UD support

13. Main New Features - Management • OpenSM: • Routing improvements • Performance improvement to min hop and up/down of over an order of magnitude • New fat-tree and LASH algorithms • SA optional record support “virtually” complete • IB router enablement • SA database dump/restore • Many diagnostic improvements since OFED 1.1 • Covered in DoE tools talk • ibdiagui • GUI for ibdiagnet • Used at SC06

14. OFED 1.2 - iWARP Support • Chelsio NIC supported • Verbs and CMA APIs are the same as Infiniband • ULPs that are supported: • MPI (mvapich2 tested) • uDAPL • Basic Testing: • uDAPL • mvapich2 • NFS-RDMA • Status: Beta

15. Main New Features - MPI • MPI implementations: • MVAPICH: version 0.9.9 • Open MPI: version 1.2.1 • MVAPICH2: version 0.9.8 (New) • Common MPI setup sourcing: • Simple menu-driven interface to choose which MPI implementation to set as the default on a per-user and/or system-wide basis

16. Main New Features - MVAPICH • MVAPICH – Version 0.9.9 • Improved message coalescing: • Reduction of per QP send queues for reduction in memory requirement • Increases the small message messaging rate significantly • Multi-core optimizations: • Optimized scalable shared memory design • Optimized, high-performance shared memory aware collective operations • Multi-port support for enabling user processes to bind to different IB ports for balanced communication performance • On-demand connection management using native IB UD support • Multi-path support for hot-spot avoidance in large scale clusters using LMC • Memory Hook Support provided by integration with ptmalloc2 library

17. Main New Features -Open MPI • Open MPI - 1.2.1 • Improvements to scalability of launching applications on large numbers of nodes • "Installdirs" functionality (install OMPI into one place and then be able to move it elsewhere; good for ISV's) • Support for fork() when using the OF libibverbs • Support for setting fixed limits on registered memory • Fixes for heterogeneous network environments (e.g., different number of IB ports on different hosts)

18. MPI - MVAPICH2 • Includes most of the features of MVAPICH • Performance and scalability comparable to MVAPICH for two-sided communication • Added MPI-2 features (one-sided communication, collectives and datatype) • Integrated Multi-rail support • Multi-threading support (MPI_Thread_Multiple) • RDMACM support for InfiniBand and iWARP • Checkpoint/Restart support for application transparent systems-level fault tolerance

19. Agenda • What is OFED? • OFED Architecture and Components • OFED 1.2 New Features • OFED 1.2 Status • OFED 1.2 Release Status • OFED 1.2 System Matrix • Third Party Components Testing • What’s next?

20. OFED 1.2 Release Status • Feature freeze: Feb 2 • Alpha: Feb 14 • Beta: Mar 14 • RC1: Apr 4 • RC2: Apr 18 • RC3: May 3 • Release: May 16

21. OFED 1.2 System Matrix • CPU Arch: • X86, x86_64, PPC64, ia64 (IB only) • kernel.org: kernel 2.6.20 and 2.6.19 • Novell: • SLES 9 SP3 • SLES 10 (SP1) • Redhat: • RHEL 4 (up3 and up4) • RHEL 5 • Free distros (Fedora, SuSE Pro, Ubuntu) • Basic testing only

22. Third Party Components Testing • Proprietary MPIs: • Intel • HP (over uDAPL) • Proprietary SMs: • Cisco, Voltaire, Qlogic • Storage Targets: • iSER: IP Store (FalconStore), Voltaire FC GW • SRP: Engenio, MTD2000, Areca-1220, DDN, Cisco GW

23. Agenda • What is OFED? • OFED Architecture and Components • OFED 1.2 New Features • OFED 1.2 Status • What’s next?

24. What’s Next? • OFED 1.3 – Oct/Nov 07 • Features that didn’t make it in 1.2: • Minimize integration effort into OS distribution • Definition immediately after 1.2 is out • QoS - collaborate with IBTA to align schedule of software delivery • IPoIB: NAPI • NFS over RDMA integration • Mellanox ConnectX IB HCA support • Including new features • Other features to be agreed upon by OFA and EWG at the conference

25. Summary • OFED becomes the industry standard • OFED 1.2 for the EDC market: • Stability • Performance • High Availability • Scalability • OFED 1.2 for the HPC market: • Scalable for large clusters • Multi-core support • Multi-rail • Performance improvements • Successful collaboration between all participants

26. Thank You