170 likes | 309 Views
SC’13 BoF Discussion. Sean Hefty Intel Corporation. SC’13 BoF Feedback. Most people I’ve spoken with are highly supportive of the concepts being discussed General alignment Active discussion at SC ‘13 Engagement is high Good participation Want even wider focus outside of MPI
E N D
SC’13 BoF Discussion Sean Hefty Intel Corporation
SC’13 BoF Feedback • Most people I’ve spoken with are highly supportive of the concepts being discussed • General alignment • Active discussion at SC ‘13 • Engagement is high • Good participation • Want even wider focus outside of MPI • Diverse viewpoints helping to drive conversation • More likely to produce better product • Need to understand all concerns • Address as many concerns as reasonable www.openfabrics.org
Main Concerns from BoF • Application centric APIs may require new protocols • Must ensure wire compatibility • RDMA CM provides a simpler abstraction over verbs • How does this effort differ • Some applications want low-level access • Technology specific apps www.openfabrics.org
Evolving OFA Software from the Viewpoint of libibverbs • Analyze libibverbs and librdmacm details • Selected issues • Previously covered fast path call overhead • Identify specific areas for improvement • Abstract the improvements to create a higher-performing, more extensible framework www.openfabrics.org
Current libraries librdmacm Verb Helpers Infiniband verbs interface libibverbs Verbs Provider User to kernel ‘command’ interface www.openfabrics.org
libibverbs Infiniband verbs interface libibverbs Verbs Provider User to kernel ‘command’ interface www.openfabrics.org
libibverbs Structures • structibv_qp • pthread objects(mutex, cond) • internal counter • internal handle • qp data(state, qpn, type) • 5 data pointers • user context • structibv_cq • pthread objects (mutex, cond) • internal counters • internal handle • 2 data pointers • user context Exposes internal implementation • IB specific QP data • Not useful for iWarp • Not needed with rdma cm Data pointers may not be valid or needed by app www.openfabrics.org
libibverbs Structures • structibv_context • structibv_device *device • structibv_context_ops ops; • intcmd_fd; • … Provider implemented functions • Embedded structure – fixed size! • Adding new calls is non-trivial • No mechanism to do so upstream • Provider functions associated with an opened device • Cannot optimize call based on use • Branches in code path and larger data structures www.openfabrics.org
librdmacm No-op for iwarp and RoCE Scalability issues • Original APIs • Create ID, resolve address, resolve route • Connect, listen, accept, reject • Newer APIs • Getaddrinfo, create endpoint • Verb helpers • Register memory • Send/receive, rdma read/write, UD send Required for UD Addresses scalability and ease of use Simplified calls, but limited by verbs API www.openfabrics.org
Evolving ‘Verbs’ • Merge libibverbs and librdmacm • Eliminate duplicate functionality • Remove internal implementation details from the API • Abstract data structures • Associate functionality with specific objects • Enable optimized code paths • Abstract objects for extensibility • Enable other use cases www.openfabrics.org
BoF Concern • Application centric APIs may require new protocols • Must ensure wire compatibility • Transport protocol is exposed by framework • API definitions and protocol definitions are separate • Vendor-specific protocols are already being defined outside of verbs • PSM, MXM, FCA • TBD: discuss related protocols • Address resolution, route lookup, connection setup www.openfabrics.org
BoF Concern • RDMA CM provides a simpler abstraction over verbs • How does this effort differ • RDMA CM abstractions are limited by functions exported by verbs • Still filter to single post send/recv call • RDMA CM is highly dependent on, but maintained separately • RDMA CM use is required for iWarp • RDMA CM functionality is carried forward www.openfabrics.org
BoF Concern • Some applications want low-level access • Technology specific apps • Low-level access will still be possible • Focus is on ease of use (librdmacm model) with high-performance (libibverbs fast path) www.openfabrics.org
(Scalable) Fabric Interfaces Fabric Interfaces Q: What is implied by incorporating interface sets under a single framework? Objects exist that are usable between the interfaces Isolated interfaces turn the framework into a complex dlopen Interfaces are composable May be used together Control Interface Atomics Message Queue RDMA Collective Operations Active Messaging CM Services Tag Matching www.openfabrics.org
Migrating Providers from Verbs to FI RDMA Message Queue CM Services FI libfabric Verbs ibverbs abstraction layer libibverbs RDMA CM Verbs Provider Verbs Provider www.openfabrics.org
Migrating Apps from Verbs to FI • Expose ‘verbs’ interfaces directly from FI • Use macros to convert ‘libibverbs’ exported calls to FI calls • Or layer libibverbs over libfabric • Applications must recompile • Minimal benefit to app • Dependent on accessed data structures and interfaces www.openfabrics.org
Migrating Apps from Verbs to FI • Define ‘verbs’ compatibility mode • Allows mapping objects between interfaces • E.g. QP fabric socket CQ EC • Restricts implementation • Mapping must be documented • Identified by protocol • Allow software to adopt new interfaces selectively • E.g. send/recv/ec_read FI verbs libibverbs libfabric Dual-Provider Library Verbs Provider FI Provider www.openfabrics.org