E128 Implementing ASE on Network Appliance Filers Using the Direct Access File System

1. E128 Implementing ASE on Network Appliance Filers Using the Direct Access File System • Stephen Daniel • Database Performance Architect • Network Appliance • Steve.Daniel@netapp.com

2. Storage Options • Storage for large databases – blocks or files? • Advantages of Files • Easier to manage • Create • Spread across multiple volumes • Grow • Copy

3. Storage Options • Files Provide Excellent Storage Virtualization Servers SAN Common Storage Pool

4. Storage Options • Storage for large databases – blocks or files? • Advantage of Block Storage (raw partitions) • Speed

5. Manageability of Files with the Performance of Blocks • Why are files slow? • Local File systems • Code path length • Network File Systems • Code path length • Data copies • Data copying is required by traditional network protocols

6. Manageability of Files with the Performance of Blocks • Components of a Solution • Manageability of files • Use an external storage system • Efficient sharing of data • Virtualization of the storage pool • Offload file system code path

7. Manageability of Files with the Performance of Blocks • Components of a Solution • Performance of Blocks • Use new RDMA-capable hardware • Allows zero-copy networking • Offloads network protocol stack • New hardware paradigm, RDMA, requires a new software stack

8. Host System Storage System User memory Buffer memory Kernel Kernel Adapter Adapter Remote DMA Technology • Remote DMA (RDMA) Technology LAN or SAN Data flows from user memory to storage system buffers without any data copies

9. Using RDMA – VI Interconnect • VI Messaging Protocol • Protocol created and promoted by Intel • Provides end-to-end management of connections based on RDMA capability • Typically used for cluster interconnects

10. Direct Access File System • What is DAFS • DAFS is a protocol for a network file system • Provides a rich API for managing complex I/O streams • Similar in function to NFS version 4 • DAFS is designed around RDMA technologies • Layered on top of messaging protocol, such as VI

11. DAFS Industry Initiative • DAFS Collaborative • 87 companies; 9 primary spokespersons • DAFS Prototypes developed by 4 companies, 3 universities • DAFS Protocol Spec v1.0 published September 2001 • DAFS API Spec v1.0 published November 2001 • Industry initiative transitioning to next stage • IETF (protocol) • DAT Collaborative (RDMA transport APIs) • DAFS Implementers’ Forum in SNIA

12. DAFS Clients • Three Styles of DAFS Client • Full network file system: fDAFS • Integrated into a kernel, just like NFS • Can provide an efficient direct I/O path • Applications must use standard system calls

13. DAFS Clients • Three Styles of DAFS Client • User-level I/O library: uDAFS • Full implementation of the DAFS API defined by the DAFS collaborative • Reduced dependency on system calls • Dramatically reduced CPU cost of I/O compared to any other form of kernel-mediated I/O, including raw disk partitions

14. DAFS Clients • Three Styles of DAFS Client • Kernel-level device driver: dDAFS • Functionally a disk device driver • A DAFS file appears as a raw partition in /dev • Provides best time-to-market DAFS client

15. DAFS Products • Current DAFS Technology Choices: • DAFS / VI / IP / Ethernet • DAFS / Infiniband • The examples in this talk: • DAFS / VI / IP / Ethernet / Solaris 8

16. User Space SYBASE ASE dDAFS driver Kernel Emulex VIPL and Driver VI Protocol IP Protocol DAFS Software Architecture Runs on the Emulex NIC

17. DAFS Hardware Architecture Database Server Conventional gigabit Ethernet with IP protocol Parallel paths for performance and failover Multi-protocol capable storage devices

18. DAFS Performance Testing Database Server CPU msec/op 113 Direct-attached storage w/ volume manager (VxVM) 89 Direct-attached storage w/ local FS (ufs) 76 Raw access to direct-attached storage 70 dDAFS kernel device driver (w/ VI/IP HBA) 28 User-level uDAFS client (w/ VI/IP HBA) • Sun E3500 (400MHz) w/ Solaris 2.8 • Mixed read/write workload • 4kB transfers; asynchronous I/O

19. TPC-C Performance Testing • Products Used • Sybase ASE 12.5.0.1 • Fujitsu-Siemens Prime Power 850 with 16 CPUs • Solaris version 8 • Network Appliance DDA 1.0 dDAFS driver • Emulex GN9000 VI HBAs • Network Appliance F820 storage systems

20. TPC-C Performance Testing • Manageability of Files • Managed the database using NFS • Build • Copy • Resize

21. TPC-C Performance Testing • Manageability of Files • Network Appliance data management • Snapshots to protect data • Recover from crashes • Undo configuration changes

22. TPC-C Performance Testing • Performance of Blocks • Ran the database using dDAFS • To convert from NFS to dDAFS • Shutdown ASE • Rename the NFS files • ddafs open the NFS files as ddafs devices • Bring up ASE

23. TPC-C Performance Testing Most comparable result...

24. TPC-C Performance Testing

25. TPC-C Results • The Numbers • 112,286.46 tpmC • $ 13.44 / tpmC • Published March 13, 2002 • Available August 31, 2002

26. TPC-C Results • Records (as of publication date): • Best price/performance on UNIX System • Best performance/CPU-MHz on SPARC • NetApp Features enabled during TPC-C: • All storage was RAID protected • 5 Snapshots were active

27. Network Appliance • Network Appliance enables enterprises to simplify, share, scale and manage their information infrastructure.