YFS: An Introduction to the next / afs ®

1. YFS: An Introduction to the next /afs® Jeffrey Altman, Daria Brashear, Marc Dionne, & Simon Wilkinson Your File System Inc. and Your File System Ltd. 2014 European AFS and Kerberos Conference

2. Your File System • Your File System Inc. (YFSI) is a New York State Corporation with HQ in Manhattan and registered as a business entity in Canada • Your File System Ltd is a wholly owned subsidiary of YFSI with HQ in London • YFSI is privately owned and operated • YFSI is a Red Hat Partner ISV

3. The /afs® Vision • Location Transparency: one name space • User Mobility: access from any device • Security: Flexible model for authentication, privacy, data protection and access control • Availability: Temporary loss to small groups for short time periods • Integrity: No user initiated backups • Heterogeneity: Multiplatform • Self service: Low Help Desk costs • Atomic Publishing: Software, documentation, web sites, .. • Real time collaboration: Distributed File Locking • Distributed Administration

4. AFS® is 30 years old The vision was decades ahead of its time in 1983 The implementation is decades behind in 2014

5. The Price of Inaction • Limited network throughput • Increased call processing latency • Decreased service reliability and availability • Elevated risk of distributed deadlocks • Inability to use full capability of available hardware • Failure to keep up with competing technologies That /afs is still in use today is a credit to its vision and the strength of its architecture.

6. 13 Years of Open Source Major system rewrites are few and far between “Contractor Model of Support” leads to many small and localized changes A lack of consistent vision and quality control Few incentives to invest in the next 30 years

7. The YFS Difference

8. The YFS Vision for /afs • Application Transparency • Be a Tier One file system on all major OSes • Embrace multi-producer, multi-consumer work flows • Extended Integrity: Disaster Recovery

9. The YFS Vision for /afs • Be performance competitive • Lustre, GPFS, Panasas, … • Best of breed data security • Improved Ease of Use • Designed for the long term

10. Preserving the old while providing the new • Improved performance with existing hardware • Cost reductions due to hardware consolidation • Zero data loss as part of a transition • No flag day required • Mixed deployments are encouraged

11. Performance

12. Performance: Why Does it Matter? Performance issues restrict the jobs that sites are willing to run in /afs Deploying excessive hardware to solve load distribution and fairness problems is expensive Support for multiple file systems costs money, requires additional staff, can result in data duplication and out of sync issues

13. RX Performance • Reduced contention in the listener thread • 10 gbit network interface saturation • 255 packet window size (per call) without degradation • Order of magnitude faster on high latency links • Dynamic Thread Pools • Thread Count limited by OS resources

14. Scalability: Name Space Growth • 64-bit volume IDs • 96-bit (79 octillion) vnode IDs • 64-bit,100ns granular timestamps • 2038 ready • Ubik databases extensible up to 16 exabytes • Partitions, volumes and quotas tracked up to 16 zettabytes

15. Performance: Message Flow Optimizations • Optimized Cache Manager handshakes • Volume Status Information • Reduces number of GetVolumeStatus RPCs • Permits RW / RO data cache sharing • Improved caching of RO volume per user permissions • Fewer FetchStatus RPCs for RO volumes

16. Performance: Fileserver • Host and callback package rewritten • Significantly faster callback breaks • Vnode lock contention dramatically reduced • Distributed writing to shared data sets now possible

17. Performance: POSIX O_DIRECT Open mode supported on some OSes Bypasses VFS cache and AFS cache for both read and write No file threshold to tune Data is copied directly to the caller, or directly from the caller to the file server

18. Security

19. Security: Why Does it Matter? • Data breaches and exposures are followed by a high cost • Public Relations Nightmare • Costs of Identity Theft Detection Services (in U.S.) • Loss of employment for key staff members • Organizational reorganization • Disruption of core mission when forced to address security concerns in crisis mode

20. What is YFS Security? • Multi-layered policies • Flexibility for self service end users • System administrator controls • Network Security • Reduced Information Exposure • Minimal Privilege Services

21. Security: End Users • Self Service Group Management • Per-Object ACLs • Cross directory hard links now permitted

22. Security: System Administrators • Volume ACLs • Limits the permissions that end users can grant

23. Security: System Administrators • Volume Security Policy • Per-Volume minimum acceptable rx connection security properties • File Server Security Policy • Per-server minimum acceptable rx connection security properties • Only volumes with weaker or equivalent security policy can be attached, moved to, or restored to.

24. Security: Network • YFS RXGK Security Class • GSS Kerberos 5 authentication • AES-256 wire privacy and integrity protection • Cell wide key for DB servers • Individual keys for file servers • Per-host keys for BOS Overseer Service

25. Security: Callbacks • YFS protects the callback channel with AES-256 privacy and integrity protection • when rxgk is used for the incoming connection • Avoids leaking information about volume and file ids accessed by a client • Prevents forged messages from invalidate callback state

26. Security: Minimal Privileges • Server Processes execute under a daemon account • Not Root

27. Security: Keyed CMs & Machine IDs • Cache Managers can be issued • a Kerberos keytab • a Protection DB Machine ID • Keyed Cache Managers can use privacy for all connections • Machines IDs are similar to User IDs • Can be placed on ACLs and added to Groups • But are not included in system:authuser

28. New Capabilities

29. File System Extensions • Per File ACLs • Cross directory hard links Extensions for Microsoft Windows • Mandatory Locks • Advisory locks are not enforced by the file server • Symlink Updates • Reparse Points can be updated without FileID change • CreateFile with Lock • Avoids races when simulating OpLock semantics

30. Command Output Clarity • Modifications to human readable and machine readable output • vos examine, listvol, rxdebug, xstat_fs, … • Consolidate output • Introduce consistency across command options • Machine readable output –format is not human formatted • All fields are now separated by single tabs • Easy to import into spreadsheets and databases

31. Library Cleanup • All libraries are thread safe • Built using libtool • Intended for use implementing language bindings

32. libacquire • A library to obtain tokens • rxkad • yfs-rxgk • aklog is a wrapper • Can be linked to pam modules

33. Automated Windows Domain Token Acquisition • Triggered by access denied errors • Automatic Token acquisition using Logon Session Kerberos Credentials • Works with all applications that use • WNet API: Network Providers • Shell API: Explorer, Office, anything with an Open dialog

34. Deployment and Configuration

35. Why are Deployment and Configuration Important? Simplify Server Configuration Provide Extensibility for New Features BOS command lines are limited in length Permit the construction of flexible test suites

36. Flexible Configuration • Greatly improved configuration flexibility and convenience • Custom file layouts are possible • All settings centralized in a single configuration area, single file or directory • A configuration directory can ease distribution of custom options • All command line options can be set in configuration

37. Test Suites • Goal • Provide a test for every new feature, library function, RPC • Provide a test with every bug fix, if possible • Requirements • Ability to spin up the various servers and provide a test configuration • All tests must be able to run as a regular user • Must be able to serve test data not necessarily under /vicep*

38. Extensive test suite coverage A complete test cell can spin up in a few seconds Many tests spin up a cell and destroy it when done, maintaining test independence Client testing through libafscp and fuse client All new features require tests before merging

39. Sample systemdyfs-server.service [Unit] Description=YFS Server Service After=syslog.targetnetwork.target [Service] EnvironmentFile=-/etc/sysconfig/yfs ExecStart=/usr/local/sbin/bosserver -config /s/yfs/server/yfs-server.conf -nofork ExecStop=/usr/local/bin/bos -config /s/yfs/server/yfs-server.conf shutdown hurricane.marcdionne.net -wait -localauth User=yfs Group=yfs [Install] WantedBy=multi-user.target

40. Sample file layout [marco@hurricane /s/yfs/server ]$ ls -l total 60 drwxr-xr-x. 2 yfsyfs 4096 Mar 23 04:00 bos -rw-r--r--. 1 yfsyfs 526 Jul 15 2013 bos.keytab -rwxr-xr-x. 1 yfsyfs 26 Jul 15 2013 cacheinfo drwxrwxr-x. 6 yfsyfs 4096 Jan 11 15:36 data drwxrwx---. 2 yfsyfs 4096 Oct 25 10:47 db -rw-r--r--. 1 yfsyfs 4 Jan 6 09:52 KeyFile -rw-r--r--. 1 yfsyfs 144 Jan 6 09:52 KeyFileExt drwxrwx---. 2 yfsyfs 4096 Mar 25 10:28 local drwxrwxrwx. 2 yfsyfs 12288 Mar 25 10:29 logs -rw-r--r--. 1 yfsyfs 15 Sep 12 2013 ThisCell -rw-r--r--. 1 yfsyfs 114 Dec 19 16:56 UserList -rw-r-----. 1 yfsyfs 2000 Aug 5 2013 vl.keytab drwxrwxr-x. 2 yfsyfs 4096 Mar 26 18:25 yfs-server.conf [marco@hurricane /s/yfs/server ]$ ls -l yfs-server.conf/ total 8 -rw-r--r--. 1 yfsyfs 645 Mar 26 18:25 cellservdb.conf -rw-rw-r--. 1 yfsyfs 792 Mar 4 15:48 yfs-server.conf

41. Sample cellservdb.conf grand.central.org = { description = "GCO Public CellServDB 23 Apr 2008" servers = { penn.central.org = { addr = 128.2.203.61 } grand.mit.edu = { addr = 18.9.48.14 } andrew.e.kth.se = { addr = 130.237.48.87 } } } [cells] example.com = { description = "Test cell" servers = { blizzard.marcdionne.net = { addr = 192.168.0.113 } } } marcdionne.net = { description = "Marc's cell" servers = { hurricane.marcdionne.net = { addr = 192.168.0.107 } } }

42. Sample server configuration [vlserver] keytab = /s/yfs/server/vl.keytab auditlog = /s/yfs/server/logs/audVl [volserver] d = 125 auditlog = /s/yfs/server/logs/audVol [bosserver] auditlog = /s/yfs/server/logs/audBos [ptserver] auditlog = /s/yfs/server/logs/audPt [salvager] auditlog = /s/yfs/server/logs/audSalv [salvageserver] auditlog = /s/yfs/server/logs/audSalvserv [dirpath] SERVER_ETC_DIR = /s/yfs/server SERVER_DB_DIR = /s/yfs/server/db SERVER_LOGS_DIR = /s/yfs/server/logs SERVER_BOSCONFIG_DIR = /s/yfs/server/bos SERVER_LOCAL_DIR = /s/yfs/server/local SERVER_PART_PREFIX_DIR = /s/yfs/server/data [fileserver] d = 125 p = 200 nojumbo = auditlog = /s/yfs/server/logs/audFile security = yfs-rxgk:cryptrxkad:clearrxnullrxkad:crypt

43. Packaging

44. Why Packaging Changes are Important? Installation is the initial experience an end user has with the product If the installation process is frustrating, the end user is likely to be unhappy with the product Lack of digital signatures can block the installation of a package or trigger a frightening dialog

45. Installation Packages • New installation packages • Windows • OSX • Linux • Debian • Fedora • RHEL6 and RHEL7

46. Microsoft Windows®

47. Microsoft Windows® • Single installer • 64-bit and 32-bit components • Heimdal Side by Side Assembly • Heimdal Command Line tools • Automatic Cache Sizing • All components digitally signed • Microsoft Cross Signing of Drivers

48. OSX

49. OSX • Flat package • Integral packages for client, server and development • Digital signatures on the package, the kext and the binaries using Apple-issued certificate

50. Linux New packaging for Debian, Fedora and RHEL Integral packages for client, db services, and file service Digital signatures on installation packages