1 / 56

Bringing AFS into the 21 st Century Jeffrey Altman, President Your File System Inc. 5 May 2010

Bringing AFS into the 21 st Century Jeffrey Altman, President Your File System Inc. 5 May 2010. Jeffrey Altman Present and Former Roles. Kermit Developer, OS/2, Win95, NT, … Kerberized Telnet, SSH, secure data transfer MIT Kerberos Core Team member Kerberos for Windows Architect

aric
Download Presentation

Bringing AFS into the 21 st Century Jeffrey Altman, President Your File System Inc. 5 May 2010

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Bringing AFS into the 21st CenturyJeffrey Altman, President Your File System Inc.5 May 2010

  2. Jeffrey AltmanPresent and Former Roles • Kermit Developer, OS/2, Win95, NT, …Kerberized Telnet, SSH, secure data transfer • MIT Kerberos Core Team member • Kerberos for Windows Architect • OpenSSL contributor • IETF Security Directorate participant • Internet Access Methods, Chief Technology Officer • Project JXTA, Founding Board Member • OpenAFS, Gatekeeper and Elder • Secure Endpoints, Inc., President and Founder • Network Identity Manager, KCA, Heimdal HSM KDC • Your File System, Inc., President and Founder

  3. OpenAFS 10 Years and Counting • OpenAFS was formed on 1 Nov 2000 • The original Elders represented IBM, Intel, Morgan Stanley, Carnegie Mellon, MIT, and Umichigan • Since then ohloh.net says that OpenAFS has become one of the largest open source projects • 233 developers since inception (51 active in the last year) • Nearly 1 million lines of source code and 100,000 lines of user and developer documentation • All major operating systems (except mobile) are supported • Untold millions of end users (no way to measure)

  4. The Beginning of the Story … • 1983-88 Andrew Project at Carnegie Mellon University creates AFS to support a distributed, heterogeneous, workstation based, computing environment • Funded by IBM to compete with DEC funded Athena at MIT • Unique properties of AFS include • Federated authentication model based • Sophisticated Access Control model • Location independent data access permits zero downtime maintainance • Heterogeneous support including PCs

  5. Commercialization • 1988 Transarc is formed • Founding member of the Open Group • Must beat SunOS in order to beat NFS (v3) • Creates AFS4 which becomes the DCE File System (DFS) • 1994 Transarc acquired by IBM • AFS3 becomes a legacy product • DFS and the Encina Transaction Process Monitor become the focus of IBM Pittsburgh Labs • AFS3 is bundled with WebSphere

  6. Significant Deployments:Morgan Stanley • 1994 Morgan Stanley deploys Kerberos and AFS as part of the Aurora project • The challenge: To come up with a distributed systems environment that would allow Morgan Stanley to centrally manage tens of thousands of systems spread out over more than 30 offices on virtually every continent on the globe in a fully production fashion. The solution: The Aurora System. (Unix only) • 2002 WinAurora is developed and deployed • Today, 80,000 Windows workstations and 20,000 application servers. 30,000 applications hosted in AFS. • Makes use of registry virtualization and environment block injection from AFS name space. No local configuration.

  7. Significant Deployments:U.S. Government • Dept of Energy Labs • U.S. High Performance Computing centers • NASA • Internal Revenue Service • U.S. Geological Survey • Naval Research Lab • “Star Wars” Missile Defense Shield

  8. Significant Deployments:Other commercial • Pictage Photography Services • General Electric Aircraft Engine Division • Goldman Sachs • Hitachi • Qualcomm • United Airlines • Many that are not publicly known

  9. Significant Deployments:Education • Carnegie Mellon University • Cornell University • Duke • Harvard • MIT • PSU • UC-Berkeley and UC-Santa Barbara • Stanford • Univ of Michigan • U of Wisconsin-Madison • Iowa State University • California Inst of Technology • Univ of North Carolina • Chapel Hill • Charlotte (Mosaic) • Univ of Stockholm • KTH • Univ of Edinburgh • Many many others

  10. The Road to Open Source • 1996 to 2000, major educational institutions with source code licenses continue to develop enhancements to AFS but are not permitted to share them • Lack of on-going development and increasing prices from IBM generates significant customer backlash • 1998, Derrick Brashear begins to ask IBM for source code access • First rx code is released as public domain • Later a commitment to release much of the AFS sources • Aug 2000, IBM announces a plan to open source AFS giving responsibility back to the educational community

  11. The OpenAFS Era • Oct 31 2000, IBM publishes OpenAFS 1.0 to the IBM DeveloperWorks web site • Nov 1 2000, OpenAFS.org is born • Nov 5 2000, OpenAFS 1.0 ships • Nov 2001, OpenAFS 1.2 ships • Nov 2005, OpenAFS 1.4 ships

  12. OpenAFS 1.4 • The last major update of OpenAFS (1.4) was announced on its 5th birthday, 1 November 2005 four years after 1.2. • The release took almost four years to develop and included: • Significant performance and stability improvements • Server support for mobile clients and NATs • Audit logging • vos copy, vosconvertROtoRW, parallel attach on restart • Windows clients that worked • AIX5, HPUX11.23, Solaris 10, Linux 2.6, MacOS 10.4

  13. Where does UNCC Mosaic fit in? • Originally built on Transarc Windows AFS Client 3.4a (1996) • First AFS cache for Windows in AFS 3.5 (1999) • AFS 3.6 (March 2000)/OpenAFS 1.0 (November 2000) • By 2003, 900 workstations, 25 servers, 150+ Sun Solaris apps, 80+ Windows XP apps, and 4700 user accounts. 2006, Introduction of the Windows Remote Desktop Web System Utilizing AFS • Today, more of everything but pretty much the same architecture • 200+ applications on Windows

  14. My Introduction to OpenAFS • I started working on OpenAFS in late 2003 • A five day project to add Kerberos v5 authentication to afscreds.exe became 5 weeks, then 5 months and now 7 years • Rodney’s presentation at the 2004 AFS Workshop at SLAC was followed by my first presentation to the community on the state of the Windows client • A new Sherriff is in town • Prior to November 2003 the OpenAFS Windows had received no love. • Frustration in the user community was boiling over.

  15. “Why don’t you promote OpenAFS more?- Rodney Dyer, 2006 Workshop • OpenAFS’s 5th Anniversary saw the release of 1.4 • I refused to promote OpenAFS because I didn’t want to make promises I couldn’t keep about code quality or performance • In the first five years, the OpenAFS community did very little but put out fires • Distributed systems are hard • Multi-threaded systems are hard • Heterogeneous systems are hard • Kernel development is hard • Doing all is nearly impossible • OpenAFS sources included just about every mistake imaginable

  16. Windows Client Read Performance:Cached Data

  17. Windows Client Read Performance:Uncached Data

  18. Windows Client Write Performance

  19. OpenAFS Roadmap? Or Wish List? • Every Workshop a roadmap is presented • but its not a roadmap • No commitments • No delivery dates • How are you supposed to plan your rollout schedule? • The problem is lack of resources • Gatekeepers/Elders compile lists of requests but have little influence on what people work on

  20. YFS Inc. Founded to Drive Demand Globally Accessible File Systems • Open source projects are funded by organizations that are dependent upon the technologies • The benefits of AFS are lost of the vast majority of the world • MobileMe, BigVault, DropBox, and similar sync and access cloud storage services are far behind the capabilities of AFS • YFS will provide services to direct to home, small business, and enterprise users and indirect through telecommunication companies • With hundreds of millions of users, there is a business case for enhancing the software on a regular basis

  21. The Mission • Develop, deploy, and operate “Write once, Access anywhere” global storage solutions • Support the on-going development of utilized open source technologies • Attempt to correct the HTTP mistake • The world wide web is wonderful but HTTP is a horrible protocol implemented at the wrong layer in the OS stack • URLs equate to global file system paths • Static web pages equate to files • Web service APIs equate to distributed named pipe RPCs • AFS Access Control and Federated Authentication is decades ahead of the web

  22. U.S. Department of Energy Small Business Innovative Research Grant • The DoE labs are large users of AFS to support their HEPiX research • YFS Inc. applied for a grant in 2007 • In 2008, received $99,000 to fund Rx improvements and a feasibility study • August 2009 was awarded $650,000 to standardize, design, and implement core protocol enhancements • All grant funded work will be open sourced

  23. Your File System Requirements • Server scalability (~60,000 clients per server vs ~1000) • Networking Improvements • 10GBit networks • IPv6 • TCP and/or SCTP in addition to UDP communications • Optimized file change notification protocol • Read/write replication in addition to read-only replication • Server based virtual query volumes • Directory improvements • Internationalization, Extended Attributes, Multiple Data Streams per Object • Mandatory locking • End-to-end Security • AES-256 encryption • Both Kerberos and X.509 certificates for authentication • Per Service Keys • Anonymous Client Access is Protected

  24. YFS Phase I Success • See openafs-info archive 10/2/2008 e-mail • Rx Packet Management Issues addressed in 1.4.8 and 1.5.53 • 1.4.8 Rx stack is capable of 124MB/sec over a 10Gbit link

  25. YFS Phase II First Year Road Map • Rx Improvements • Path MTU Discovery • Large Data Buffers • New Jumbograms • Window Size Negotiation • Dynamic Retransmit Calculation • Max Call Negotiation • Async API • TCP transport • Protection Service • Anonymous Machine Accounts • Ubik enhancements • RxGK • Client Improvements • Byte Range Locking • Direct and Synchronous I/O • Demand Prefecting

  26. YFS Phase II Second Year Road Map • Server Improvements • Event driven workflow • Posix EA backend • Service Port Independence • Split Horizon Support • Volume Release Optimizations • Read Write Replication • Extended Attributes • Partition UUIDs • Long Volume Names • Per File ACLs • Modern Directory Format • RxTCP IPv6 Support

  27. OpenAFS Roadmap! Not a Wish List • At Fall HEPIX OpenAFS committed to a road map of deliverables over the next two years. • 1.6 Spring/Summer 2010 • 1.8 Fall/Winter 2010 • 2.0 Spring/Summer 2011 • 2.x Fall/Winter 2011 • An aggressive schedule to say the least. Especially given the commitments but it can be done.

  28. OpenAFS 1.6 • Its been more than four years. 1.4.x releases have received many bug fixes and even some new features and performance improvements but major change has all been held back for 1.6. • Other than Windows which is always using the 1.5.x series for production. • What has taken so long? • Source Code Quality and Demand Attach File Service

  29. 1.6: Source Code Quality • When 1.5 was branched there were close to 20,000 warnings produced as part of the x86 MacOS X build • Today it is possible to build the entire source tree excluding 21 files without warnings • In the process hundreds of real bugs were fixed • As was evident from 1.2 instability, there were many lock safety issues resulting in race conditions. Today there are many fewer. • Prior to the release of 1.6, YFS Inc. will complete a regression test harness that will permit the testing of failure cases in addition to those that are expected to succeed.

  30. 1.6: Rx Performance Improvements • Packet leaks, free packet queue management • MTU size negotiation failures • RTT calculation errors • Unnecessary lock contention • Rx statistics • NewCallvsEndCall • All Write and Read paths • Races due to improper locking • Window size errors • Transmit queues dumped packets on the floor • NAT Keep-alive support • > 260MB/second per Rx connection • File Server performance restricted by global locks above the rx layer

  31. 1.6: Linux Cache Manager • Performance improvements • Dynamic allocation of AFS kernel cache entries to support inotify()-pinned entries • Path MTU detection

  32. Linux Cache read performance:AFS should match ext3 below 1GB

  33. 1.6: MacOS X Cache Manager • Many Finder Improvements • Authentication events now refresh • Insert only dropboxes • Improved installation experience • GUI queries for local cell information • AFS Command Preferences Pane • Kerberos v5 ticket renewal • Growl notification service integration • Significant Rx event handling improvements • Bulk-stat RPC support for faster directory enumeration

  34. 1.6: Demand Attach File Service • an enhanced volume management library that supports: • lock-less I/O • on-demand attachment of volumes • parallel shutdown of the file server • on-line salvaging of volumes • automatic detachment of inactive volumes • a new salvageserver daemon which can salvage volumes on-demand • a modified bos and bosserver • fileserver state saving and restoration • host and callback state

  35. 1.6: Other • Major Documentation Improvements • NFS -> AFS translator for Linux • DNS SRV record support (replaces AFSDB records) • /afs/.:mount/cell:volume[:vnode:uniq] direct object access • Larger than 2TB partitions (1.4 backport) • Tivoli X/Open Backup API • Libuafs (userlandafs cache manager library) • AIX6, FreeBSD7.x,8.x, Solaris11, …

  36. 1.6: Microsoft Windows • Nothing new for 1.6. Everything is already in 1.5.74 • Support for all existing operating systems from Windows 2000 to Win7/2008-R2 • Fine grained locking everywhere • Performance is bound by the SMB implementation • Unicode character set support • Native client running on my Win7 laptop to be integrated into 1.7.

  37. What happens Post 1.6? • When 1.6 branch is cut for release candidates, the master branch becomes 1.7 • All major submissions ready for 1.8 will begin to merge onto the master • In order for this to happen in an orderly fashion, projects must be able to break their code into small patch sets for submission to http://gerrit.openafs.org/ • One change per patchset • Each patchset reviewable in less than an hour • No patchset may break the build or reduce stability • Documentation to reviewers describing the protocol changes, architecture, and patch submission plan is strongly advised.

  38. 1.8 Feature Targets • Heimdal crypto replaces OpenAFS crypto • rxk5 security class • Object storage • Native AFS redirector client for Microsoft Windows (no support for Windows 2000) • Rx UDP performance improvements • Window Size Negotiation* • Dynamic Retransmit Calculation* • Path MTU Discovery • Large Data Buffers • Improved Jumbograms • Max Call Negotiation

  39. 1.8: More Feature Targets • PTS authentication name extensions • Kerberos v5 and extendible to other name forms (GSS, X.509, SCRAM, …) • Extended callbacks • Significant reductions in network traffic • More Linux Cache Manager enhancements • Byte Range Locking • Direct and Synchronous I/O • Demand Prefetching • PthreadedUbik servers

  40. 2.0: Feature Targets • rxgk security class • Kerberos v5, X.509 and SCRAM authentication • Protection of anonymous connections • Protection of the server to client callback connection • Permitting full use of Extended Callbacks • Metadata changes can be sent from server to clients as part of the notification avoiding even more network traffic and reducing cross-client change contention • File server coordinated byte range locking • Whatever else is ready based on work from YFS,Inc and others

  41. Unfunded Wish List • Many things are not funded and not on the roadmap • Direct vicep access for Lustre or dCache • dCache as an OSD backend • Faster metadata performance in the file server backend • Improved Fetch/Store Data RPCs • Scatter / gather variants • Fetch Data with Hash • Avoid retransmitting data that is already valid in the cache • Multiple writers use-case • More File Servers per cell • Unix CM Profiling and use of Fine Grained Locking to improve concurrency • Direct to object mount points • On-the-fly volume splitting and / or striping • LDAP backend for Protection Server • Native Windows client • Initial version in 1.8 but there are many improvements that can be implemented • AFS Explorer Shell integration • AFS PAGs for MacOS X • ZFS specific backend for AFS File Server • Disconnected AFS Usability Improvements • Performance Monitoring Instrumentation • Extended Attributes and Multiple Data Streams

  42. How to Move from Wish List to Road Map Targets? • There is not enough money nor developers to implement all of the functionality in the next two years • Implementation designs and Cost/Time estimates for each of the proposals must be developed • Priorities need to be determined not only by the funders desires but should include what the OpenAFS leadership believes is necessary to further adoption • This must include client side usability improvements • User Shell integration (Explorer, Finder, Gnome, …) • Porting Network Identity Manager to Linux and MacOS

  43. OpenAFS Governance is Key • Incorporation or Joining an Umbrella organization is blocked by the IBM trademarks of “AFS” and “OpenAFS” • Once the necessary permissions for use are obtained, the not-for-profit corporation must be formed so that funds can be raised and pooled efficiently • Priorities would be set via a Technical Advisor Board (TAB) consisting of all large contributors, representatives of medium sized contributors, and representatives of individual users and developers • Gatekeepers would be advisors to the TAB providing expert review of proposals and producing architecture design documents • The corporation would issue RFQs to find developers to implement the approved designs, communicate with the standards communities, and manage the contractors • The Gatekeepers would be compensated for their time and an Executive Director would be hired to handle administrator functions

  44. Network Identity Manager

  45. Heimdal Kerberos • MIT has for all practical purposes abandoned the Windows platform • Secure Endpoints is porting Heimdal to Windows • Including the KDC, KCA and Administration Services • Intends to support a Hardware Secure KDC option

  46. WinAurora Technologies • Your File System, Inc. will be migrating WinAurora to the Windows 7 platform • Morgan Stanley has agreed in principal to open source the underlying technologies • YFS hopes to build a public database of application configurations that will permit organizations to quickly deploy applications to thousands of desktops from AFS

  47. Registry Virtualization • A kernel driver virtualizes the registry on a per application basis using hives that are stored within AFS • At process startup, an environment block configuration is injected • Custom Access Control Entry strings are stored in the process default access control list (DACL) to identify the virtual registry associated with the process • Local Procedure Calls communicate the DACL to the executing service thread permitting the registry configuration to be used during all stages of process execution

  48. OpenAFS End User Experience • The biggest bang for the buck comes from upgrading end user experience • Improving the end user experience will increase the demand for the service • Users do not ask for particular technologies • No user ever said they wanted WebDav storage • Here are some ways that the AFS experience can be improved for end users on Microsoft Windows • Similar improvements can be made on other operating system environments

  49. Explorer Shell Integration:AFS Column Selection

  50. Explorer Shell Integration:AFS MetaData within Columns

More Related