The Consequences of Decentralized Security in a Cooperative Storage System

1. The Consequences ofDecentralized Security in a Cooperative Storage System Douglas Thain, Chris Moretti, Paul Madrid, Phil Snowberger, and Jeff Hemmes University of Notre Dame http://www.cse.nd.edu/~ccl IEEE Workshop on Security in Storage 2005

2. Abstract • Suppose that security in storage has been deployed at all endpoints. • How does this affect the design of distributed storage systems that rely upon these devices? • Clients must become much more: • Fault tolerant, adaptive, and self reliant. • Aware of resource allocation issues. • Helpful to the end user! • Environment: Storage Pool at Notre Dame

3. Traditional File System Security appl appl appl I am John Doe! untrusted network: Ethernet Internet Security Interface placement, replication, reliability File System Abstraction trusted network: PCIRAIDSAN Myrinet Ethernet disk disk disk Owner of Inode 9842 is UID 56 file

4. Decentralized Security appl appl appl placement, replication, reliability Abstr. Abstr. Abstr. untrusted network: Ethernet Internet I am John Doe! Security Security Security disk disk disk Owner of File /foo/bar is John Doe file

5. CooperativeStorage SystematNotre Dame

6. What is Cooperative Storage? • Many devices bound together that can accomplish more than one device alone. • Improve capacity, reliability, performance... • Could be one person, or many cooperating users. • Key property: • Each person retains absolute control of their own resources by setting local policies. • People share and collaborate with others that they know and trust. No free love! No central control! • However, some resources are set up for the common good by an authority. (CS workstations usable by any member of the CS department, says the chair.)

7. App file transfer App Adapter Central Filesystem Distributed Filesystem Abstraction Adapter Distributed Database Abstraction file server file server file server file server file server file server file server 3PT UNIX UNIX UNIX UNIX UNIX UNIX UNIX UNIX UNIX UNIX UNIX UNIX UNIX UNIX Cluster administrator controls policy on all storage in cluster Workstations owners control policy on each machine. App Adapter ??? file system file system file system file system file system file system file system

14. CFS: Central File System appl appl appl adapter adapter adapter CFS CFS CFS file server file file file

15. access data lookup file location DSFS: Dist. Shared File System appl appl adapter adapter DSFS DSFS file server file server file server file file file file file ptr file file file file file ptr ptr

16. DSDB: Dist. Shared Database appl appl adapter adapter DSDB DSDB insert query direct access file server database server file server create file file file file index file file file file file file file file

17. Applications • Simple and Secure Remote Access • CDF: Remote Dynamic Linking • BaBar: Remote Database Access • LHC: Semantic Remote Filesystems • Distributed File Systems • GRAND: Scalable Archive for Online Data • Distributed Databases • GEMS: Molecular Dynamics Simulation • CVRL: Biometric Image Storage/Analysis

18. Challenges of Decentralization • Unbounded Set of Users • There is no global /etc/passwd or /etc/group! • Multiple Identities per User • Kerberos creds from Notre Dame / Wisconsin. • GSI creds from ND/UW/DOE/NCSA. • New Decision Points • Placement decision made, but action fails! • Directory op succeeds, but file creation fails! • Unexpected Policy Coupling • Data placement may affect access control!

19. Outline of Paper • Centralized vs Decentralized Security • Architecture of Cooperative Storage • Basic Security Mechanism • Problem: Complexity Confuses! • Detail: Reservation Right • Challenges • Authorization in Distributed File Systems • Logistics of Third Party Transfer • Mechanisms for Active Storage • Semantics of Distributed Group Management

20. Basic Security Mechanism • Negotiate an Authentication Method • Client proposes, server agrees/disagrees. • Default ordering works for most + manual override. • Different servers/clients may support diff subsets. • Then, Authenticate via Chosen Method • May involve challenges, cert exchange, etc... • Yields a Subject Name for the Session: • kerberos:dthain@nd.edu • globus:/O=NotreDame/CN=DouglasThain • hostname:hedwig.cse.nd.edu • unix:dthain

21. Authorization Mechanism • Unix Access Controls Are Not Sufficient • Integer UIDs are not sufficient for principals. • Nine owner/group/others bits are restrictive. • Mapping from subjects to Unix is a mess. • Place Variable Length ACLs on dirs: globus:/O=NotreDame/CN=DThain RWLAX kerberos:dthain@nd.edu RWL hostname:*.cs.nd.edu RL globus:/O=NotreDame/* RL

22. Problem: Complexity Confuses! • For beginning users: • Negotiated authentication makes life easy. • Everybody can authenticate in some way. • Most users don’t think about it first. • For advanced users: • Negotiation has unexpected effects. • What happens when credentials expire? • For long running / large tasks, better to manually specify the authentication mode. • AuthN failure is easier to retry than authZ failure! • Unexpected authentication is hard to debug. • Full detail logging mode reveals auth algorithm. • Always prominently display subject name in all tools!

23. test.c test.dat a.out cms.exe Problem: Shared Namespace file server globus:/O=NotreDame/* RWLAX

24. /O=NotreDame/CN=Monk /O=NotreDame/CN=Ted mkdir mkdir /O=NotreDame/CN=Monk RWLA /O=NotreDame/CN=Ted RWLA test.c a.out test.c a.out Solution: Reservation (V) Right file server mkdir only! O=NotreDame/CN=* V(RWLA)

25. Outline of Paper • Centralized vs Decentralized Security • Architecture of Cooperative Storage • Basic Security Mechanism • Problem: Complexity Confuses! • Detail: Reservation Right • Challenges • Authorization in Distributed File Systems • Logistics of Third Party Transfer • Mechanisms for Active Storage • Semantics of Distributed Group Management

26. access data lookup file location DSFS: Dist. Shared File System appl appl adapter adapter DSFS DSFS file server file server file server file file file file file ptr file file file file file ptr ptr

27. DSFS Logistics • Consider Creating a File: • Fetch list of resources: • online catalog / static list / user selected • Make placement decision: • random / fill in order / user selected • Create stub file on dir server. (fail?) • Create actual file on data server. (fail?) • Note that two access controls are in play: • One controls access to the namespace. • Another controls access to the data storage.

28. DSFS Applications • Personal Mass Storage • Expand your local filesystem to include all the disks available in a cluster / lab / basement. • Distributed /tmp for Cluster Computing • Harness remote cluster for the duration of a job. • Multi-User Scalable Storage • Department provides directory, but no space. • /O=NotreDame/O=CSE/CN=* RWL • Participants provide their own data servers. • /O=NotreDame/O=CSE/CN=JohnDoe RWLA • Separates provisioning from access!

29. Dealing with Failure • Failure to place data is very common! • Unexpected access controls on device. • Device is temporarily unavailable. (reboot?) • Device is newly installed or creds expired. • Owner changed the sharing policy. • Soln: Client Needs to Model the System • Track successes and failures on each device. • Failed devices are not tried again for a time. • Of course, cannot avoid a device forever...

30. Outline of Paper • Centralized vs Decentralized Security • Architecture of Cooperative Storage • Basic Security Mechanism • Problem: Complexity Confuses! • Detail: Reservation Right • Challenges • Authorization in Distributed File Systems • Logistics of Third Party Transfer • Mechanisms for Active Storage • Semantics of Distributed Group Management

31. PINS: Processing in Storage • Observation: • Traditional clusters separate CPU and storage into two distinct systems/problems. • Distributed computing is always some direct combination of CPU and I/O needs. • Idea: PINS • Cluster HW is already a tighly integrated complex of CPU and I/O. Make the SW reflect the HW. • Key: Always compute in the same place that the data is located. Leave newly created data in place.

32. Compute via Passive Storage Compute Y=F(X) where X={A,B,C,D} F Y1 Y2 Y3 Y4 file server file server file server file server (X 200) A B C D S2 S3 S4 S1

33. Y1 Y2 Y3 Y4 Compute via Active Storage Compute Y=F(X) where X={A,B,C,D} F F F F file server file server file server file server (X 200) A B C D S2 S3 S4 S1

34. storage owner client > open x.nd.edu > put sim.exe > put in.dat > exec sim.exe > get out.dat sim.exe out.dat Technique: Identity Boxing file server / directory ACL: hostname:*.cse.nd.edu RWLX globus:/O=NotreDame/* RWLX Identity Box: /O=NotreDame/CN=Monk sim.exe in.dat

35. Unified Semantics • Same Identity for Exec and Data Access • Stage in data as user X. • Program runs as user X, data is protected. • Access results as user X. • Same ACLs for Exec and Data Access • Need the X right to run a program. • RX rights – given user can run fixed F(X). • WX rights – given user can stage in any F(X).

36. Outline of Paper • Centralized vs Decentralized Security • Architecture of Cooperative Storage • Basic Security Mechanism • Problem: Complexity Confuses! • Detail: Reservation Right • Challenges • Authorization in Distributed File Systems • Logistics of Third Party Transfer • Mechanisms for Active Storage • Semantics of Distributed Group Management

37. Fully Decentralized User Groups • Distributed Orgs Have Complex Needs • CMS Collaboration: 10s of institutions, 100s of PIs, 1000s of graduate students staff. • There is no centralized database for CMS. • Local managers add/remove members locally. • Want Storage Systems that Allow Reference to Groups Managed by Others: • Allow access to all staff involved in CMS. • Allow access to any NSF program manager. • Allow access to all CS faculty at ND/Purdue.

38. read client file server check ACL Access Control List group:ccl.nd.edu/faculty RWL group:serv.nsf.gov/managers RL group:ftp.cern.org/members RL file system file univ.edu univ.edu group lookups ccl.nd.edu serv.nsf.gov ftp.cern.org members members faculty managers members Fully Decentralized ACLs group lookups

39. Challenges of ACLs • Performance / Availability / Consistency • Give the group/ACL owner control. • Specify maximum time for stale data. • Implemented, but continuing experience leads to reflection on the semantics. • Example: What to do under failures? • Partial answer: servers fail quickly, client retries up to a user-controlled limit. • Consider: Group A gives W access, group B gives R. • What happens when group A is unavailable? • Two very different questions: • What rights does user X have? • Can user X perform a read?

40. Outline of Paper • Centralized vs Decentralized Security • Architecture of Cooperative Storage • Basic Security Mechanism • Problem: Complexity Confuses! • Detail: Reservation Right • Challenges • Authorization in Distributed File Systems • Logistics of Third Party Transfer • Mechanisms for Active Storage • Semantics of Distributed Group Management

41. Practical Lessons • In a system with decentralized security... • Users need debugging tools! • Simple examples: whoami, rwhoami • Client software must become “heavier” • Must carefully parse a vast array of errors. • Must maintain a model of remote devices. • High level names must be used deep within the system software stack. • Run processes with subject name, not Unix UID.

42. For more information... Cooperative Computing Lab http://www.cse.nd.edu/~ccl Cooperative Computing Tools http://www.cctools.org Douglas Thain • dthain@cse.nd.edu • http://www.cse.nd.edu/~dthain