Using Shibboleth as an SSO

1. Using Shibboleth as an SSO University at Buffalo Joel Murphy jmurphy@buffalo.edu

2. overview • Id mgmt @ UB • UB deployment Timeline • Deployment considerations • Certificate management • LDAP • Load Balancer • Web Farm and Service Providers • Webmail proposed configuration • Cosign • Shibboleth Identity Provider • Load Testing • Challenges

3. Id Mgmt @ UB • “UBIT Name” branding • All University affiliated persons (faculty, staff, students, etc.) • single username/password, many directories, many applications • usernames and passwords and groups synced across directories • One user namespace for campus • People retain same username after returning to Univ • Unix groups primary mechanism for authorization (authz) and entitlements • Applications responsible for authorization

4. Id Mgmt @ UB • “Accounts Management” oracle database and application core to Id Mgmt • Home grown Perl, C, oracle SQL • Downstream Directories • DCE – authentication, authorization, dir svcs • LDAP – dir svcs, authn via Kerberos, authz • Kerberos – authn, enables encrypted authenticated communication channels • Active Directory • X.509 Certificates

5. Password management • password replication across directories

6. Long road to SSO • Pre-1997 – Unix passwords/NIS – just Unix services • 1997 - DCE – first success at consolidation of Id mgmt and password database, multiple platforms • 1998 – DCE/DFS deployed as enterprise filesystem • 2003 – Kerberos/Active Directory/LDAP/Shibboleth/Cosign/Cisco CSS • Summer 2005 – production Oracle RAC database • Fall 2005 – First production Shibboleth Services – Course Management, Download service • Fall 2005 - LDAP Campus Portal and administrative apps (was DCE) • Summer 2006 – Network Appliance deployment for DFS replacement for enterprise filesystem (CIFS/NFS) • Summer 2006 – Business Continuity site deployment and machine move • Fall 2006 – Shibbolize authn portions of Campus web server • Fall 2006 – Shibbolize Campus Portal and apps (from LDAP) • Fall 2006 or Spring 2007 – Shibbolize Web Mail (IMP or @mail)

7. Id Mgmt/SSO Outliers • Enterprise Active Directory “exception” accounts (eg. OU administrators) – not in Kerberos/LDAP • Science and Engineering NIS passwd (migrating to enterprise Kerberos for auth) • Various Active Directory Forests outside Enterprise AD • Various applications with limited technical resources or need for Enterprise integration

8. Id Mgmt challenges • Authorization not always done or works by “accident” (user not in /etc/passwd) • Application owners don’t always know who should be authorized – typically request a group • DCE still core for to IdM • Authoritative for groups and data for deactivated users • RPCs use heavily, replacing with SOAP/SSL • CIO level requests • Better authorization for network access • Online password reset • Requests for LDAP access for outsourced apps • Hard to get resources for not-so-low hanging fruit

9. Some Shibboleth/SSO and Web Farm Goals/Considerations • Survive network split (multiple campuses) • Don’t preclude logoff • Ability Load balance commodity hardware • Detect and Isolate failures - Transparent failover of Identity Providers and Service Providers (multiple machines, same configuration) • Capacity/Responsiveness – Shibboleth is Enterprise SSO • Commitments made to sell Shibboleth solution

10. How we got there • Internal certificate signing service • LDAP environment for our attribute data (eduPerson, UBEduPerson) • Cisco CSS (Content Switch) load balancer • Multi-purpose Oracle RAC database (“Real Application cluster” - active/active) • Hitachi SAN • Cosign SSO • Shibboleth • Lots of glue • Lots of help and contributions from UB and I2 community

11. Did we make it? • Survive network split (multiple campuses) • No – multiple load balancers, but can’t migrate IPs across subnets, opted against DNS delegation • Business continuity site not on one of the campuses (no people) • Don’t preclude logoff • Supported in Cosign, limited Shib 2.0 support? • Detect and Isolate failures - Ability Load balance commodity hardware • Large farm of Sun 280R Solaris servers for admin applications • Farm of 8 Dell 1750 Linux servers for shibboleth/cosign • 4 Sun 280R Solaris servers for LDAP • 4 Sun V120 Solaris servers for Kerberos • Transparent failover of Identity Providers and Service Providers • Yes, except for failure mid-transaction • Capacity/Responsiveness – Shibboleth is Enterprise SSO • Yes, our auth systems are very under capacity • Adding 2 additional LDAP servers (new/bad applications with unpredictable performance profiles)

13. Physical Layout

14. Shibboleth IDP Setup • InCommon member • Apache 1.3/Tomcat connector/Tomcat • Handle Service authenticated by Cosign, trusted by Shib • Still at version 1.2 IDP (origin) • Hindsight: put IDP source tree in CVS • Web Farm setup requires Shib Handle configured as “CryptoHandleGenerator” instead of shared memory • All Attributes currently resolved via LDAP (from Person objects) • Non-web apps may have similar needs • Entitlements are currently mapped from group memberships and stored in LDAP • Once configured, IDP changes very infrequently • Attribute release is negotiated when configuring new targets • Meta-data for Federations (sites and trusts) changes periodically • We continue to deploy all configuration in one “war” file. • Log files are somewhat large and unwieldy • No deterministic policies, data custodians authorize attribute release

15. Application Web Farm/Shibboleth SP Setup • Apache 1.3 • Caching apache proxy in front of application apache • Shibboleth integrated into application apache • Apache proxy/cache runs SSL • Machines load balanced with Cisco CSS • Currently assume UB IdP for applications • Standard attribute release and acceptance • Multiple services (virtual hosts) on one machine • Haven’t needed to adjust ARP for Virtual Hosts • One certificate and ARP shared by all web farm machines • Modified MySQL cache to be Shibboleth Oracle shar cache • Sticky connections on CSS makes unnecessary • Was one source of early stability problems

16. Application Web Farm/Shibboleth SP Issues • Canonical Service names • Forcing Canonical host names (UseCanonicalName) makes IDP configuration of AssertionConsumerServiceURL simple. • Prevents dreaded error message from Handle Service (unknown ACS URL…) • Makes it very difficult for developers or sysadmins to poke a specific machine (use hosts file) • Alternative – list every possible hostname/short name in meta-data? • Web cache • Cookie based authentication doesn’t prevent caching of data where HTTP BasicAuth does automatically • Need to set header in response of authenticated pages: “Cache-Control: private" • Web proxy (URL rewrite) • We needed to set checkAddress=“false” in shibboleth XML • ShibURLScheme, ServerName and Port needed for apache • Shibboleth needs to know how to write a “return address” for redirects • Multiple virtual hosts, odd port mapping schemes allow us to run proxy and server on same hosts

18. Shibboleth SP Issues • Provide a “UB” shibboleth distribution and instructions • Build out of date, XML config, metadata and instructions mostly valid • Where to put certs • What startup files are needed • What should be monitored • What settings to tweak in a web farm environment • We generally recommend rebuilding apache module for local apache • Unnecessary? • New versions of shib from I2 come as packages and are significantly simpler than the past

19. Blackboard/Course Management • Shibboleth just used for SSO (authentication) • Nightly data processing feeds Blackboard system data from institutional systems • Creates users object in Blackboard • Manages course registrations

20. Download Service • Shibboleth used as SSO • Group membership used for authorization • Authorization based on licensing • Microsoft software download

21. Campus Portal • Shibboleth to be used as SSO • Current phase is to transition from LDAP auth (was DCE) • Nightly processing generates data about people authorized portal in portal application

22. Webmail Proposal • Cyrus IMAP email • Would like to link Campus Portal/Webmail/Course mangement • Multiple possibilities • Integrate with cosign directly and use Kerberized IMAP session • Shibbolize webmail and use secure fabricated password for auth • Trusted webmail • Pass Kerberos ticket via Shibboleth Attribute??? • Upcoming Shibboleth solution for n-tier apps??? • Concerns/Issues • Desire to abstract out Cosign • Fabricated password must not be reproducible or usable via normal IMAP connections • Fabricated password requires maintaining a password file • Kerberizing IMAP possibly more Webmail customization than tinkering with password

23. Webmail Proposal • Current working proposal is to use shib with fabricated passwords

24. Shibboleth Authorization • Our group management is inflexible (DCE still authoritative) • Working to remove DCE, would like Grouper (groups of groups?) • Business rules don’t always match single group membership • Needed to match sets of groups • Require membership in active and some staff group: ShibRequireAll on require memberof ~ ^…staff$ require memberof ~ ^active$ • Require membership in active or some staff group: ShibRequireAll off require memberof ~ ^…staff$ require memberof ~ ^active$ • Shib1.3b adds XML access control, nested requirements • https://authdev.it.ohio-state.edu/twiki/bin/view/Shibboleth/XMLAccessControl • Using mod_setenvif/mod_access does not work with Shibboleth environment variables • Shibboleth module doesn’t have a chance to generate env variables before mod_access and order is Apache hard coded, not dependant on module order

25. Cosign Setup • SSO just for Shibboleth Handle Service • Cosign supports logoff (hidden) • Cosign can pass Kerberos tickets and do certificate based auth • SSO’s at the time of our deployment did not support logon server redundancy. • Cosign writes state information to disk, extended this to write state to Oracle. • Oracle transactions through an external server (cookie_server)

26. Cosign Issues • Custom code makes upgrading difficult • Backported fixes from newer cosigns • Bugs in our oracle interface • Oracle RAC configured Active/Active, but updates not synchronous • Needed max_commit_propagation_delay=0 on Oracle RAC instances • Newer cosign versions have async replication of state • Could loose some state in logon server failure • Move state files from Oracle RAC to HA Netapp NFS share?

27. Cosign diagram

28. LDAP Environment • 4 load balanced Sun ONE 5.2 Directory Servers • One giant directory • ou=People, dc=buffalo, dc=edu • posixAccount • Person • OrganizationalPerson • InetOrgPerson • eduPerson • UBEduPerson • ou=Groups, dc=buffalo, dc=edu • posixGroup • groupOfUniqueNames • Added institutional IDs and UBEPmemberOf to UBEduPerson • memberOf on Person makes shib Attribute release simple and fast • Future deployments should use eduMember schema • http://middleware.internet2.edu/dir/docs/internet2-mace-dir-ldap-group-membership-200507.html

29. LDAP Environment (cont) • Hourly processing updates LDAP with changes from Id mgmt database (Perl Net::LDAP) • Oracle layout • One row per principal • One column per attribute, multi-valued attributes glued with a separator • Goal to co-locate non-institutional data in directory • Available via LDAP or via Shibboleth • Allow administrators to manage source oracle table or write changes to LDAP • All Access management policy at the root of the directory granting to groups, some based on attribute value (FERPA) • Avoids confusion by avoiding object acls or deny acls • Multi-master replication – supports application writes with load balancing • No passwords – simple binds proxied to Kerberos with a modified version of preAuth plugin by Mike Gettes • Forces simple binds over SSL

30. LDAP Issues • Large groups don’t perform well, use memberOf • Can’t “browse” the directory – administrative limits • Last modified time • DS 5.1 didn’t support multi-master replication • Introduced memory leak in Kerberos pre-auth plugin when porting to DS 5.2 • Replication of schemas created redundant definitions (fixed?) • Replication failures with large directory and lots of changes • Fixes for bugs 6242420 and 6283717 included in DS 5.2 patch 4 • Turned multi-master back on to support LDAP applications needing to write data

31. Internal Certificate Management • Manages certs for test servers, shibboleth SP’s and VPN access • Openssl on Id mgmt system • requests via email • Daily job for monitoring expirations • No audit trail and little metadata associated with certs • Not particularly safe for multiple administrators • Tightening access/clarifying procedures • Plan on moving to OpenCA

32. Load Balancer Setup • Began with Cisco Local Directors for email system load balancing (SMTP/IMAP/POP) • Current - Two Cisco CSS 11503, active/standby • Load balanced services on private network behind load balancers, using normal UB address space • Servers see IP address of Clients • NAT, health monitoring, sticky connections, load balancing • Standard architecture for all UB load balanced services (LDAP, Shibboleth, Cosign, web farm, etc.) • Ability to directly connect to a specific machine in a pool

33. Load Balancer Issues • Services see the IP of client connecting • Can easily roll-in/deploy and test changes with no downtime • Network management of private net • Everything on one subnet • Allows migrating IPs between CSS • Extended subnet to Bus. Cont. site, but not other campus sites • Danger of private net connecting to public • Load Balanced services talking to other load balanced services • Must NAT both source and destination host • Sysadmin Management • Too many services, anyone can take everything down • Shared serial console (HTTP disabled for security/auditing concerns) • Many ports on many machines • Secure web tool in development • Monitoring services, removing from pool • CSS can monitor HTTP results for testing server health • Notification for down servers (hooked into Big Brother)

34. Load Balancer example

35. Troubleshooting • Check our Big Brother console for alerts • Check Load balancer – what’s active? • Application Service provider • Check access_log/error_log • Check Shibboleth shar.log and shire.log • Is shar (shibd) running? • Can we connect to Shib AA with openssl s_client?

36. Troubleshooting (cont) • Identity Provider • Are apache/cosignd running and accepting connections? • Is cookie_server running? • Check all apache logs • Check tomcat bootstrap log – did shib start? • Check shib application logs

37. Load Testing • Webload (Radview) • JMeter • Perl LWP • SAR (Linux/Solaris) • Infrastructure survives intense load testing • Shibboleth initial logon “costs” lessen with longer sessions (perhaps less in some cases?)

38. Challenges • Transitioning into production • Project team not all of support team • Too much change at one time • Service maturity takes time • Lots of layers • Lots of machines and services • Troubleshooting complex • Errors don’t go travel upstream well • Multiple support teams • Help Desk support complex, especially for Fed apps