Administering PennGroups

1. Administering PennGroups Chris Hyzer ISC/ASTT August 1, 2011 ISC

2. Administering PennGroups • Install • DDL • API • UI • WS • Source control • Upgrade • Maintenance • Roadmap ISC

3. Demo server • Internet2 has a Grouper Demo Server • Address is: https://grouperdemo.internet2.edu/ • Host various versions of Grouper • Show features (e.g. permissions, external users, syncing between groupers) • Allow users or potential users to kick the tires (not for production obviously)

4. Demo server for this training • 9 accounts setup • /home/test1, /home/test2, etc • Each participant gets a temporary account and pass (ask Chris) • The pass will work for SSH, Apache (UI, WS), mysql • Each account has a mysql database, and user with access to that database

5. Demo server training tomcats • Except for development, you should only have one application per Tomcat • One application wont take all memory and kill tomcat • You can restart after deploys without affecting other apps • Note: for development, you can use either basic authentication • Or link cosign/shib tomcat on test cluster to dev tomcat

6. Demo server training tomcats (continued) • Install dir:[test6@i2midev1 tomcat]$ pwd/home/test6/tomcat[test6@i2midev1 tomcat]$ lsconf logs temp webapps work • TOMCAT_HOME, CATALINA_HOME elsewhere[test6@i2midev1 tomcat]$ more \ /etc/init.d/tomcat_test6…export TOMCAT_HOME="/opt/tomcat6base" export CATALINA_HOME="/opt/tomcat6base"

7. Demo server training tomcats (continued) • Note you have your own Java symlink (for ps)[test6@i2midev1 tomcat]$ more /etc/init.d/tomcat_test6…export JAVA_HOME="/opt/javas/java_test6" • Start/stop tomcat, lets run these commands (for your user) [test6@i2midev1 ~]$ tomcat start [test6@i2midev1 ~]$ ps -ef | grep java_test6 [test6@i2midev1 ~]$ netstat -pan | grep 5231(note the port numbers) [test6@i2midev1 ~]$ jstack 5231 | less [test6@i2midev1 ~]$ tomcat stop [test6@i2midev1 ~]$ ps -ef | grep java_test6

8. Demo server training tomcats (continued) • To set this up: $ sudouseradd -g i2mi test1 $ sudopasswd test1 • create user in DB, database assign permissions with sqlyog mysql> create database grouper; mysql> create user 'grouper'@'localhost' identified by 'somesecret'; mysql> grant all on grouper.* to 'grouper'@'localhost'; $ sudohtpasswd /etc/httpd/conf.d/users.pass test1 $ sudohtpasswd /etc/httpd/conf.d/users.pass test1a $ cp -R /opt/tomcats/tomcat6bullet /home/test6/tomcat ports start at 9010, edit the ~/tomcat/conf/server.xml, set the three ports: http: 90x0, jk: 90x1, shutdown: 90x2 [root@i2midev1 init.d]# cp tomcat_k tomcat_test1 [root@i2midev1 init.d]# chkconfig --add tomcat_test1 [appadmin@i2midev1 javas]$ ln -s ../java6 java_test1 [test1@i2midev1 test1]$ mkdir bin [test1@i2midev1 bin]$ ln -s /etc/init.d/tomcat_test1 tomcat

9. Demo server training tomcats (continued) • Put this in /etc/profile JAVA_HOME=/opt/java6 export JAVA_HOME ANT_HOME=/opt/ant export ANT_HOME #note: maven3 is needed for grouper 2.0 M2_HOME=/opt/maven export M2_HOME M2=$M2_HOME/bin export M2 PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$M2:$PATH:$HOME/bin export PATH

10. Demo server training tomcats (continued) • Put this in /var/www/html <b>test1</b> <ul> <li><a href="test1_grouper/">Grouper UI</a></li> <li><a href="test1_grouperWs/">Grouper WS</a></li> </ul> • Put this in /etc/httpd/conf.d/proxy_ajp.conf ProxyPass /test1_grouper/ ajp://localhost:9011/test1_grouper/ ProxyPass /test1_grouperWs/ ajp://localhost:9011/test1_grouperWs/ • Bounce apache [root@i2midev1 init.d]# /sbin/service httpdconfigtest Syntax OK [root@i2midev1 init.d]# /sbin/service httpdgraceful [root@i2midev1 init.d]#

11. Install Grouper API (First step in training) • Download the API (1.6.3, latest stable) $ mkdir 1.6.3 $ cd 1.6.3 $ pwd /home/test1/1.6.3 $ wget http://www.internet2.edu/grouper/release/1.6.3/grouper.apiBinary-1.6.3.tar.gz $ tar xzvf grouper.apiBinary-1.6.3.tar.gz $ cd grouper.apiBinary-1.6.3/conf/ $ emacs grouper.properties

12. Install Grouper API (continued) • Note, Powerpoint might mess-up chars (e.g. dashes)… might need to type them in if there is a problem, instead of copy/paste  • Note, a backslash: \ means the next line is the same line  • Note, all commands are in the "notes" section of the presentation, copy/paste from there

13. Emacs cheatsheet Note: Feel free to use your favorite editor obviously $ emacssomefile.whatever Note: might need to do CTRL-backspace instead of backspace Save: CTRL-x CTRL-s Exit: CTRL-x CTRL-c Find: CTRL-s Find (wrap): CTRL-s, put in criteria, CTRL-s Find backwards: CTRL-r Stop command: CTRL-g Replace: ESC-x query-replace (then “y” to replace each) Cut rest of line: CTRL-k Paste: CTRL-y Put in background: CTRL-z BASH get back: fg

14. Configure grouper.properties $ emacs grouper.properties groups.wheel.use = true configuration.autocreate.system.groups = true configuration.autocreate.group.name.0 = etc:webServiceClientUsers configuration.autocreate.group.description.0 = users allowed in WSconfiguration.autocreate.group.subjects.0 = GrouperSystem,testX,testXaconfiguration.autocreate.group.name.1 = etc:sysadmingroup configuration.autocreate.group.description.1 = sys admin users configuration.autocreate.group.subjects.1 = testX grouperIncludeExclude.use = true grouperIncludeExclude.requireGroups.use = true

15. Configure grouper.hibernate.properties $ emacs grouper.hibernate.properties hibernate.dialect = org.hibernate.dialect.MySQL5Dialect hibernate.connection.driver_class = com.mysql.jdbc.Driver hibernate.connection.url = jdbc:mysql://localhost:3306/testX hibernate.connection.username = testXhibernate.connection.password = *******************

16. Configure grouper-loader.properties $ emacs grouper-loader.properties loader.autoadd.typesAttributes = true

17. Init DB and startup GSH $ mysql -utest1 -p test1 mysql> show tables; Empty set (0.00 sec) mysql> exit Bye $ cd /home/testX/1.6.3/grouper.apiBinary-1.6.3/bin/ $ ./gsh.sh -registry <Shows menu> $ ./gsh.sh -registry -check -runscript $ ./gsh.sh -registry -check $ mysql -utestX -p testX mysql> show tables; 76 rows in set (0.00 sec) mysql> exit Bye

18. Quickstart and subjects $ wget -O quickstart.xml http://anonsvn.internet2.edu/cgi-\ bin/viewvc.cgi/i2mi/tags/GROUPER_1_6_3/grouper-qs-\ builder/quickstart.xml?view=co $ wget -O subjects.sql http://anonsvn.internet2.edu/cgi-\ bin/viewvc.cgi/i2mi/tags/GROUPER_1_6_3/grouper-qs-\ builder/subjects.sql?view=co $ ./gsh.sh -registry -runsqlfile subjects.sql $ ./gsh.sh -xmlimportold GrouperSystem quickstart.xml

19. MySQL GUI Optional (if you have SQLYog free gui, or whatever) Port forward over SSH, remote port 3306, local port whatever (3302?) Connect to localhost 3302 In either case, lets open a GSH window, a mysql window, and a linux window

20. Add your users $ ./gsh.sh (note: testX was automatically inserted and into wheel and WS) gsh 0% grouperSession = GrouperSession.startRootSession(); gsh 1% addSubject("testX", "person", "John Smith"); gsh 2% addSubject("testXa", "person", "Johna Smitha"); gsh 3% addMember("etc:webServiceClientUsers", "testXa"); gsh 4% exit Review other GSH commands Note: GrouperSession is a ThreadLocal ActAs for the API

21. Add your users (continued) $ mysql -utestX -p testX INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\ VALUES ('testX', 'loginid', 'testX', 'testX'); INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\ VALUES ('testX', 'name', 'John Smith', 'john smith'); INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\ VALUES ('testX', 'description', 'John Smith', 'john smith'); INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\ VALUES ('testXa', 'loginid', 'testXa', 'testXa'); INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\ VALUES ('testXa', 'name', 'Johna Smitha', 'johna smitha'); INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\ VALUES ('testXa', 'description', 'Johna Smitha', 'johna smitha'); COMMIT; exit;

22. Get the UI $ cd ~/1.6.3 $ wget http://www.internet2.edu/grouper/release/1.6.3/grouper.ui-\ 1.6.3.tar.gz $ tar xzvf grouper.ui-1.6.3.tar.gz $ cd grouper.ui-1.6.3/ $ cp build.properties.template build.properties $ emacs build.properties grouper.folder=../grouper.apiBinary-1.6.3 should.copy.context.xml.to.metainf=false webapp.name=testX_grouper default.webapp.folder=/home/testX/tomcat/webapps/${webapp.name}

23. Get the UI (continued) $ emacs webapp/WEB-INF/web.core.xml From the bottom, remove security-contraint,login-config,security-role Note, if you are running on tomcat locally, leave that in, and edit tomcat-users.xml $ ant default $ tomcat restart Go to: https://grouperdemo.internet2.edu and click on your UI: https://grouperdemo.internet2.edu/testX_grouper/

24. UI authentication In this case, I added this to the httpd.conf or include: <LocationMatch ^/test.*> AuthType Basic AuthName "By Invitation Only" AuthUserFile /etc/httpd/conf.d/users.pass Require valid-user </LocationMatch> Bounce apache: [root@i2midev1 conf.d]# /sbin/service httpdconfigtest Syntax OK [root@i2midev1 conf.d]# /sbin/service httpd graceful Authentication is pluggable, can write a servlet filter to put in remote user Works with Shib, CAS, Cosign, Basic (web server), Basic (app server), any web server plugin, out of the box Note: if UI is internal users, but using external users via invites, might map multiple URLs to multiple authn schemes (like Penn or demo server)

25. Create objects Act as admin in upper right. Create a root folder named "test". (system and friendly) Inside that folder (aka stem), create a group called "testGroup", but uncheck allow all to "read" and "view". (system and friendly name is testGroup) Add some members to testGroup: babl, babr, babu

26. Get the WS $ cd ~/1.6.3 $ wget http://www.internet2.edu/grouper/release/1.6.3/grouper.ws-\ 1.6.3.tar.gz $ tar xzvf grouper.ws-1.6.3.tar.gz $ cd grouper.ws-1.6.3/grouper-ws $ emacs build.properties grouper.dir=../../grouper.apiBinary-1.6.3 webapp.name=testX_grouperWs $ emacs conf/grouper-ws.properties ws.client.user.group.name = etc:webServiceClientUsers $ emacs webapp/WEB-INF/web.xml From the bottom, remove security-contraint,login-config,security-role Note, if you are running on tomcat locally, leave that in, and edit tomcat-users.xml

27. Get the WS (continued) $ ant dist $ cp -R build/dist/testX_grouperWs ~/tomcat/webapps $ tomcat restart Go to: https://grouperdemo.internet2.edu and click on your WS: https://grouperdemo.internet2.edu/testX_grouperWs/servicesRest Should get error  though there is a URL to get members… 

28. WS authentication In this case, it’s the same as UI: <LocationMatch ^/test.*> AuthType Basic AuthName "By Invitation Only" AuthUserFile /etc/httpd/conf.d/users.pass Require valid-user </LocationMatch> Bounce apache: [root@i2midev1 conf.d]# /sbin/service httpd configtest Syntax OK [root@i2midev1 conf.d]# /sbin/service httpd graceful Authentication is pluggable, or can write a servlet filter to put in remote user Works with Kerberos, SOAP WS-security, Basic (web server), Basic (app server), any web server plugin, out of the box

29. Loader example Lets make a table with subjectIds in itmysql> CREATE OR REPLACE VIEW loader_employee AS \ (SELECT subjectId AS subject_id FROM testX.subject\ WHERE subjectId LIKE 'b%'); Make a folder for community under root Add a group called employee under folder "community" Edit the employee group, select type "grouperLoader" (make sure acting as admin in upper right of UI)

30. Loader example (continued) Edit attributes on the group: grouperLoaderDbName: grouper NOTE: configure other DB connections in grouper-loader.properties NOTE: every minute just for testing… grouperLoaderQuartzCron: 0 * * * * ? grouperLoaderQuery: select subject_idsubject_id from \ loader_employee grouperLoaderScheduleType: CRON grouperLoaderType: SQL_SIMPLE

31. Loader example (continued) Run manually just to see it work: gsh 0% grouperSession = GrouperSession.startRootSession(); gsh 1% loaderGroup = GroupFinder.findByName(grouperSession, \ "community:employee"); gsh 2% loaderRunOneJob(loaderGroup); loader ran successfully, inserted 12 memberships, deleted 0 memberships, total membership count: 12 gsh 3%

32. Loader example (continued) Run continuously: % ./gsh.sh -loader Change the view: mysql> CREATE OR REPLACE VIEW loader_employee AS \ (SELECT subjectId AS subject_id FROM SUBJECT \ WHERE subjectId LIKE 'b%o' or subjectId like 'el%'); At the top of the minute, check the memberships, and there should be some deleted, and some added

33. Loader details at Penn • In the fastGrouperProdDaemon web application, we run the loader jobs in a FAST daemon • All loader jobs are based on views, to ease maintenance • Then we don’t have a command line application to monitor etc. • The grouper_loader_log table has an entry for every daemon run • Generally the only problem we have is Warehouse jobs for people without PennId’s, the people can’t be found, and it’s an error • The warehouse will assign a fake pennId which starts with 0 • Change the loader view to have where clause PENN_ID not like '0%'

34. Loader include/exclude example Create a group Read/update should not be granted to everyone Use addIncludeExclude type Look in folder, there will be 5 groups created with that type. Open the system of record, and lets make that the loader group Create this view in the DB: mysql> CREATE OR REPLACE VIEW loader_student AS \ (SELECT subjectId AS subject_id FROM SUBJECT WHERE \ subjectId LIKE 'fi%');

35. Loader include/exclude example (continued) Never edit the loader group, unless you expect it to get overwritten Add fico to the excludes group Add bapo to the includes group Look at the overall group Generally the privileges are: Assign READ on all to admins Assign UPDATE on include/exclude groups to admins Assign READ to service principal of app for overall group or other people who need to use the group

36. Customize UI text % cd ~/tomcat/webapps/testX_grouper/WEB-INF/classes/resources % mkdir custom % cd custom % touch media.properties % emacs nav.properties find.browse.here=Current folder is: % emacs init.properties default.module=grouper default.locale=en_US % cd ~/tomcat/webapps/testX_grouper/WEB-INF/classes/resources % emacs init.properties default.module=custom default.locale=en_US Bounce tomcat: % tomcat restart

37. Customize UI text (continued) Should see:

38. Customize lite UI for an application % cd ~/tomcat/webapps/testX_grouper/WEB-INF/classes % mkdir membershipLiteName % cd membershipLiteName % touch testName.properties % cd ~/tomcat/webapps/testX_grouper/WEB-\ INF/classes/resources % emacs custom/nav.properties Add line: membershipLiteName.testName.simpleMembershipUpdate.updateTitle = PTO\ admins Bounce tomcat: % tomcat restart https://grouperdemo.internet2.edu/testX_grouper/grouper\ Ui/appHtml/grouper.html?operation=SimpleMembership\ Update.init&groupName=apps:pto:ptoAdmins_systemOfRecord\ &membershipLiteName=testName

39. Customize lite UI for an application (continued)

40. Get the Grouper Client $ cd ~/1.6.3 $ wget \ http://www.internet2.edu/grouper/release/1.6.3/ \ grouper.clientBinary-1.6.3.tar.gz $ tar xzvf grouper.clientBinary-1.6.3.tar.gz $ cd grouper.clientBinary-1.6.3 $ emacs grouper.client.properties grouperClient.webService.url = \ https://grouperdemo.internet2.edu/testX_grouperWs/servicesRest grouperClient.webService.login = testX grouperClient.webService.password = **************

41. Get the Grouper Client (continued) Get usage: $ java -jar grouperClient.jar $ java -jar grouperClient.jar --operation=getMembersWs \ --groupNames=test:testGroup Customize the output: $ java -jar grouperClient.jar --operation=getMembersWs \ --groupNames=test:testGroup \ --outputTemplate='${wsSubject.id}$newline$'

42. Try from your local machine (win, mac, etc) NOTE: you need java6+ > cd c:\temp (or translate for mac or whatever) Download and unzip: http://www.internet2.edu/grouper/release/1.6.3/grouper.clientBinary-1.6.3.tar.gz > cd grouper.clientBinary-1.6.3 Edit grouper.client.properties grouperClient.webService.url = \ https://grouperdemo.internet2.edu/testX_grouperWs/servicesRest grouperClient.webService.login = testX grouperClient.webService.password = ************** Get usage: > java -jar grouperClient.jar > java -jar grouperClient.jar --operation=getMembersWs \ --groupNames=test:testGroup

43. Grouper deployment control at Penn • See document • The Grouper team hopes to have a maven version of this some time soon • Everything in Penn’s CVS, external encrypted passwords, separated out filters, customizations • Localdev, Dev, Test, Prod managed for custom application, WS, UI, client • Checkout pennGrouper project from CVS and look at customizations

44. Grouper upgrades at Penn • See document of 1.6 to 1.7 upgrade • I believe we skipped 1.5, though we generally upgrade to new versions when available since we need features • Generally we want everything to be up as much as possible in readonly mode • Switch to readonly mode for WS/UI, stop updates on LDAP • Get counts of important groups for sanity test • Upgrade the DB (generally most time consuming part, if have to edit millions of rows) • Upgrade the WS/UI, test everything • Confirm counts of important groups

45. Grouper maintenance at Penn • Process forms from DA, add kerberos principals • Add new loader jobs • Add new folders and delegate to new clients • Consult on design of how applications use Grouper • Look at errors emailed from Grouper log4j • Run GSH scripts

46. Grouper sample GSH script • See this documentation • E.g. will need to do this to delete old course groups

47. Daily report • Email sent everyday to give a pulse of Grouper • Need to deal with unresolvable subjects

48. Monitoring • grouperWs has a status servlet hooked up to nagios • /grouperWs/status?diagnosticType=all

49. Roadmap • Hopefully uses for central permissions • E.g. warehouse permissions • E.g. PennCommunity Direct permissions • Always available read-only web services • Shibboleth entitlement group membership integration • PennCommunity Direct getPerson WS secure attributes

50. Questions? ISC