User Interface Toolkit (UIT)

1. User Interface Toolkit (UIT) ERDC, ITL UIT Workshop April 23, 2008

2. UIT versus ezHPC • The UIT is: • An application programming interface (API) implemented as a SOAP web service. • A library. • Meant for developers. • Not directly usable to the average end user.

3. UIT versus ezHPC (cont.)‏ • ezHPC is: • A UIT client application. • A front-end to accessing much of the UIT's functionality. • A web-enabled application. • Meant for end users.

4. What does the UIT offer developers? • About 60 methods for interacting with the MSRC HPC systems. • A “One Stop Shop” for developers to access the HPC systems. • No Kerberos software is required on the end user's machine. • Off-the-shelf development tools can be used to interact with the UIT. • No significant learning curve for developers familiar with web services.

5. Authentication • UIT eliminates the need for dealing directly with Kerberos libraries. A simple web service call is used for authentication. • This eases the integration of HPC access into desktop applications.

6. System Information • A developer has access to system information such as: • System load • Queue listing • Allocation usage • System news

7. File Management • A developer can use the UIT for file management activities such as: • Deleting, copying, and renaming files. • File permissions and ownership can be modified. • File listings can be requested. • File transfer between HPC systems and with the mass storage facilities.

8. Job Management • A developer can use the UIT for batch job management, such as: • Submitting jobs to the queuing system. • Cancelling jobs. • Checking job status. • Storing and sharing job scripts.

9. Execute Commands • A developer can use the UIT to directly execute arbitrary commands on HPC systems. • Similar to the using the system() or exec() call in C. • Currently limited to the commands that are not interactive.

10. How to Access the UIT • To access the UIT, a developer needs only: • The URL for the UIT web service • The DoD root SSL certificates installed for their development platform • A development platform with support for SOAP web services • Microsoft .NET • Apache Axis (Java)‏ • More developer information and current URL for the UIT web service can be found at https://www.uit.hpc.mil

11. Upcoming Challenges CAC!

12. Questions?

13. Philip Amburn Science Applications International, Corp. UIT with Python UGC 2007

14. Topics • Why FMS is using UIT – Divide and Conquer Application • SOAP with Python • Example Code • Lessons Learned

15. Military Analyst Requirements • Military Analysts • Routinely Use Constructive (Combat) Models • Long established codes, were not written to take advantage of HPC • Evaluate Tactics Or Weapon Systems • Statistically Significant Results • Need: • Reduce The Wall Clock Time It Takes To Conduct An Analysis • More Complex Analyses In Shorter Time • Provide A Scaleable Environment • Constraints: • Don’t Touch My Model! • I Want To Work At My Desk!

16. The Ah-ha Moment • Don’t Change The Model, Change The Way It Is Used • If The Runs Are Independent, Run Them Task Parallel • Migrate From Single Processor Workstation Environment To Multi-Processor HPC • Same Number Of Runs, Shorter Overall Time –or- More Runs in Same Overall Time

17. Historical and Proposed Approach • A largely sequential process that places the burden on the analyst's insight and ability • Limited number of runs available within a given time frame placed a premium on the individual runs • Grew out of the limitations of the computers: Slow, high value resources “Tweak” Problem Definition Parameter Encoding Production Run Production Run Final Analysis Initial Run Analyze • Taking advantage of the parallel HPC resources to run multiple iterations concurrently. • Creates more of a response surface in less time. • Reduces the criticality of “correct” parameters for any given run. • Allows for more “what if” runs. • Individual runs are now expendable “Tweak” Production Run Production Run Production Run Initial Run Problem Definition Parameter Encoding Production Run Final Analysis Initial Run Production Run Analyze Initial Run Production Run Initial Run Production Run Production Run Production Run

18. Divide and Conquer System Architecture browser UIT UIT Web server SAIC Orlando TurboGears framework Analyst UIT UIT UIT Server ERDC MSRC UIT HPC Assets

19. Login Page • Use HPCMO login credentials • SecurID card • UIT provides user authentication

20. Edit Symbol Table • Modifying values for multiple, independent runs of combat model SUPPRESSOR with scenario suppp55 • Variables • FLOAT, INTEGER • LIST, LOOP, SET

21. Submit Batch Job • Submit job to HPC asset • Need HPC account information • Need destination directory on HPC system

22. Python SOAP • Web reference: http://diveintopython.org/toc/index.html • Simple Object Access Protocol (webopedia.com)‏ • XML-based messaging protocol used to encode information • Web service request and response messaging over a network • Chapter 12 is SOAP Web Services

23. Setting up SOAP with Python • PyXML • XML libraries with more functionality than the set that comes with Python distribution • http://pyxml.sourceforge.net/ • Tied to version of Python • Installer program • >>> import xml • >>> xml.__version__ • fpconst • Constants and functions for IEEE Flt Pt • http://www.analytics.washington.edu/statcomp/projects/rzope/fpconst/ • *HARD* to find • Install with “python setup.py install” • Test • >>> import fpconst • >>> fpconst.__version__ • SOAPpy • http://pywebsvcs.sourceforge.net/ • Install with “python setup.py install” • Test • >>> import SOAPpy • >>> SOAPpy.__version__

24. # #------------------------------------------------ # authenticate: get Kerberos credentials # def authenticate( self, name, password, passcode, realm ): credentials = """ <credentials> <username>%s@%s</username> <password>%s</password> <passcode>%s</passcode> </credentials>""" % (name, realm, password, passcode)‏ self.userName = name status = False try: self.authtoken = self.server.authenticateUser(credentials, 'MSRC_KERBEROS')‏ except: status = False print "\n----> failed to get kerberos credentials\n" else: status = True #print "\n----> got credentials\n" Sample Python Wrapper Class UITConnectionFactory: authtoken = "" userName = "" server = "" def __init__ ( self ): #print "initializing the UIT SOAP interface\n" self.server = SOAPpy.SOAPProxy ("https://www.uit.hpc.mil/UITAPI/UITAPI.jws")‏ #self.server.config.dumpSOAPOut = 1 #self.server.config.dumpSOAPIn = 1 # This method intializes a connection factory using authentication information # from a previous successful login (used by the webapp). def previousLogin(self, authtoken, username): self.authtoken = authtoken self.userName = username return status

25. Python Classes for UIT Access • Two Principle Classes – wrappers for UIT API • UITConnection • createDir • deleteDir • listDir • copyFileToHPC • copyFileFromHPC • deleteFileFromHPC • doCommand • submitBatchJob • cancelBatchJob • UITConnectionFactory • getHostList • getConnection

26. Enter user name: amburnep Warning: Problem with getpass. Passwords may be echoed. Enter password: ******** Enter passcode: 423947 Kerberos credentials successful 38|eagle-1.asc.hpc.mil|MSRC_KERBEROS|ASC eagle SGI Altix BX2 39|eagle-2.asc.hpc.mil|MSRC_KERBEROS|ASC eagle SGI Altix BX2 40|eagle-3.asc.hpc.mil|MSRC_KERBEROS|ASC eagle SGI Altix BX2 41|eagle-4.asc.hpc.mil|MSRC_KERBEROS|ASC eagle SGI Altix BX2 20|eagle.asc.hpc.mil|MSRC_KERBEROS|ASC eagle SGI Altix BX2 2|emerald0.erdc.hpc.mil|MSRC_KERBEROS|Description 42|falcon-1.asc.hpc.mil|MSRC_KERBEROS|ASC falcon HP Opteron XC Cluster 43|falcon-2.asc.hpc.mil|MSRC_KERBEROS|ASC falcon HP Opteron XC Cluster 44|falcon-3.asc.hpc.mil|MSRC_KERBEROS|ASC falcon HP Opteron XC Cluster 45|falcon-4.asc.hpc.mil|MSRC_KERBEROS|ASC falcon HP Opteron XC Cluster 18|falcon.asc.hpc.mil|MSRC_KERBEROS|ASC falcon HP Opteron XC Cluster 21|hpc09.asc.hpc.mil|MSRC_KERBEROS|ASC hpc09 Compaq ES45 22|hpc10.asc.hpc.mil|MSRC_KERBEROS|ASC hpc10 Compaq ES45 … Example 1 from UIT import * from getpass import * from sys import * user_name = raw_input("Enter user name: ")‏ # print "user name %s" %user_name password = getpass("Enter password: ")‏ # print "here's the password %s" % password passcode = raw_input("Enter passcode: ")‏ # print "and, here's the passcode %s" % passcode uitcf = UITConnectionFactory()‏ success = uitcf.authenticate( user_name, password, passcode, "ASC.HPC.MIL" )‏ if success: print "Kerberos credentials successful\n" else: print "Error getting Kerberos credentials\n" hosts = uitcf.getHostList()‏ for host in hosts: print " %s" % host

27. creating and deleting directories through eagle drwxrwx--- amburnep amburnep 4096 May 30 20:28 . drwxr-xr-x amburnep amburnep 8192 May 30 20:28 .. -rw-rw---- amburnep amburnep 0 May 30 20:28 touch1 -rw-rw---- amburnep amburnep 0 May 30 20:28 touch2 OK! Now streaming a file to Eagle status of copyFileToHPC True Success copying file ======================stdout total 9416 -rw-rw---- 1 amburnep amburnep 322048 2005-03-14 09:12 050310 -rw-rw---- 1 amburnep amburnep 2465448 2005-06-16 12:45 20000619-maintainer-tools.tar.gz drwxrwx--- 2 amburnep amburnep 4096 2006-01-11 10:36 ACTF drwxrwx--- 4 amburnep amburnep 4096 2006-08-10 11:33 afit_modeling drwx------ 2 amburnep amburnep 4096 2004-03-25 15:12 almond drwxrwx--- 2 amburnep amburnep 4096 2005-05-02 08:24 backgrounds drwxrwx--- 5 amburnep amburnep 4096 2006-08-02 10:52 battlecam drwxrwx--- 3 amburnep amburnep 4096 2007-04-09 16:03 BattleCam -rw-r--r-- 1 amburnep amburnep 126442 2007-05-04 10:34 bc.jpg drwxrwx--- 3 amburnep amburnep 4096 2006-09-15 13:34 bcpp drwxrwxr-x 15 amburnep amburnep 20480 2006-11-17 10:29 bin drwxrwx--- 3 amburnep amburnep 4096 2005-04-27 08:19 classifiers -rw-rw---- 1 amburnep amburnep 135840 2004-08-31 08:38 cliptank.jpg drwxr-xr-x 2 amburnep amburnep 4096 2004-02-25 18:41 cluster drwxrwx--- 7 amburnep amburnep 4096 2006-08-25 11:57 dac drwxrwx--- 4 amburnep amburnep 4096 2007-05-11 16:03 dacrun drwxrwx--- 6 amburnep amburnep 4096 2007-05-10 15:49 dac_runs drwxr-xr-x 10 amburnep amburnep 4096 2004-09-24 10:59 decimate -rw-rw---- 1 amburnep amburnep 489593 2005-07-07 15:02 defs.zip drwxr-xr-x 4 amburnep amburnep 4096 2007-05-21 16:53 Desktop … Example 2 # establish a connection to hpc11 at ASC MSRC hpc11 = uitcf.getConnection( "hpc11" )‏ #establish a connection to eagle at ASC MSRC eagle = uitcf.getConnection( "eagle" )‏ print "creating and deleting directories through eagle\n" eagle.createDir( "/hafs13/amburnep/junkDir2" )‏ eagle.doCommand( "touch /hafs13/amburnep/junkDir2/touch1" )‏ eagle.doCommand( "touch /hafs13/amburnep/junkDir2/touch2" )‏ elements = eagle.listDir( "/hafs13/amburnep/junkDir2" )‏ for element in elements: print element eagle.deleteDir( "/hafs13/amburnep/junkDir2" )‏ print "OK! Now streaming a file to Eagle\n" status = eagle.copyFileToHPC( "c:/temp/irma_um.pdf", "/hafs13/amburnep/irma_um.pdf" )‏ print "status of copyFileToHPC ", status, "\n" if status == False: print "Error copying file to HPC machine\n" else: print "Success copying file " # results = eagle.doCommand( "ls -l" )‏ print "======================stdout\n" print results[0] print "=====================stderr\n" print results[1]

28. Lessons Learned • Why Use the User Interface Toolkit? • Security Approved by HPCMPO • SOAP accessible • We created Python wrappers for subset of UIT API • Provides easy access to HPCMO resources • Local network policies can be an issue • Can’t reach UIT server using 88th CG network • Utility of Web-based Application • Small, lightweight application on top of TurboGears framework • Web hosting • Database • Python wrappers made UIT easy to integrate