1 / 25

RESTful Web Services for Scientific Computing

Learn how NERSC provides high-performance compute, data, network, and information services through RESTful web services for scientific computing. Discover the NERSC Scientific Gateways and the NERSC Web Toolkit (NEWT) for easy access to backend resources.

amartinez
Download Presentation

RESTful Web Services for Scientific Computing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. RESTful Web Services for Scientific Computing • Joshua Boverhof, LBNL • Shreyas Cholia, NERSC/LBNL • OSCON 2011 • July 28 2011, Portland OR

  2. NERSC • National Energy Research Scientific Computing Center • DOE Office of Science HPC User Facility at Lawrence Berkeley Lab • Provides high performance compute, data, network and information services to scientists across the world

  3. NERSC HPC Clusters

  4. Web Gateways • Old way - SSH + command line + batch system • People now expect web interfaces for everything • Usability - scientific computing should be as easy as online-banking • don’t want generic options/tools not applicable to your science • don’t want to deal with backend, middleware, UNIX CLI etc.

  5. NERSC Scientific Gateways • DeepSky • Astronomical Image Database 11 million images (70TB) • The Gauge Connection • QCD Lattice Gauge Data • CXIDB • X-Ray Image Data Bank • 20th Century Reanalysis • Reanalysis of 20th Century Climate Data • Dayabay • Dayabay Neutrino Detector Gateway • ESG • Earth System Grid Climate Gateway and Data-node

  6. Motives for developing NERSC Web Toolkit (NEWT) • Make it very easy for science teams to build web gateways to their data and computation • We have already built several science specific gateways - want to encapsulate common patterns • Provide Web APIs for access to backend resources for portal and web front-end developers.

  7. NEWT Web Stack • Web Service • Built with Django Web Framework • Exposes NERSC Resources as HTTP URLs • Generally use REST conventions • Access HPC Resources over the web using HTTP + JSON • Frontend Development • javascript Library “newt.js” • AJAX

  8. Things you can do ... • Authenticate using NERSC credentials • Check machine status • Upload and download files • Submit a compute job • Monitor a job • Get user account information • Store app data • Issue UNIX commands

  9. Files Batch Jobs Status CouchDB NIM Authentication MyProxy CA Client: Web Application - HTML 5/AJAX Accounting Information Internal DB: session, cred, user information System Resources (via Globus) Persistent Store (NoSQL DB) NEWT Django Shell Commands Architecture http request JSON data

  10. RESTful Conventions • Resources represented as a set of URLs • HTTP verbs • GET: Idempotent operation, retrieve resource representation • PUT: Idempotent operation, set resource representation • DELETE: Idempotent operation, delete resource • POST: Avoid overloading to use as RPC. Typically use as a factory resource.

  11. NEWT Resources

  12. Login Resource: Authenticate • $.newt_ajax({ • url: ”/auth/", • type: ”POST", • data: {'username':username, 'password':password}, • success: (res, textStatus, jXHR) {} • });

  13. Login Resource • $.newt_ajax({ • url: "/login/", • type: ”GET", • success: function(data){}, • }); • 200 OK • {"username": ”joe", "session_lifetime": 14384, "auth": true}

  14. Queue Resource: PBS job submission • $.newt_ajax({ • url: "/queue/hopper/", • type: "POST", • data: {"jobfile": filename}, • success: function(data){ • $("#output").append(data.jobid); • }, • }); • This is a jQuery JavaScript function that calls the NEWT API. NEWT returns a JSON object that looks like • {"status": "OK", "error": "", "jobid" : "hop1234.id" }

  15. Queue Resource: PBS job submission • $.newt_ajax({ • url: "/queue/franklin/", • type: "POST", • data: {"jobscript”: “#PBS -l mppwidth=8\n mpirun -n 8 /bin/hostname”}, • success: function(data){ • $("#output").append(data.jobid); • }, • }); • This is a jQuery JavaScript function that calls the NEWT API. NEWT returns a JSON object that looks like • {"status": "OK", "error": "", "jobid" : "7259874 " }

  16. Command Resource: Fork job submission • $.newt_ajax({ • url: “/command/franklin", • type: "POST", • data: {”executable": “/bin/date”}, • success: function(data){ • $("#output").append(data.jobid); • }, • }); • This is a jQuery JavaScript function that calls the NEWT API. NEWT returns a JSON object that looks like • {"output": "Wed Jul 20 22:51:58 PDT 2011", "error": ""}

  17. Simple Usage curl • $ curl -k -c cookies.txt -X POST -d "username=boverhof&password=$PASS" https://portal-auth.nersc.gov/newt/auth; • {"username": "boverhof", "session_lifetime": 14397, "auth": true} • $ curl -k –b cookies.txt -X GET https://portal-auth.nersc.gov/newt/status/franklin; • {"status": "up", "system": "franklin"} • $ curl -k –b cookies.txt -d "executable=/bin/date" https://portal-auth.nersc.gov/newt/job/franklin/fork/; • {"status": null, "executable": "/bin/date", "user_id": 18, "url": "https://franklingrid.nersc.gov:60886/81661/1311833735/", "jobmanager": "fork", "submitted": "2011-07-20T06:15:35", "machine": "franklin", "finished": null, "output": null, "id": 47789} • $ curl -k –b cookies.txt -X GET https://portal-auth.nersc.gov/newt/job/jobs/47789; • {"status": "DONE", "executable": "/bin/date", "user_id": 18, "url": "https://franklingrid.nersc.gov:60886/81661/1311833735/", "jobmanager": "fork”, "submitted": "2011-07-20T06:15:35", "machine": "franklin", "finished": "2011-07-20T06:15:36", "output": "Wed Jul 20 23:15:36 PDT 2011\n", "id": 47789}

  18. Django settings: Pluggable Authentication • Authenticate using NERSC credentials to a myproxy-server • Add AuthenticationMiddleware • django.contrib.auth.middleware.AuthenticationMiddleware • Configure authentication backend • AUTHENTICATION_BACKENDS = ( 'newt.authnz.myproxy_backend.MyProxyBackend’ ) • Implement authentication backend • class MyProxyBackend: • def authenticate(self, username=None, password=None): • # Myproxy logon

  19. Django settings: File Upload • File Upload: Upload to portal, store in temporary file, then transfer to remote file system. • Configure file upload handler ( settings.py ) • FILE_UPLOAD_HANDLERS= ( 'newt.file.uploadhandler.RemoteCopyTemporaryFileUploadHandler’ ) • Implement authentication backend • from django.core.files.uploadhandler import TemporaryFileUploadHandler as _TemporaryFileUploadHandler • class RemoteCopyTemporaryFileUploadHandler(_TemporaryFileUploadHandler): • def upload_complete(self): • # Transfer to remote filesystem

  20. Implementation Details ( Hacks ) • Django v1.[1,2,3?] support for HTTP verbs lacking • PUT: Data is not loaded, used code “coerce_put_post” from django-piston • Looking at using Tastypie, a webservice API framework for Django. It provides a convenient, yet powerful and highly customizable, abstraction for creating REST-style interfaces.

  21. NOVA: VASP portal

  22. NOVA: VASP portal

  23. NOVA: VASP portal

  24. NOVA: VASP portal https://newt.nersc.gov https://portal-auth.nersc.gov/nova/

More Related