Introduction to NMRbox Project & Platform

1. Introduction to NMRbox Project & Platform Mark Maciejewski UConn Health “Think inside the box”

2. Lecture Motivation for the project NMRbox platform Benefits to users and developers Usage Hands on Account management Connect to NMRbox VM Set the display resolution Software inventory File transfers Outline

3. Motivation: Abundance of software Hundreds of packages cited in BioMagResBank depositions, J. Bio-NMR, and other journals.

4. Motivation: Fragmentation Operating systems Programming languages Libraries BLAS

5. Motivation: Challenges to persistency Platforms become obsolete Developers graduate Software time bombs Grants end

6. Motivation: Meta-software packages Sparky NMRPipe Python scripts Rosetta SHIFTX2 MODELLER Experimental protein structure verification by scoring with a single, unassigned NMR spectrum. Courtney, Rienstra, et al., Structure, 2015.

7. Motivation: Computational reproducibility A computational study is reproducible when it provides the “complete software environment needed to reproduce the figures” - D. Donoho, Stanford Obstacles Missing primary empirical data Missing meta-data Missing software (scripts, programs) Non-persistence of software Manual interventions

8. Challenges Abundance of software (discovery) Fragmentation of OS’s, programming languages, libraries Persistency of software Complexity of design and installation Reproducibility of results QuestionHow do we address these challenges? AnswerNMRbox VM

9. NMRbox Project Deliverables – primary tools Platform • NMRbox VM: A virtual machine pre-configured with a wide range of software used in biological NMR • Significant computational resources Data • BMRB integration & richer depositions • Metadata management and workflow annotation Analytics • Bayesian tools to enhance data analysis and interpretation • API for developers to incorporate Bayesian inference

10. Deliverables – community services Training and Dissemination • Workshops, tutorials, and guides • User and developer support Driving Biological Projects (DBPs) • Test beds for NMRbox technology development • What limits your progress? Collaboration and Service (C&S) • Apply technologies to challenging biomedical research problems

11. NMRbox VM • Acquisition • Agnostic – Install all software available • Persistent • Archive all versions • Access • Platform-as-a-Service (PaaS) • Standalone / Downloadable VM • Guest OS • Xubuntu 16.04 LTS

12. NMRbox VM Content – Software packages 100+ packages installed (see https://nmrbox.org) • Spectral reconstruction • Spectral visualization • Automated assignment • Structure determination • Molecular visualization • Validation • Chemical shift prediction • Dynamics • Residual dipolar coupling • Meta packages • General purpose • Instrument manufactures

13. NMRbox VM Content – Productivity Tools • over a dozen editors • scientific python packages • R and R tools • office tools • drawing tools • Octave • shells • browsers • Dropbox • virtual keyboard (helpful with VNC)

14. NMRbox VM Features added in Release 3 • GPUs to support 3D drawing • PyMOL, VMD, Chimera, and others • GPUs to support CUDA processing • NAMD, others coming soon • Commercial software • dataChord spectrum Analyst, dataChord spectrum Miner, MestReNova • Matlab compiled binaries • ALATIS, GUARDD, TITAN See Release notes at - https://nmrbox.org/files/release-notes-version-3-0.pdf

15. Virtual Machine Terminology A software-based emulation of a guest computer backed by the physical resources of a host computer, managed by a hypervisor. VM = • Access • Local installation (standalone or downloadable) • Connect to server (PaaS = Platform-as-a-Service) • Advantages • Over-subscribe the host computer • Snapshot the VM and restore to any point • Run multiple OS’s on a single computer • “spin-up” VMs in minutes • Dynamically load balance VMs across multiple hosts • No performance penalties on modern computers

16. Standalone NMRbox VM host computer hypervisor NMRbox (guest) OS / NMR software shared folder user accounts

17. PaaS NMRbox VM Authentication Server VM host server Remote Users NMRbox VM - 2 CPU, Ram, NIC NMRbox VM - 1 CPU, Ram, NIC Cloud Storage user data user home folders user data NMR Software OS Files backups High Performance Storage

18. PaaS deployed with enterprise-class resources • 100 GB network • 12 VM servers • 480 cores • 3.8 TB memory • Redundant internal network • Network attached storage • 100’s of TBs available to NMRbox • Ultra reliable cloud storage in excess of PB 38 NVIDIA GPUs dramatically increasing graphic performance & CUDA processing

19. VM Requirements for Users • Server based PaaS VM • ssh or VNC (Windows, OSX, Linux, tablet, phone, 32/64-bit hardware, …) • network connection • Standalone / Downloadable VM • 64-bit hardware (Windows, OSX, Linux,…) • any modern laptop and desktop Oracle VirtualBox VMware Workstation VMware Fusion VMware Player

20. Benefits • Users • “Zero-configuration” • Access • Training • Computational resources • Discovery • Persistence • Reproducibility • Cost • Developers • Stable platform • Discovery • Usage metrics • Persistence • Community • Developer tools • Computational resources • Instructors • Access to NMRbox VMs for courses and workshops

21. Practical aspects • Large VM model • Many cores, high memory, and GPUs • Multiple users per VM, each user has two VMs (username.nmrbox.org and username2.nmrbox.org) • GPUs restrict VM management • Backups • User home folders backed up daily • Snapshots taken daily • Backups / snapshots are for catastrophic failures • Data deleted accidentally by users MAY be able to be restored • timing (was the file captured, still exist?) • Downloadable version • Downloadable version in final testing

22. Practical aspects • NMRbox versions and updates • Super major release (every 2 years) • Update the version of the OS • Ubuntu LTS (two year release cycle, 5 years of updates) • Major releases (1 - 4 months) • Update software versions & add new software • Short PaaS outage (a few minutes) • Minor releases (as needed) • Add software packages • Packages added “live” to PaaS (no disruption to service) • Patches (as needed) • Security patches and bug fixes (no disruption to service)

23. Practical aspects • Older versions • All versions archived • After major release, old version remains running with reduced resources (e.g. version2.nmrbox.org) • Older versions available for download

24. Practical aspects • Large memory VM • A large memory VM can be “spun-up” for users if needed • Home folder and archive folder • Each user has two home folders; /home/nmrbox/username and /nmr/archive/username • Google Group • We have started a Google Group at https://groups.google.com Search for NMRbox to join. • Support • Email support@nmrbox.org

25. Practical aspects • Host workshops with NMRbox VMs • The NMRbox team will “spin-up” custom VMs to support other workshops • File permissions and access • Home and archive folders are not accessible to others by default. Will setup lab groups if desired. • /public folder for quick sharing • Contact us • Suggestions for packages to include • Suggestions about the package • Issues with the NMRbox platform • Trash Folder • Trash folders cleaned (files older than 30 days)

26. NMRbox Usage 500+ worldwide users

27. NMRbox Usage 37/50 US States, ~80% of users from US

28. NMRbox Usage 25 Countries

29. NMRbox Usage

30. Cite NMRbox Very Important!! If you utilize NMRbox in your research please cite and acknowledge us. Details at https://nmrbox.org NMRbox: A Resource for Biomolecular NMR Computation. Maciejewski, M.W., Schuyler, A.D., Gryk, M.R., Moraru, I.I., Romero, P.R., Ulrich, E.L., Eghbalnia, H.R., Livny, M., Delaglio, F., and Hoch, J.C., Biophys J., 112:1529-1534, 2017. [PMID: 28445744, DOI: 10.1016/j.bpj.2017.03.011] "This study made use of NMRbox: National Center for Biomolecular NMR Data Processing and Analysis, a Biomedical Technology Research Resource (BTRR), which is supported by NIH grant P41GM111135 (NIGMS)."

31. Logistics • VNC client • RealVNC Connect/Viewer (v6.x) • https://www.realvnc.com/download/viewer • NMRbox account • reset password if needed @ https://nmrbox.org/forgot-password

32. RealVNC Benefits • VNC server starts automatically when connecting • Display port determined automatically • Fully encrypted connection • Single sign-on authentication • Default local printer mapped to VNC server • VNC sessions stay active when closed • Free for users

33. Quick tour • account management • connect to VM • set the display resolution • software inventory • file transfers • support resources

34. Account management https://nmrbox.org

35. Account management https://nmrbox.org

36. Connect to NMRbox PaaS • Launch RealVNC client • In the address bar enter … • ordinarily: <username>.nmrbox.org • today: umd1.nmrbox.org or umd2.nmrbox.org • NOTE: Do not manually start a VNC server and do not enter a VNC port number • Enter your NMRbox username / password • NOTE: username is pre-filled with your local machine username, which may not be your NMRbox username • Connections are saved for easy access. • If you sign-up for a RealVNC account, the connections can be shared across your devices.

37. Connect to NMRbox PaaS – Common issues • Entering the incorrect username / password • Getting your computer banned • Eight unsuccessful RealVNC logins in a short period of time will ban the computer for a short while • ssh can also be banned, but is controlled by a different system • Firewalls • NMRbox PaaS is open • Some institutions block outgoing traffic • Blocking generally occurs on guest WiFi which sometimes only allows web traffic

38. Window size & screen resolution • Resolution changer • launch menu (top left) • nmrbox-util • Resolution Changer • ALT: resolution-changer.py • select resolution • Apply #4 #2 #3 #1 #5

39. Window size & screen resolution • RealVNC Window • Window can be resized • Behavior is dictated by Settings  Options  Scaling parameter • Full screen mode • Automatically scales both dimensions • For high resolution laptops, use a smaller resolution and scale-up • Display can span two screens. Three steps • resolution_changer.py  Double horizontal width • RealVNC Settings  Expert  UseAllMonitors = True • RealVNC Menu bar  Full screen mode #5

40. Window size & screen resolution • RealVNC Window • Closing VNC window does not close server. You remained logged into the NMRbox VM server • What if the VNC display is mis-behaving or you want to force a logout? • Open a terminal • > kill-vnc-server • All VNC server sessions that are running under your username are shown • Should only ever be one • Current display is indicated • Select the session to kill • This will log out of the server • Kill any running jobs • Kill the VNC session • Re-establish a connection from RealVNC #5

41. Software inventory • Inside NMRbox • /usr/software • /usr/software/bin • launch menu • NMRbox.org website • Software Registry: https://nmrbox.org/registry • Release Notes: https://nmrbox.org/documentation #5

42. File transfer RealVNC built-in file transfer Good old fashion scp / sftp Dropbox Globus #5

43. File transfer: RealVNC built-in file transfer • SEND files from local computer to NMRbox • hover mouse near top center of VNC window • click on “File transfer” icon • follow directions • FETCH files from NMRbox to local computer • click VNC icon in Xubuntu status bar (top right, next to username) • select the icon in the top right with three horizontal lines • select “File transfer …” menu item • follow directions #5

44. File transfer: scp / sFTP • Secure Copy (scp) • from a terminal change to appropriate directory • scp -r directory <username>.nmrbox.org:~/<path> • scp file <username>.nmrbox.org:~/<path> • Secure FTP (sFTP) • Open your favorite sFTP client such as FileZilla • Enter login information • Host: <NMRbox username>.nmrbox.org • Username: <NMRbox username> • Password: <NMRbox password> • Port: 22 #5

45. File transfer: scp / sFTP • accept the hostkey • drag and drop files #5

46. File transfer: Dropbox • Inside NMRbox open a terminal • > dropbox start -i • This will launch an installer • enter your Dropbox username / password, or • create an account • Useful commands • > dropbox help • > dropbox status • > dropbox stop • > dropbox start • Once connected the Dropbox icon • will appear in the upper right hand • corner of the RealVNC window #5

47. File transfer: Globus • What is it? • manage file transfers between any of your “endpoints” • personal “endpoints” • endpoints on your personal computers running “Globus Connect Personal” • institutional “endpoints” • NMRbox is a registered institutional “endpoint” • How do I get it? • homepage: https://www.globus.org • look in footer for “Globus Connect Personal” (not “Server”) • download for Mac / Linux / Windows • follow directions • NOTE: Globus Connect Personal is installed on your personal computer, not NMRbox #5

48. File transfer: Globus • How do I authenticate? • Gotohttps://www.globus.org and select “Log In” button #5

49. File transfer: Globus • How do I start the file transfer window? • Method 1: From the Globus icon (located in system tray) • Control Globus Personal Connect • Transfer Files • Method 2: Gotohttps://www.globus.org/app/transfer #5

50. File transfer: Globus Select endpoints and transfer… #5