The Anton Project at the National Resource for Biomedical Supercomputing

1. The Anton Project at the National Resource for Biomedical Supercomputing HPC User Forum Houston, TX April 6, 2011

2. Contributors • Joel Stiles, Ph.D, M.D. Director, NRBSC • Markus Dittrich, Senior Scientific Specialist NRBSC • Ralph Roskies, Scientific Director, PSC • Jim Kasdorf, Director of Special Projects, PSC

3. Disclaimer • Nothing in this presentation represents an official view or position of the U.S. National Institutes of Health, nor of D.E. Shaw Research, nor of the Pittsburgh Supercomputing Center nor of Carnegie Mellon University. All material is from public sources. All errors are my own (JK).

4. The PSC Biomedical Initiative • Founded in 1987 • Funded by U.S. National Institutes of Health • First extramural NIH supercomputing program • Renewed every five years • With 2006 grant, renamed: National Resource for Biomedical SuperComputing

5. National Resource for Biomedical Supercomputing • Performs leading-edge research in high performance computing and the life sciences • Fosters exchange between PSC expertise in computational science and biomedical researchers nationwide

6. National Resource for Biomedical Supercomputing • Computational biomedical research • Outreach to the national biomedical research community • User training • Software distribution • Publications • Online courses and workshop webcasts.

7. NRBSC Research Areas • Microphysiology • Volumetric visualization and analysis • Computational structural biology

8. Anton • Anton van Leeuwenhoek • (October 24, 1632 – August 26, 1723) • Perfected the microscope • “Father of microbiology”

9. Anton •  Massively parallel supercomputer • Designed and built by D.E. Shaw Research (DESRES) • Special-purpose system for Molecular Dynamics (MD) simulations of proteins and other biological macromolecules

10. David E. Shaw • B.S. / M.S. UCSD • Ph.D. Stanford, 1980 • Columbia CS Department until 1986 • Formed D.E. Shaw Group in 1988 • “King Quant” (Fortune, Feb. 5,1996) • Assets Under Management ~$20B • Net worth: $2.2B (March 2011)

11. David E. Shaw • Founded DESRES, 2002 • Senior Research Fellow, Columbia University, 2005 • PCAST appointments, 1994 and 2009 • Council of Competitiveness Executive Committee • Fellow of AAAS • American Academy of Arts and Sciences • American Association for the Advancement of Science

12. D.E. Shaw Research(www.deshawresearch.com) • Independent research laboratory • Conducts basic scientific research in computational biochemistry • Involved primarily in: • The design of novel algorithms and machine architectures for high-speed molecular dynamics (MD) simulations of proteins and other biological macromolecules • The application of these simulations to basic scientific research in structural biology and biochemistry, and to the process of computer-aided drug design

13. The MD Challenge Molecular dynamics (MD) simulation allows scientists to study the microscopic dynamics of biomolecular systems and thus gain crucial insight into processes such as: • Protein folding • How drug molecules interact with proteins and nucleic acids • The mode of operation of many important cellular proteins such as ion channels or other transport proteins Importantly, most relevant biological events take place on a timescale of milliseconds (10-3 s) or longer

14. Anton • Anton is a special-purpose supercomputer designed to specifically address the need of MD simulations to access the biologically-relevant realm of millisecond timescales. • Anton was designed to dramatically increase the speed of MD simulations. • Anton runs MD simulations fully in hardware (as opposed to running MD software on general purpose hardware).

15. Anton • Compared to the previous state-of-the-art Anton provides a speedup of ~100x, rendering millisecond timescales attainable • Anton uses custom-designed ASICs and novel simulation algorithms

16. Anton Processing Node

17. Anton ASIC

18. A 512-Node Anton Machine

19. Anton Benchmarks DHFR benchmark, simulated biological time)

20. MD Simulations on Anton • Standard, unbiased MD runs in NVE, NVT, and NPT ensembles are possible. • Simulate proteins, lipids, water, RNA, DNA. • On a 512-processor Anton, simulation systems with up to 120,000 atoms can be simulated effectively. • Supports standard MD forcefields(CHARMM, AMBER, OPLSAA, ...). • Input is in Maestro format (similar to what is used in the DESRES Desmond code).

21. Anton Benchmarks All simulations used 2.5-femtosecond time steps with long-range interactions evaluated at every other time step and a Berendsen thermostat applied every 100 time steps

22. Anton @NRBSC/PSC • The availability of Anton at the National Resource for Biomedical Supercomputing (NRBSC) at PSC was made possible by a partnership with DESRES. • DESRES made available an Anton machine without cost for non-commercial research use by not-for-profit institutions. • The Anton at NRBSC is the only one available outside DESRES.

23. Anton @NRBSC/PSC • Operations are funded by a two-year $2.7 million grant from the National Institute of General Medical Sciences (NIGMS), part of NIH. • This is the first time NIH has provided funds to make available a supercomputing system for national use. • Grand Opportunities (GO) grant mechanism • from ARRA funds • One of only 14 GO awards from NIGMS

25. Anton Hardware @NRBSC • 512-node Anton production machine • Four cabinets, each with 128 special-purpose ASICs • Operational since October 2010 • 64-node Anton test machine • One cabinet with 64 special-purpose ASICs • Operational since May 2010 Access to both Anton back-ends is via a Linux front-end running a standard PBS queuing system.

26. Anton Hardware @NRBSC • AntonFS storage cluster • Dedicated to Anton MD trajectory storage • Lustre-based file system with ~450 TB of storage • Kollman analysis cluster • Dedicated to analysis of MD trajectories generated on Anton • Four nodes each with 12 cores (2 * 3.33GHz Intel Xeon Westmere EP X5680 6-core), 96 GB of RAM and one NVIDIA Tesla C2050 card, QDR IB

27. Anton Allocations • In spring of 2010 NRBSC solicited proposals from the scientific community; 67 proposals were received. • Proposal review and assignment of allocations was conducted by a committee convened by the National Research Council at the National Academies of Science. • 15 proposals were awarded 100,000 node-hours each during the first phase of the project (October 2010 through March 2011).

28. Anton Allocations • 32 proposals were awarded ~50,000 node hours each during the second phase of the project (April 2011 through September 2011).

29. Science Projects on Anton The range of science topics proposed for simulation on Anton is quite diverse: • Studies of membrane channels and transporters • Protein folding • Design and evaluation of novel enzyme catalysts • DNA sequencing • Protein dynamics • Signal transduction • Force field development • ….

30. Science Projects on Anton Currently (as of March 2011), the investigators of phase 1 are wrapping up their simulations and are conducting preliminary analyses of their generated data.

31. References • “New Architectures for a New Biology”, Martin Deneroff, February 22, 2007 www.nersc.gov/projects/SDSA/meetings/SIAM_CSE07/SIAM_2-07-desres.pdf • “Anton, a special-purpose machine for molecular dynamics simulation”, D.E. Shaw, et al., Communications of the ACM,  Volume 51, Issue 7, July 2008 • “Herculean Device for Molecular Mysteries”, John Markoff, The New York Times, July 8, 2008

32. References • “Millisecond-scalemoleculardynamics simulations on Anton”, D.E. Shaw, et al., SC’09 Proceedings • “Exploiting 162-Nanosecond End-to-End Communication Latency on Anton”, Ron O. Dror, et al., SC10 Proceedings

33. References • “Atomic-Level Characterization of the Structural Dynamics of Proteins”, D.E. Shaw, et al. Science 15 October 2010: Vol. 330 no. 6002 pp. 341-346 • “Custom-Built Supercomputer Brings Protein Folding Into View”, Robert F. Service, Science 15 October 2010: 308-309.

34. References • “Anton Supercomputer Proves Its Mettle”, Chemical & Engineering News, October 14, 2010http://pubs.acs.org/cen/news/88/i42/8842notw1.htm • “Supercomputer sets protein-folding record”, Nature, October 14, 2010, http://www.nature.com/news/2010/101014/full/news.2010.541.html

35. Jim Kasdorf kasdorf@psc.edu www.psc.edu www.nrbsc.org