Vijay K. Agarwala Senior Director, Research Computing and Cyberinfrastructure

1. Research Computing: Critical Needs and Opportunities at a University Based Academic Computing Center HPC User Forum October 13 -14, 2008 High Performance Computing Center Stuttgart (HLRS),Germany October 16th, 2008 Imperial College, London Vijay K. Agarwala Senior Director, Research Computing and Cyberinfrastructure The Pennsylvania State University University Park, PA 16802 USA vijay@psu.edu, 814.865.2162

2. Vice Provost for Information Technology and CIO Associate Vice Provost Financial Services ITS Organization Research Security Research Computing and Cyberinfrastructure (RCC) Security Operations and Services (SOS) Marketing and Communication Human Resources Libraries Instruction Teaching and Learning with Technology (TLT) Digital Library Technologies (DLT) Communication Business Telecommunications and Networking Services (TNS) Administrative Information Services (AIS) Consulting and Support Services (CSS) Support

3. Information Technology Services Faculty Advisory Committee on Research Computing and Cyberinfrastructure Senior Director Research Computing and Cyberinfrastructure Meets the high-end computing technology needs of scholars in their research and teaching endeavors. The group partners with faculty members and collaborates with technology companies and other research organizations. Domain Expertise and Consulting Support Software Development and Programming Support Visualization and Telecollaborative Systems High Performance Computing Systems

4. Research Computing and Cyberinfrastructure • Provide systems services by researching current practices in operating system, file system, data storage, job scheduling as well as computational support related to compilers, parallel computations, libraries, and other software support. Also supports visualization of large datasets by innovative means to gain better insight from the results of simulations. • Enable large-scale computations and data management by building and operating several state-of-the art computational clusters and machines with a variety of architectures. • Consolidate and thus significantly increase the research computing resources available to each faculty participant. Faculty members can frequently exceed their share of the machine to meet peak computing needs • Provide support and expertise for using programming languages, libraries, and specialized data and software for several disciplines. • Investigate emerging visual computing technologies and implement leading-edge solutions in a cost-effective manner to help faculty better integrate data visualization tools and immersive facilities in their research and instruction. • Investigate emerging architectures for numerically-intensive computations and work with early-stage companies. For example: interconnects, networking, and graphics processors and FPGA for computations. • Help build inter- and intra-institutional research communities using cyberinfrastructure and grid technologies. • Maintain close contacts with NSF and DoE funded national centers, and help faculty members with porting and scaling of codes across multiple architectures.

5. Compute Engines LION-XJ 144 nodes 288 quad-core procs Memory-16GB /node IB interconnect Pleiades 170 nodes 340 processors 340 GB RAM 35.1 TB storage Hammer/LION-XD 16 nodes 64 processors Memory-128GB /node Infiniband interconnect Unisys ES7000 3 nodes 64 processors 192 GB RAM Myrinet interconnect LION-XK 144 nodes 288 quad-core procs Memory-32GB /node IB interconnect LION-XO 132 nodes 368 processors 1280 GB RAM Silverstorm Infiniband interconnect LION-XB 16 nodes 128 processors 512 GB RAM Pathscale Infinipath Infiniband interconnect HPC Storage Farm 100 TB of disk needs to be 1000 TB LION-XC 140 nodes 560 processors 1664 GB RAM

6. Programs, Libraries, and Application Codes in Support of Computational Research • Compilers and Debuggers: AbsoftProFortran, GNU Pascal, IBM XLF, IBM XLC/C++, Intel Fortran, Intel C/C++, Lahey/Fujitsu Fortran 95 Pro, Portland PGI Compilers and Tools, Java, PathScale EKO compiler suite, TotalView, DDT, Valgrind • Computational Biology: BLAST, Blastall, Cister, ClustalW, ClustalX, Dotter, FASTA, fastDNAml, GeneMachine, GENSCAN, HMMgene, MrBayes, MZEF, PHRED/PHRAP/Consed, PHYLIP, ReadSeq, RepeatMasker, SEG, sim4, Sputnik, Treetool, wuBlast • Computational Chemistry and Material Science: FHI98MD, Gamess, Gaussian 03, GaussView, SemiChem, NWChem, Jaguar, Maestro, CHARMM, WIEN2K, VASP, ThermoCalc, Accelrys Material Studio, ADF, tmolex, Amber, Gromacs, NAMD, WxDragon, Molden, CPMD, Rosetta, CCP4 • Finite Element Solvers: ABAQUS, ANSYS, FLUENT, GAMBIT, FIELDVIEW, LS-DYNA, MD/Nastran, OpenFOAM • Mathematical and Statistical Libraries and Applications: ATLAS, BLAS, ESSL, IMSL, LaPack, ScaLaPack, MASS, GOTO, Intel MKL, AMD ACML, Mathematica, MATLAB, Maple, Distributed MATLAB, PETSc, NAG, StarP, Watson Sparse Matrix Solver • Solid Modeling: MD/Patran • Statistics: R, SAS • Parallel Libraries: MPICH, Optimized versions of MPICH for high-performance cluster interconnects, Parallel IMSL, Distributed MATLAB, StarP, Distributed Maple • Optimization: GAMS / CPLEX, Csim, Tomlab • Multiphysics: Comsol All software installations are driven by faculty. The software stack on every system is customized and entirely shaped by faculty needs.

7. Participating Research Centers Joint research and education initiative (NSF, DOE, PSU) focused on understanding molecular issues related to environmental chemical kinetics, geochemical cycling of elements, fate and transport of contaminants, and carbon sequestration. (Dr. Susan L. Brantley, Professor of Geosciences) Virtual center that integrates genetic, immunological, ecological and other studies to understand how disease processes work, and how they inter-relate across time and length scales. (Dr. Ottar Bjornstad, Associate Professor of Entomology and Biology; Dr. Bryan Grenfell, Alumni Professor of Biology ) Center for Gravitational Wave Physics (NSF, PSU) fosters research of a truly interdisciplinary character linking the highest caliber astrophysics, gravitational wave physics and experimental gravitational wave detection in the pursuit of the scientific understanding of gravity. (Dr. Lee S. Finn, Professor of Physics, Astronomy and Astrophysics) Interdisciplinary center (NSF, NIH, PSU) focused on identifying issues in statistics, research design, and measurement emerging in the prevention and treatment of problem behaviors, particularly drug abuse. (Dr. Linda M. Collins, Professor of Human Development and Family Studies, Statistics)

8. Participating Research Centers Collaborative effort (NSF, PSU, Georgia Tech) aimed to educate the next generation of scientists and engineers in the emerging field of materials design. (Dr. Zi-Kui Liu, Professor of Materials Science and Engineering) IceCube@Penn State Large collaboration (NSF, DOE, PSU, and others) of about 150 scientists working to use the AMANDA and IceCube telescopes to detect ultra-high energy neutrinos. (Dr. Douglas F. Cowen, Professor of Physics, Astronomy and Astrophysics) Center aimed at describing, modeling, and understanding the Earth's climate system. (Dr. Michael E. Mann, Associate Professor of Meteorology) A hub to connect experimental and simulation activities through the organization of collaborative projects, short courses and workshops. (Dr. Jorge O. Sofo, Associate Professor of Physics, Astronomy and Astrophysics)

9. Low level High level -90.4 ppm 2-layer ONIOM Method HF/6-311+G(d,p)/HF3-21G* HPC

10. Performance of LS-DYNA Case Study – Blast Loading • Blast load using ConWep algorithm • 450,000 dofs, spherical blast for 11ms • Study of mesh convergence for plastic strain stability Professor: Ashok D.Belegundu Student : Vikas Argod Dept of Mechanical and Nuclear Engineering

11. Visualization Services Staff members provide consulting, teach seminars, assist faculty and support facilities for visualization and VR. • Recent support areas include: • Visualization system design and deployment • 3D modeling and geometry exchange (e.g. FormZ) • Visualization development applications and programming toolkits (e.g. OpenDX, VTK ) • VR development and device libraries (e.g. VRPN, VMRL, JAVA3D, OpenGL, OpenSG, CaveLib) • Domain specific visualization tools (e.g. VMD, SCIRun) • Telecollaborative tools and facilities support (e.g. Access Grid, VNC) • Parallel graphics and online visualization (e.g. Paraview, DCV) • Programming for graphics (e.g. C/C++, JAVA3D, Tcl/Tk, Qt)

12. Visualization Facilities Our goal is to foster more effective use of visualization and VR techniques in research and teaching across colleges and disciplines via strategic deployment of facilities and related support. • Locating facilities strategically across campus for convenient access by targeted disciplines and user communities • Leveraging existing applications and workflows so that visualization and VR can be natural extensions to existing work habits for the users being served • Providing outreach seminars, end-user training and ongoing staff support in use of the facilities • Working on an ongoing basis with academic partners to develop and adapt these resources more precisely to fit their needs • Helping to identify and pursue funding opportunities for further enhancement of these efforts as they take root

13. Immersive Environments Lab (IEL) in partnership with School of Architecture and Landscape Architecture208 Stuckeman Family Building • Focused on teaching and research in the experiential understanding of design spaces by architecture and landscape architecture students • 3-screen 3D-stereo multi-OS display offers multi-modal immersive environment • Experiential design review in design studio and digital media courses • Telecollaborative studio using standard definition video and 3D application sharing with Carleton University, Spring 2007 (lab also supports Access Grid) • Research into immersive and collaborative tools for design professions • Develop application and data integration workflows for ARCH/LARCH (Building Information Modeling, energy and structural analyses, land use planning, etc.) in conjunction with ICON lab under internal Bowers support

14. Immersive Construction Lab (ICon) in partnership with Architectural Engineering306 Engineering Unit C • Focused on research and teaching in the use of immersive visualization and VR techniques for planning and management of large construction projects • 3-screen, 3D-stereo, Windows desktop, immersive information environment • VR extensions to commercial construction planning applications, custom development of “VR-like” teaching modules • Industry partners provide real world use cases for studying the practical application of tools under development • Linked SMART Board allows dynamic VR updates from scheduling applications, etc. • Laptop display sharing facilitates group collaboration among students • Building upon ICON lab, IEL and related work, Penn State hosted CONVR 2007 international conference on construction applications of virtual reality

15. Visualization/VR Lab, 215 Osmondin partnership with Materials Simulation Center • VR facility for central campus science community (Materials Science, Molecular Biology, Physics and more) • 8’ x 8’ active stereo display • Tracked devices for user interaction • Linux console workstation • Complement of open source data visualization tools (VMD, VTK, Paraview - can be built upon in response to user needs) • Seminars for teaching graduate students on use of VR tools • Initial users in Materials Science and Molecular Biology

16. Visualization/VR Lab, 336 IST Buildingin partnership with Computer Science and Engineering • Facility targets compute-intensive applications in science, engineering and related disciplines • Large-format 3D stereo display (6.75 x 9 ft., 1400 x 1050 pixel) for VR applications • 2 x 2 tiled display (6 x 8 ft., 2800 x 2100 pixel) for high-resolution applications • Linux console workstations • Interactive device support • Initial complement of data visualization tools (VMD, VTK, Paraview, SCIRun) to be built upon in response to user needs • Opened in October 2007 • Seminars and outreach activity underway

17. Sports Medicine VR Labassisting a partnership between Kinesiology, Athletics and HMC Lasch Football Building • Special purpose lab supports study of perception action disruptions in posture and balance related to mild traumatic brain injury (e.g. concussions), elderly populations, etc. • Motion in VR display is synchronized with measurement from EEG, postural tracking and force plate instrumentation • Enhanced two-screen lab (wall and floor) in development for Recreation Building for broader use by faculty in kinesiology, psychology, engineering science and mechanics, SSRI and Hershey Medical Center

18. Collaborative Tools in Research • Telecollaboration • Adobe Connect (site license) • Vyew • Access Grid (many-to-many) • Document / Source Sharing • Subversion • Wiki • MediaWiki (public domain) • Confluence Wiki (commercial) • Web-Based Science Gateways • Materials Simulation Center Gateway (with Materials Research Institute)

19. ACCESS Grid Teleconferencing FacilityRoom 140 Computer Building • Allows faculty to participate in Access Grid events with international academic community • Scalable, multi-group telecollaboration (voice, video, application sharing) using multicast internet connections. • Small number of highly satisfied users • Needs greater awareness and adoption Ongoing research collaboration: Dean Snow (ANTHY), Craig Cameron (BMB) Research reporting: Richard Alley (GEOSCI), Donald Bryant (BMB), Mark Gahegan (GEOG) Virtual conferences: Genomics and Bioinformatics, SC Global, SC Desktop

20. Industry Outreach in partnership with Institute for Computational Science (ICS) and Industrial Research Office (IRO) " ... to out-compete, we must out-compute .... " • Putting Pennsylvania based companies at a competitive advantage by helping meet their large-scale computing needs. Many small-to-medium sized companies, and even larger ones, do not have enough in-house expertise and resources for large-scale computations and as a result have not been able to use simulation and analysis tools with far greater frequency to help them innovate faster and become more competitive. • Helping develop stronger relationship between faculty and industry by providing computational services; may lead to more alignment between faculty research areas and industry needs and positive impact on economic development along the I-99 corridor and throughout the commonwealth.