NCAR Supercomputing ‘Data Center’ Project

1. NCAR Supercomputing ‘Data Center’ Project An NCAR-led computing ‘facility’ for the study of the Earth system

2. Outline • A Problem • An Opportunity • NSF’s Petascale Roadmap • A Solution • Facility Proposal: Site and Cost • Partners • The Scientific Payoff • Next Steps • Schedule and Panels

3. NCAR Leadership in Supercomputing… • One of the founding missions of NCAR was: “…to provide, or arrange for provision of facilities for the scientific community as a whole that whose initial cost and upkeep lie beyond the capability of individual universities or research groups.”–Preliminary Plans for a National Institute for Atmospheric Research. 1959 – NCAR Blue Book • Note: the wording does not imply physical collocation. • This mission does confer a responsibility that cannot be delegated - namely maintaining an complete integrated cyberinfrastructure (CI) system for modeling and data analysis that meets our scientific community’s needs.

4. Examples of NCAR simulation science today • Global change climate ensembles • Weather Research Forecast • Geophysical Turbulence • Fire storm front modeling • Space weather • More…

5. A problem • NCAR Mesa Lab computer facility is quickly becoming obsolete • Power, cooling and floor space will be inadequate beyond the next procurement • Science is being restricted by focusing on capacity ahead of capability

6. CMOS Trends Continue …

7. Chips: Faster, Cheaper but Hotter

9. An OpportunityNSF’s Petascale Roadmap DRAFT “Overarching Recommendation: Establish a Petascale Collaboratory for the Geosciences with the mission to provide leadership-class computational resources that will make it possible to address, and minimize the time to solution of, the most challenging problems facing the geosciences.” www.joss.ucar.edu/joss_psg/meetings/petascale/

10. Software Service Provider (SSP) S&E Community Portable, Scalable Applications Software & Services Science-Driven HPC Systems HPC Resource Providers Agency Partners SSP Local Storage Visualization Facilities Compute Engines SSP Private Sector Strategic Plan for High Performance Computing(2006-2010)

11. NSF Conclusions • NSF is committed to developing and implementing a strategic plan for cyberinfrastructure • Broad based plan involving the university, Federal agencies, vendors, and International partners • ATM, OCE, and EAR take different approaches to the realization of CI for their discipline • Dependent on the readiness of the community • Petascale facility is an integrating theme for the Geosciences community • High potential for the generation of new knowledge and paradigm for the conduct of research • Building and sustaining a petascale facility will be a significant challenge to budgets and technology • Consistent with NSF strategic vision for CI

12. A solution for NCAR • A new computing facility (not at the Mesa Lab) • Extensive investigations, working with consultants and internal needs resulted in a detailed set of options • Provides for 5-20 years of computing (capacity and capability) diversity based on current and predicted future trends in CMOS technology • Allows NCAR to reach beyond its current research scope

13. The facility needed • Data Center Expansion Report from NCAR’s Computing and Information Systems Lab • 20,000 (initial to 60,000) sq. ft. • 4 (to 13) MW power + generators • Cooling, etc. • On 13 acres (20 year lifetime) • Accommodates computers, staff, open space, initial and future requirements

14. Birds Eye View

15. Phase 2 Addition Phase 3 Addition Architectural View

16. (Water, Political Complications, Fiber Optics) (Electrical Capacity) Importance of Site Selection • Limited selection of sites that meet criteria • Size (10-15 acres) • Electrical capacity (up to 24 MW) • Fiber optic route (dark fiber) • Investigated • Marshall • Louisville • Longmont • Westminster • New partners and options are now being sought • IBM • Colorado School of Mines • Colorado State University • University of Colorado • University of Wyoming

17. Cost Drivers • Primary Drivers • Tier III Reliability • Mechanical Systems • Electrical Systems • Engineering • Secondary Drivers • Building Size • Land Site • Facility - up to $75M (one time) • Operations - $15M/year? (2X) • Computing increments - $15M/year (2X) • Computing infrastructure - $5M/year

18. The Scientific Payoff…

19. A petascale computer will enable scientists to … • Do credible regional climate modeling for decision support. Requires resolving individual mountain ranges and ocean western boundary currents. • Model climate and weather in a fully coupled mode. • Better understand the marine biogeochemical cycles. Requires resolving ocean mesoscale eddies. • Accurately simulate the dynamical, microphysical and radiative cloud processes. • Improve seismic predictions and understand the structure of the inner core as well as the fine structure of the lower mantle.

20. A petascale computer will enable scientists to • Perform new research in solid earth and environmental engineering • Assimilate thousands of earthquakes bringing the fine structure of the Earth’s mantle and inner core into focus. • Study the physical basis of land surface parameterizations by modeling soils, topography and vegetation at sub-meter scales. • More accurately predict the damaging effects of solar flares on satellites and power distribution systems by resolving the fine structure of the corona magnetic field. • Investigate energy management applications

21. Science examples …

22. 2005 Hurricane Katrina, Track Forecast 4 km, 62 h forecast Observed Track 12 km, 86 h forecast Official Forecast Landfall on 8/29 14Z, Louisiana/Mississippi Border

23. Hurricane Katrina Reflectivity at Landfall 29 Aug 2005 14 Z 4 km WRF, 62 h forecast Mobile Radar

24. WRF 4 km Hurricane Katrina 72 h Forecast Initialized 27 Aug 2005 00 Z Radar Composite Reflectivity WRF Max Reflectivity

25. Coupled Climate System Model

26. Integrated Space Weather Modeling

27. Thus …

28. Main Points • Huge scientific discoveries await geoscience modelers at 1 PFLOPS and beyond. • CMOS continues to get hotter and cheaper. The most recent acquisition tracks this trend. • Every center is (or will be) facing facility challenges in the race to these discoveries. This situation is NOT unique to NCAR. • NCAR now has a facility plan, that if successful, uniquely positions it as a world leader in geoscience simulation. • The new facility is not a crisis: it is an opportunity.

29. The Opportunity • Understanding of fundamental physical processes in the Sun-Earth system • Environmental and Energy applications not yet possible • NCAR and partners will scope/define these options • Such a facility would be a computational equivalent of the Hubble Telescope for geoscience simulation.

30. Next Steps

31. The Schedule • Formed NCAR project committee • Forming Blue Ribbon Panel and hold teleconference - mid-Oct. 2005, meet mid-Nov. • Project plan development Oct-Dec • Community engagement - Nov-Jan • Formalize partnerships - Oct-Dec • Present initial plan to National Science Foundation, mid-October, 2005 • Forge international collaborations - Nov. 2005 • Complete project plan - Feb. 2005 • Initiate facility - June 2006? • First electrons - June 2008 - March 2009?

32. Contacts at NCAR • Tim Killeen (killeen@ucar.edu) - NCAR Director • Lawrence Buja (southern@ucar.edu) and Peter Fox (pfox@ucar.edu) are co-chairs of the NCAR project team • Aaron Anderson (aaron@ucar.edu) is the computing facilities contact • Jeff Reaves (jreaves@ucar.edu) is the financial/ contracts contact

33. Concluding remarks …