Operated for the U.S. Department of Energy by Midwest Research Institute • Battelle • Bechtel

1. Operated for the U.S. Department of Energy by Midwest Research Institute • Battelle • Bechtel The Role of Computational Science in Energy Efficiency and Renewable Energy ResearchSteve Hammond, DirectorComputational Sciences CenterNational Renewable Energy LaboratoryPACT02September 2002

2. Outline • Why NREL? • Energy Scene • NREL mission and programs • Computational Sciences at NREL • Architecture and Programming Model Observations

3. Why NREL?

4. The Energy Scene • Viewed over decades, changes to energy markets are sweeping and fundamental including impacts to economy, environment, and national security. • Just after WW II, homes were heated with coal, traveled on steamships, and leaded fuel. • Today, heat with natural gas, travel on jet planes, use unleaded fuel. • In 1950, America imported 5% of its oil. • Today, U.S. imports 62% of its “fossil fuel.”

5. Dependence on Foreign OilU.S. Transportation use of Petroleum 30 Domestic Petroleum Usage Today 25 Imports 20 15 Millions of Barrels per Day 10 5 0 1970 1980 1990 2000 2010 2020 Actual: Annual Energy Review 2000 Tbls 1.2, 5.1 and 5.12 Forecast: Annual Energy Outlook 2002 Tbls 7 and 11 Split between Autos and Lt Truck: Transportation Energy Data Book Edition 21 Tbl 2.6 Updated April 2002

6. Affluence Requires Energy – Poverty Breeds Global Insecurity Affluence Japan United States France United Kingdom South Korea Mexico Poland El Salvador Russia China Poverty Bangladesh Burkina Faso Source: Energy Information Administration, International Energy Annual 2000 Tables E1, B1, B2; Gross Domestic Product per capita is for 2000 in 1995 dollars. Updated May 2002

7. Renewable Energy “factoids” • North Dakota has enough wind to supply 35% of the U.S. electrical demand. • The sunlight falling on the U.S. in one day contains more than twice the energy we consume in an entire year. • Renewable energy is plentiful. • Continued R&D is necessary to ensure that “renewables” are efficient, reliable, and affordable.

8. National Renewable Energy Laboratory • DoE Lab, located in Golden, CO • Only national laboratorydedicated to renewable energy and energy efficiency R&D • Research spans spectrum of fundamental science to technology solutions and assessment • Collaboration with industry and university partners is a hallmark • About 1000 employees, and approx. $200M/year

9. NREL Mission Statement NREL develops renewable energy and energy efficiency technologies and practices, advances related science and engineering, and transfers knowledge and innovations to address the nation's energy and environmental goals.

10. Major NREL Technology Thrusts Supply Side Wind Solar - Photovoltaics - Concentrating Solar Power - Solar Buildings Biomass - Biopower - Biofuels - Biorefinery Geothermal Hydrogen Superconductivity Distributed Energy Demand Side FreedomCAR and VehicleTechnologies - Hybrid Vehicles - Fuels Utilization Building Technologies- Zero Energy Buildings Federal Energy Management Advanced Industrial Technologies Crosscutting Basic Energy Science Analytical Studies Computational Science

11. National Center for Photovoltaics Facilities Solar Energy Research Facility Outdoor Test Area Outdoor Test Facility

12. Solar Photovoltaics • In the last 25 years, cost of electricity from PV has dropped from $1/kWh to about 20¢/kWh today. • Strong efforts, government support in Japan and Germany • U.S. market share 40% • R&D: fundamental science of materials, advanced solar cells and processes, scale-up, lower cost Sacramento Municipal Utility District (01026) Source: Technology Opportunities to Reduce U.S. Greenhouse Gas Emissions, October 1997

13. National Wind Technology Center • Facility for NREL scientists and the U.S. wind energy industry to conduct research, engineering and development of advanced wind technology • Unique facilities • 280 acres of ideal wind research land • 16 outdoor turbine test pads • Structural test facilities for turbine blades and rotors • Test facilities for wind turbine components Wind turbine structural test facilities (04364)

14. Wind Energy • In the last 25 years, cost of wind energy has decreased from 40¢/kWh to about 4¢/kWh. • 2010 goal is 2–3¢/kWh • Strong European research efforts. • R&D: improvements in turbine designs, structural dynamics, lower cost Source: Technology Opportunities to Reduce U.S.Greenhouse Gas Emissions, October 1997 Green Mountain Power Wind Plant, Vermont (05592)

15. GE WindEnergy 3.6 MW Prototype Turbine in Spain

16. GE WindEnergy 3.6 MW Prototype Turbine in Spain Boeing 747-200

17. Buildings Research Activities Investigation of boundary layer effects in transpired collectors Analyzing air leaks in HVAC equipment using infrared cameras Analyzing energy performance

18. Solar Buildings Program • Combine solar energy technology with energy-efficient construction techniques. • Help create a new generation of cost-effective buildings that have zero net annual need for non-renewable energy.

19. NREL Bioenergy Facilities Alternative Fuels User Facility ThermochemicalProcess DevelopmentUnit Bioethanol Process Development Unit Field Test Laboratory Building

20. Biomass Sources Poplars Switch grass Wood chips Sugar cane residue Municipal solid waste Alfalfa

21. Biomass Electric • Direct combustion – 7500 MWe installed capacity • Cofiring (wastes) – demonstrations • Biomass gasification combined cycle (energy crops) – in development • Regrowing biomass (energy crops) results in very low or zero net CO2 emissions • R&D: ash chemistry and deposition, advanced gas turbine technologies Source: Technology Opportunities to Reduce U.S. Greenhouse Gas Emissions, October 1997

22. Biomass Transportation Fuels • Ethanol costs have plummeted: • 1980’s: $4/gal • Current: $1.22/gal estimate • 2010: $0.67/gal estimate • Develop energy crops for bulk fuel • Employ biochemical and thermochemical processing • Displacing gasoline with ethanol in light-duty vehicles gives 90% reduction in carbon emissions • R&D: low-cost production of enzymes, development of microorganisms,energy crop advances Source: Technology Opportunities to Reduce U.S. Greenhouse Gas Emissions, Oct 1997

23. Distributed Energy Resources Goal • Increase sources of energy production – reduce dependence on large critical “targets.” • Triple distributed power generation by 2010 R&D Needs • Interconnection standards development and controlled interconnection testing critical to market development • Distribution systems research • Distribution system behavior (modeling, testing) • Distributed command and control systems development

24. Transportation Technologies & Systems • R&D in advanced vehicle and fuel technologies. • Many small efficiencies add up quickly • Smart tires • Reduced auxiliary loads • Strong partnerships with public and private organizations. • Goal: reduce dependence on imported oil and improve air quality.

25. National A/C Fuel Use • 7.1 billion gallons used for air conditioning annually. • Equivalent to 10% of U.S. imported crude oil.

26. Computational Sciences Challenge • The application area is compelling and is of increased importance since Sept. 11 • Lessened dependence on foreign energy sources improves homeland security. • Distributed energy reduces risk. • Global affluence and stability requires Energy. • NREL has a tremendous, dedicated staff. • Very strong theory and experimental focus. • People ask, “Why fix it if isn’t broken?”

27. Computational Science Theory is difficult to validate and building prototypes slow and expensive. Numerical simulation enables the study of complex systems and natural phenomena that would be too expensive or dangerous, or even impossible, to study by direct experimentation. Computation is an equal and indispensable partner, along with theory and experiment, in the advance of science and engineering.

28. Vision • Create a world-class computational science center that enable the best research in energy efficiency and renewable energy research. • Augment and substantially enhance core laboratory-wide HPC research, capability, and competency in support of NREL’s mission.

29. Goal for 2005 • Establish 1st class research group • Computational scientists for simulation, data mgmt, analysis, visualization, and collaborative problem solving environments. • Integrate CS into NREL mission and national/international renewable energy research community. • Establish multidisciplinary “Expedition Teams” to solve most pressing energy efficiency and renewable energy problems. • Establish links with other DoE Labs, C.S. programs. • Establish summer student, post doc, and visitor program.

30. Three CS focus areas at NREL • Simulation: Numerical methods & algorithms • Computational fluid dynamics • Computational chemistry • Structural mechanics • Data: Currency of the scientific process • Analysis and visualization • Data mining • Integration: Holistic view • Tools, Systems, and Enterprises • Collaborative Problem Solving Environments

31. Wind Turbines • 70m blade challenges: • No wind tunnels large enough • Shifting winds cause rotation and blades cut back into own wake • Structural analysis • Aero-acoustics • Simulate turbine ensemble • Incorporate accurate PBL information into simulations

32. Materials Research Internationally recognized Solid State Theory Group: • Study materials and alloys for more efficient photovoltaic systems and solid state lighting. • Nanostructures – given desired properties, what is the best material?

33. Model chemical systems using empirical force fields to describe how molecular potential energies change with atomic coordinates. Forces in a molecule can be computed from potential energies. With the atomic forces, the motions of the molecule can be computed by solving Newton's Equations of Motion for all of the atoms in the molecule. Molecular Mechanical Modeling of Cellulases H2O H2O H2O enzyme H2O “Reaction Box” H2O cellulose

34. CFD - HVAC Systems • Simulation predicts the distribution of air velocity and temperature using building geometry, vent locations, and fan performance curves. • Avoid over-engineered system. • Plumes of hot gas and smoke from fires can be predicted in order to assist in safety analysis • Additional applications to biohazard or chemical attacks.

35. Transportation Efficiency • Passenger comfort is the focus of vehicle climate control. A computational model of the interior of the vehicle yields comfort indicators that allow the design to be improved. • Study thermal loading. • Optimized climate control leads to smaller air conditioners and heating systems – greater fuel economy.

36. ComponentSuppliers VehicleOEMs Digital Functional Vehicle Wheel CAD Tools Mechanica Ansys Abaqus ADVISOR The Math Works Saber Packaging/Visualization StructuralAnalysis Vehicle System Level Analysis partner info (current,concepts) ICEM-CFD Fluent, STAR/CD New, Better Components Component Data CFD Data Integration #1 - Energy Focus Digital Functional Vehicle energy interactions data Users Vehicle Dynamics ADAMS ADAMS/CAR integrated systems data #2 - Emissions Cleaner, More Efficient Vehicles Data Integration VehicleData Materials MVision Matls partner info Lab Partners Market Data (current,concepts) Optimization Motor Wiz Crash Analysis/Dyna Customers Research Genesis Visual Doc BMX, DOE

37. Strategy • Visit the diverse research groups to assess short-term and long-term needs. Set priorities. • Cannot go it alone – • Look for collaboration opportunities with university community. • Develop close ties to other DoE CS programs. • Seek appropriate funding opportunities. • Initiate student, post doc, & visitor program. • Integrate NREL’s research agenda and CS Initiative into the national CS infrastructure.

38. Initial Assessment • Tremendous opportunity to make a difference. • Most extant modeling and analysis occurs with greatly simplified models. • Most work currently done on PCs and workstations • Limited use of machines at other DoE labs. • Solid State Theory, Wind, Molecular Mechanical Modeling, and Transportation Systems – biggest initial HPC users.

39. First steps … • Started March 4th. • IBM SP “starter system” arrived March 28th. • Hired three staff members. • Planning office space, fall occupancy, 9 staff. • Identify key early adopters and set milestones for early success stories. • Work with other Research Center staff to develop CS strategic plan and implementation. • Long term - planning new building for FY09 occupancy.

40. NREL’s Applications • Complex geometries lead to unstructured grids. • Heterogeneous computations. • Often grids are adaptively refined (dynamic) leading to load balancing issues. • Global communications required.

41. Scientific Computing Observations • The high performance computing market is a niche market at best. • With the exception of a few Grand Challenge applications and “publicity stunts,” vast majority of scientific computing now done on 128 PEs or less. • Limited scaling in current application codes. • Better turn-around times - queue wait times excessive for large PE counts. • Linear charging schemes.

42. The Case for Shared Memory • Relatively easy to build shared memory systems with 128 PEs • What’s so good about shared memory systems? • Much simpler programming model. • Easy to start parallelizing. • Natural load balancing – threads. • Dynamic load balancing nearly impossible with MPI. • Tools, tools, tools, …

43. Shared Memory Systems • Need improved hardware and software support for shared memory programming. • Low thread overhead (spawn, synchronization, …) • User control of memory placement (critical in distributed shared memory, NUMA, systems) • Application development environment in shared memory systems far superior to distributed memory systems. • Sun and SGI continued viability??? • Multiple processors on a chip could help.

44. Large Systems • Suppose that we have efficient shared memory systems. • Can’t ignore the grand challenge problems. • “Chunky” clusters - build large systems from clusters of shared memory systems. • While at NCAR we studied use of large systems for climate modeling.

45. Message Passing Analysis • Suppose that you could sustain 16 Gflops on a single shared memory system. • A sustained Teraflop system could be assembled from a cluster of 128 of these nodes. • Used 2D decomposition of NCAR’s CCM3.2 (MPI). • For simplicity, we assumed computation and communication are both perfectly load balanced. • Counted number of bytes and number of messages per day of simulated climate (2 day - 1 day) per node.

46. High Resolution Climate Model Message Passing Statistics NCAR’s CCM3 T170L18 4x32 decomposition, 128 “node” system.

47. Interconnect Needs: Thin and Fat Pipes • Low Latency “Thin Pipes” • Control, Status, • Global Reductions • High Bandwidth “Fat Pipes” • Bulk data transfer • Data transpositions • Real world applications need both. • CM5 had the fat tree and the control network.

48. Summary • Energy is crucial for prosperity and security. • NREL is America’s premier laboratory for energy efficiency and renewable energy research. • NREL is establishing a world class Computational Science capability in support of and commensurate with its other research efforts. • Improvements in shared memory systems would be beneficial to scientific computing community.

49. Operated for the U.S. Department of Energy by Midwest Research Institute • Battelle • Bechtel