Bob Yantosca Software Engineer

1. GEOS–Chem Updateand a look ahead to new things Bob YantoscaSoftware Engineer Atmospheric Chemistry Modeling GroupSchool of Engineering & Applied Sciences Harvard University Jacob Group MeetingWednesday, January 23, 2008

2. Bob Y – Jacob Group Meeting – Page 2 Table of Contents • Model updates since my last group meeting (Dec 2006) • Current met field availability for GEOS–Chem • GEOS–Chem benchmarking updates • GEOS–5 validation and outstanding issues • Adoption of Earth System Model Framework and porting of GEOS–Chem to IBM BlueGene architecture • Stuff in the pipeline

3. Bob Y – Jacob Group Meeting – Page 3 • Model updates since my lastgroup meeting (Dec 2006)

4. Bob Y – Jacob Group Meeting – Page 4 Review: state of the model at last grp mtg • Internal version v7–04–11 was the development version in Dec 2006 • Improvements and additions to chemistry • Ability to do chemistry up to the location of the actual tropopause (Philippe, Brendan) --DONE! • Added 2 tracers (SOG4, SOA4) to track the SOA prod from isoprene + OH reaction (Daven Henze) -- DONE! • HO2 uptake by aerosols is now turned off in the SMVGEAR mechanism (Bastien Sauvage) – DONE! • Improvements and additions to emissions • Option to use GFED2 biomass emissions for gas phase + aerosol species -- DONE! • Option to use David Streets' regional emissions for China and SE Asia -- DONE! • Option to use EDGAR global NOx, CO, SO2 emissions -- DONE! • Option to use BRAVO NOx, CO, SO2 emissions over Northern Mexico -- DONE! • New module for IPCC future emissions scenarios (primarily for GCAP) -- DONE! • Near-land lightning NOx emissions module – DONE (NOW REPLACED IN v7-04-13!)‏ • Updated Hg simulation • More recent Hg simulation updates still need to be standardized – READY FOR INCLUSION! • Technical updates • Support for Sun 4100 platform (with AMD/Opteron chipset) – DONE! • Removal of obsolete GEOS-1 and GEOS-STRAT met fields – DONE! • Removal of support for obsolete LINUX_IFC and LINUX_EFC compilers – DONE! • Various minor bug fixes & updates for the GCAP simulation – DONE!

5. Bob Y – Jacob Group Meeting – Page 5 Model updates since my last group meeting! • v7–04–13 is latest benchmarked internal release • TPCORE advection now works with GEOS–5 met! • Yuxuan Wang & Dan Chen are very close to a working TPCORE for GEOS–5 nested grids • v8–01–01 (public release w/ GEOS–5) is very close • A 1-year benchmark was done • A couple of things still need attention

6. Bob Y – Jacob Group Meeting – Page 6 v7–04–13 is latest benchmarked internal release • GEOS–Chem v7–04–13 is latest internal release • H2 / HD offline simulation (Jaegle group @ UW) • Major bug fix in SMVGEAR! (May & Philippe)

7. Bob Y – Jacob Group Meeting – Page 7 SMVGEAR Bug in more detail • Basic unit of SMVGEAR work = the “block” = 24 grid boxes • Each processor gets 24 G–C grid boxes to work on simultaneously • Abs & rel error tolerance checks are applied to the whole block • Internal iterations • SMVGEAR iterates over all of the grid boxes in the block until the absolute & relative error tolerance for the block as a whole is satisfied. • If SMVGEAR does not converge to a solution, it reduces the timestep adaptively and then starts re-iterating over the block once again • The problem • Some G–C users reported that SMVGEAR was giving them extremely high Ox and extremely low CO in a particular grid box • This caused the G–C simulation to die w/ an error

8. Bob Y – Jacob Group Meeting – Page 8 SMVGEAR Bug in more detail • The cause • It was possible to have a single grid box in the block be very negative, while the overall local & global error tolerance was satisfied • SMVGEAR would then reset the negative value to “zero” • The “zero” concentration value caused problems later on in places where it showed up in the denominator or in a logarithm. • The solution (Thanks May and Philippe! )‏ • If any of the 24 grid boxes in the block are negative, then: • Revert to the conc's at the end of the last successful internal iteration • We reduce the internal iteration timestep and begin again • Repeat until convergence • If it still does not converge, then stop with an error

9. Bob Y – Jacob Group Meeting – Page 9 v7–04–13 is latest benchmarked internal release • GEOS–Chem v7–04–13 is latest internal release • H2 / HD offline simulation (Jaegle group @ UW) • Major bug fix in SMVGEAR! (May & Philippe) • Improved lightning NOx emissions algorithm (Lee Murray) • Eliminate NOx emissions per path length • Scale tropics to 260 moles N/flash (cf. Randall Martin) • Eliminate top-down scaling • New OTD/LIS local redistribution (cf. Murray et al 2007 in prep) • Bug fix in variable tropopause • Cap tropopause at 200 hPa poleward of 60S and 60N

10. Bob Y – Jacob Group Meeting – Page 10 Getting TPCORE to work w/ GEOS–5 met Pressure at grid edges: Pe(IJL) = A(L) + [ B(L) * Psurf(IJ) ] Pressure at grid centers: Pc(IJL) = [ Pe(IJL) + Pe(IJL+1) ] / 2 Near the surface, the hybrid grid has a “sigma-like” terrain-following coordinate. Near the model top, the hybrid grid has fixed-pressure levels. The pressure at each level is the same for all latitudes & longitudes. Pros: Results in much smoother winds near the top of the model, better STE.

11. Bob Y – Jacob Group Meeting – Page 11 Getting TPCORE to work w/ GEOS–5 met • From the GEOS–5 file spec document (p.14) • “In the GEOS–4 eta files, one could compute the pressure on the edges by using the “ak” and “bk” values and the surface pressure; once the edge pressures were known, they could be used to compute the average pressure in the layer.” • “In GEOS–5, the full 3-dimensional pressure variables are explicitly provided at both layer centers (PLijl) and layer edges (PLEijl). As of this writing the pressures reported are on a hybrid-sigma coordinate, and could be obtained from the “ak-bk” relationship. But this may change in the future and so users should rely on the reported 3-dimensional pressures and not attempt to compute them from “ak” and “bk”.

12. Bob Y – Jacob Group Meeting – Page 12 Our first attempts to modify the TPCORE transport code for GEOS–5 were not very successful. This tagged Ox simulation indicates very little mixing even after 7 years.

13. Bob Y – Jacob Group Meeting – Page 13 As a result of a telecon between Harvard and GMAO on 10/30/07 • GMAO agreed to give us the “A’s” and “B’s” which define the GEOS-5 grid. In this way, we can use the existing TPCORE code without having to modify it. Everyone agreed that this was the fastest solution (particularly since Harvard needs to get a working GEOS-5 code ASAP for ARCTAS support). • NASA/SIVO has a newer version of the TPCORE transport code that takes mass fluxes instead of winds. However, this version is fully ESMF-ized. To use it in the current GEOS-Chem version (aka “GEOS-Chem classic”) would require more time than we have before ARCTAS. • However, the newer transport code will be useful as we create the new, ESMF-ized GEOS-Chem (for porting to MPI and eventually to the BlueGene).

14. Bob Y – Jacob Group Meeting – Page 14 TPCORE now works with GEOS–5 met fields! GMAO provided us with the A’s and B’s that are used by TPCORE to compute the pressure at altitude from the surface pressure. Ox tends to steady state after 10 years! NOTE: This Ox spinup was used to create the initial conditions for the 1-year benchmark simulations with GEOS–5.

15. Bob Y – Jacob Group Meeting – Page 15 v8–01–01 will be next public release • GEOS–Chem v8–01–01 is very close • This will be the first version that is compatible with the GEOS–5 met fields (hence the version # change). • We will also update our web site for the public release • New GEOS–Chem user manual • Wiki will be publicly visible and editable • It’s currently only internally visible at Harvard • I will speak as to some of the ongoing GEOS–5 issues in a bit ..

16. Bob Y – Jacob Group Meeting – Page 16 • Current met field availabilityfor GEOS–Chem

17. Bob Y – Jacob Group Meeting – Page 17 Current met field availability for GEOS–Chem GEOS–1 GEOS– STRAT GEOS–3 GEOS–4 GEOS–5 # of levels 20 sigma 26 sigma 48 sigma 55 hybrid 72 hybrid Native grid 2 x 2.5 2 x 2.5 1 x 1 1 x 1.25 0.5 x 0.667 Native File Format Binary Binary HDF4–EOS HDF4–EOS HDF4—EOS Regridded to 2 x 2.54 x 5 2 x 2.54 x 5 1 x 1 nested2 x 2.54 x 5 2 x 2.54 x 5 2 x 2.54 x 5 + native Temporal Coverage 1985 – 1995 1996 – 1998 2000 – 2002 1985 – present (ending in Feb 2007)‏ 2004 – onward + 30 yr reanalysis (MERRA)‏ We also have GCAP met available for several years but these are not shown on the tables below. GMAO Met Field Comparison Chart O B S O L E T E !! Pres, Fut. Past GEOS–3 and GEOS–4 are both still supported in GEOS–Chem. GEOS–4 is currently still heavily used in GEOS–Chem research. However, GMAO has now turned off the GEOS–4 data stream. GEOS–5 is now the only dataset currently being produced by GMAO.

18. Bob Y – Jacob Group Meeting – Page 18 Current met field availability for GEOS–Chem GEOS5 4x5 GEOS5 0.5 x 0.667 nested China GEOS5 2 x2.5 GEOS5 0.5 x 0.667 nested NA 2007 • Extra Slides GEOS–4 ends on 3/31/07. We have late-look data thru 2006. GEOS 4 2x25 GEOS4 4x5 2006 Met field date 2005 NOTE: This plot is now on the GEOS–Chem web page under “Simulation Years”

19. Bob Y – Jacob Group Meeting – Page 19 Current met field availability for GEOS–Chem GEOS5 4x5 GEOS4 4x5 GEOS5 China GEOS5 2 x2.5 GEOS4 2x25 GEOS5 NA 2007 2004 GEOS4 1 x 1.25 NOTE: GEOS-5 data exists for 2004, we just haven’t processed it yet GEOS 4 2x25 GEOS4 4x5 2006 2003 Met field date GEOS-3 ends 10/31/02 GEOS3 4x5 2005 GEOS3 2x25 2002

20. Bob Y – Jacob Group Meeting – Page 20 Current met field availability for GEOS–Chem GEOS5 4x5 GEOS4 2x25 GEOS4 4x5 GEOS3 2x25 GEOS3 4x5 GEOS4 4x5 GEOS5 2 x2.5 GEOS5 NA 2002 GEOS3 1x1 nested China GEOS3 1x1 nested NA GEOS4 4x5 2001 Met field date 2000 GEOS3 4x5 GEOS3 2x25 GEOS–3 begins Jan 2000(some non-std data for 1998 exists)‏ GEOS-4 continues back to 1985

21. Bob Y – Jacob Group Meeting – Page 21 GEOS‒5 has more levels in the PBL Vertical levels up to 4 km – PBL GEOS–3 GEOS–5 GEOS–4 GEOS–5 has much finer resolution in the PBL!

22. Bob Y – Jacob Group Meeting – Page 22 GEOS–5 also has more levels in the lower FT Vertical levels – up to 30 km GEOS–4 GEOS–3 GEOS–5

23. Bob Y – Jacob Group Meeting – Page 23 • GEOS–Chembenchmarking updates

24. Bob Y – Jacob Group Meeting – Page 24 Upshot: We have to keep track of what changes go into G–C.For changes that affect the full-chemistry simulation, we need to perform 1-year benchmark simulations. We didn’t do such a good job of that in the recent past. Example: Cold Bay, Alaska CO (ppbv)‏ cf. Rynda Hudman Note spread of different G-C versions w/r/t the observations! Even updates that seem “minor” at first glance can have a major impact on GEOS–Chem results! 20 ppbv

25. Bob Y – Jacob Group Meeting – Page 25 GEOS–Chem Version History (1-yr benchmarks) Slide from Rynda HudmanChanges are w/r/t the previous version

26. Bob Y – Jacob Group Meeting – Page 26 Updates for GEOS–Chem benchmarking • A hard object lesson was learned: • We cannot rush revisions into the code quickly and then “pray to God” that it will all work the first time thru … • We now must be more diligent in benchmarking G–C • Especially for versions that contain revisions which could have a major impact on the full-chemistry simulation • We ask for your understanding • “It’ll be ready when it’s ready …” • We know that delays are sometimes frustrating • Sometimes we encounter “open-ended” bugs or problems that are hard to solve quickly (cf. SMVGEAR bug fix in v7–04–13)‏ • However, It’s better to take more time and ensure the validity of the G–C version than to find out a major flaw with it later on

27. Bob Y – Jacob Group Meeting – Page 27 Updates for GEOS–Chem benchmarking • 1-month benchmarks will be done for each internal & public version (GEOS–4 and GEOS–5)‏ • When do we do a new 1-yr benchmark simulation? • For updates to emissions • Anthro, biofuels, biomass, lightning, etc. • For changes to chemical mechanism • Reaction rates • Photolysis • For dynamical changes • New met fields (e.g. GEOS–4 vs. GEOS–5) • New convection and/or advection schemes

28. Bob Y – Jacob Group Meeting – Page 28 Updates for GEOS–Chem benchmarking • IDL codes for creating plots from 1-month and 1-year benchmarks have been overhauled (L. Zhang, Bob Y.)‏ • 1-yr benchmarks now include the following plots(3 models vs. observations)‏ • Profiles of differences along 15S and 42N • Difference maps at surface and 500 hPa • Seasonal cycle • Model ozone profiles vs. sondes • Seasonal cycles of model ozone and CO vs. observations • Model CO vs CMDL and ship track observations • Ozone & CO vs. MOZAIC observations • Ozone & CO vs. observations from various aircraft missions

29. Bob Y – Jacob Group Meeting – Page 29 Updates for GEOS–Chem benchmarking • 1-month benchmarks now include the following plots(this version vs. previous version)‏ • Frequency distribution • J-Value ratios at surface and 500 hPa • Tracer ratios at surface and 500 hPa • Tracer absolute difference at surface and 500 hPa • Profiles of differences along 15S and 42N • Tracer concentration maps • Budget • Emissions totals • Benchmark plots will be accessible from our FTP & web site

30. Bob Y – Jacob Group Meeting – Page 30 • GEOS–5 validationand outstanding issues

31. Bob Y – Jacob Group Meeting – Page 31 GEOS–5 Outstanding Issues • A 1-yr benchmark simulation was done with GEOS–Chem v8–01–01 driven with GEOS–5 met • A few areas still need attention: • Convection • GEOS–5 cloud convection is weaker than GEOS–4? • Lightning • Less LNOx produced w/ GEOS–5 • Lack of lightning over certain areas (e.g Central Asia) • Wet deposition (rainout & washout) • Differences in precip fields between GEOS–4 and GEOS–5

32. v7-04-12-Run2 (geos4);v7-04-13 (geos4);v8-01-01 (geos5);data Plot by Inna Megretskaia Note that the GEOS–5 Ox profiles are very similar to the simulations done with GEOS–4 up to 200 hPa, but diverge higher up. This can be an indication that cloud convection in GEOS–5 is weaker than in GEOS–4. 200 hPa Strong convection; wet season Dry season at these stations Bob Y – Jacob Group Meeting – Page 32

33. GEOS–4 vs. GEOS–5 cloud mass fluxes NOTE: original unit of GEOS–4 cloud mass flux is Pa/s, we have converted to kg/m2/s to compare w/ GEOS–5. No cloud mass flux below ~1 km in GEOS–5! Altitude (km) 0.8 km Bob Y – Jacob Group Meeting – Page 33

34. Bob Y – Jacob Group Meeting – Page 34 GEOS–4 puts lightning in Central Asia, but GEOS–5 doesn’t. Figure from Lee Murray GEOS–4 LNOx (CTH) GEOS–5 LNOx (CTH) We are currently looking into this! 0 1.2e–04 flash/min/km2 This is most likely due to lower cloud-top heights in GEOS–5. Global LNOx total w/ GEOS–5 met: ~ 5.4 Tg N/yr Global LNOx total w/ GEOS–4 met: ~ 6 Tg N/yr

35. This plot shows the effect that the lower lightning NOx total in GEOS–5 has on the concentrations of Ox at midlatitudes. v7-04-12-Run2 (geos4);v7-04-13 (geos4);v8-01-01 (geos5);data Ox (ppbv) Plot by Inna Megretskaia Ozone seasonal cycle at 4 stations and 3 altitudesNote: GEOS–5 (blue line) is lower than GEOS–4 (red, green lines) in summer! Bob Y – Jacob Group Meeting – Page 35

36. v7-04-12-Run2 (geos4);v7-04-13 (geos4);v8-01-01 (geos5);data Ox (ppbv) Plot by Inna Megretskaia Ozone seasonal cycle at 4 stations and 4 altitudes Bob Y – Jacob Group Meeting – Page 36

37. QQ: GEOS–Chem Precipitation Fields (in wet dep module) Also note GEOS– 5 has more conv precip over tropics! Hongyu and Bob Y. are looking at the wet deposition in G–C with Rn-Pb-Be simulations. This is a work in progress. NOTE: The GEOS–5 precipitation is much higher at about 1km than in GEOS–4!

38. Bob Y – Jacob Group Meeting – Page 38 • Adoption of ESMF & MPI • and Porting of GEOS–Chemto IBM BlueGene/L architecture

39. Bob Y – Jacob Group Meeting – Page 39 ESMF Overview Summary • What is ESMF? • It’s a software framework for earth science models that is being implemented at NASA and elsewhere • http://www.esmf.ucar.edu • What ESMF will do: • It will make it easier to connect “pieces” of the same model together • It will make it easier to take “pieces” of one model and mix them with “pieces” from other models. • It connects “pieces” of models together regardless of which language the models were written in (Fortran, C, C++, etc.)‏ • It will probably eventually reduce model development time (however, some up-front development will be needed)‏ • What ESMF will not do: • It will NOT improve the quality of the science of a model

40. Bob Y – Jacob Group Meeting – Page 40 With ESMF, you can reduce a CTM to a collection of “pluggable” black box “Components” with standard inputs & outputs ESMF library functions pass data into & out of components! Diagnostics NOTE: All of the data that flows into a component goes in through the “front door”, I.e. through argument lists. No more global common blocks! In the future, GEOS–Chem will be a configuration of ESMF-compliant components instead of a standalone model

41. Bob Y – Jacob Group Meeting – Page 41 Porting GEOS–Chem to BlueGene • Harvard/SEAS has recently purchased an IBM BlueGene/L • Several research groups in SEAS are moving to this platform • Earthquake fault & thrust modeling • Protein folding • Quantum chemistry • Advantage: provides a high-quality computational platform to several research groups • Advantage: users outside Harvard would also be able to take advantage of this platform • We were asked by the Dean of IT to consider porting GEOS–Chem to the BlueGene • Philippe & I have sat in on a couple of BlueGene workshops

42. Bob Y – Jacob Group Meeting – Page 42 Porting GEOS–Chem to BlueGene • BlueGene/L is a radically new supercomputer architecture • 4 of the Top 10 fastest computers are BlueGene’s (top500.org)‏ • LLNL & DOE use BG’s to “have a blast”, among other things  • Why BlueGene? • Need lots of computing power to solve today’s complex problems in the interdisciplinary sciences • Advantage: massively parallel machine • Think big: you have the opportunity to submit a job using hundreds or thousands of cores! (core = processor)‏ • Each small task is assigned to its own core • I/O can be done in parallel • BG architecture minimizes power consumption

43. Bob Y – Jacob Group Meeting – Page 43 IBM BlueGene/L Architecture K. Jordan, IBM Take notice! Users log in to the “front end” (which is a more traditional Unix environment) and submit jobs to the BG/L via the LoadLeveler queuing software.

44. Bob Y – Jacob Group Meeting – Page 44 IBM BlueGene/L Architecture K. Jordan, IBM NOTE: One must break down arrays into chunks that are less than 256 MB! Each core has 256 MB of memory and does not see the memory of the other cores. Cannot use OpenMP, must use MPI! Harvard SEAS has a 2-rack BG/L machine

45. Bob Y – Jacob Group Meeting – Page 45 IBM BlueGene/L Architecture Very complex design! K. Jordan, IBM

46. Bob Y – Jacob Group Meeting – Page 46 Challenges in porting GEOS–Chem to B/G • Radical rewriting of GEOS–Chem will be necessary • Cannot use much of existing G–C due to BG/L memory per node • Existing G–C requires 2+ GB of shared memory • Parallel file system implies file format will have to be switched from binary to netCDF or HDF • Cannot read binary sequential files on BlueGene • Because we will want to interchange codes w/ NASA, we will have to pass data around via calls to ESMF/MAPL libraries • Implies familiarity with ESMF programming interface • But all is not lost … • We can start with existing routines from NASA/SIVO and/or GMI • We can rely on the expertise of the folks at NASA

47. Bob Y – Jacob Group Meeting – Page 47 Challenges in porting GEOS–Chem • Technical things • ESMF and MAPL libraries must be installed on BG/L • Learning curve! Bob & Philippe will have to learn many new things, including MPI programming and ESMF & MAPL API’s • We must also communicate these new things to the G–C user base • Existing ESMF / MPI etc. documentation is not very good • Human Resources • Perhaps up to ½ of Bob's or Philippe's time will be spent on porting • More time will be spent on strictly software issues • Significant up-front work may be required before we get results

48. Bob Y – Jacob Group Meeting – Page 48 Challenges in porting GEOS–Chem • Testing concerns • We cannot move to the IBM BlueGene/L in one jump • We must port the model to a more traditional Unix environment first • On the BlueGene/L, you cannot run the model on one processor. You have to use at least 32. This makes single vs. multi-processor testing impossible on the BlueGene/L. • Scalability • At this point we don’t know how scalable a new G–C code would be on the BlueGene • If the code does not scale well on BlueGene then there is no point in porting there • Better to develop the ESMF-ized version of G–C on a more traditional computational environment first. If there are scalability problems then they will more than likely show up at this stage.

49. Bob Y – Jacob Group Meeting – Page 49 • Stuff in the pipeline

50. Bob Y – Jacob Group Meeting – Page 50 Things to be added into GEOS–Chem • List of revisions, in no particular order • Add glyoxal etc. rxns to chemistry mechanism (May Fu, Hong Kong Polytechnic U.) • Updated Hg simulation (Noelle Selin, now @ MIT) • Interface CASA model with mercury simulation (Nicole Smith-Downey, Harvard)‏ • Emissions scale factors (A. van Donkelaar, Dalhousie) • GFED2 8-day biomass emissions (P, Kasibhatla, Duke) – now in testing • Aerosol phase transition (J. Wang, U. Nebraska @ Lincoln) • Updated ozone quantum yields for photolysis from JPL • Updated EPA/NEI99 emissions w/ California transport fix (available for use now but not in std code) • CAIR (EPA 2020) ship emissions, accounting for offshore limits (P. LeSager) • For ARCTAS • Construction of near-real-time simulation (Bob Y.)