Using FLUXNET data to evaluate land surface models

1. Using FLUXNET data to evaluate land surface models Ray Leuning and Gab Abramowitz 4 – 6 June 2008

2. Land surface model evaluation framework Reto Stockli’s ‘Model farm’ CSIRO.Using FLUXNET data to evaluate land surface models

3. Schematic diagram of model components from asystems perspective • system boundary, B • inputs, u • initial states, x0 • parameters, θ • model structure, M • model states, x • outputs, y Errors in each component affects model performance Liu, Y. Q. and Gupta, H. V. (2007). Uncertainty in Hydrologic Modeling: Toward an Integrated Data Assimilation Framework. Water Resources Research 43, W07401, doi:10.1029/2006/WR005756. CSIRO.Using FLUXNET data to evaluate land surface models

4. Parameter estimation Multiple objective functions possible CSIRO.Using FLUXNET data to evaluate land surface models

5. Parameter estimation Multiple criteria possible, e.g. λE, NEE The dark line between the two criteria’s minima, α and β, represents the Pareto set CSIRO.Using FLUXNET data to evaluate land surface models

6. Comparing RMSE of models of varying complexity across sites after parameter optimization λE Sites Ideal result (0,0) Models H Hogue, T. S., Bastidas, L. A., Gupta, H. V., and Sorooshian, S. (2006). Evaluating Model Performance and Parameter Behavior for Varying Levels of Land Surface Model Complexity. Water Resources Research 42, W08430, doi:10.1029/2005WR004440. CSIRO.Using FLUXNET data to evaluate land surface models

7. SOLO neural network - cluster analysis Abramowitz, G., Gupta, H., Pitman, A., Wang, Y.P., Leuning, R. and Cleugh, H.A. (2006). Neural Error Regression Diagnosis (NERD): A tool for model bias identification and prognostic data assimilation. Journal of Hydrometeorology, 7:160-177. CSIRO.Using FLUXNET data to evaluate land surface models

8. observed cable solo Poor model performance not just due to poor parameter estimation CABLE with 4 different parameter sets SOLO – cluster analysis CSIRO.Using FLUXNET data to evaluate land surface models

9. No model or single performance measure is best for all fluxes CABLE, ORCHIDEE, CLM, MLR multiple linear regression, ANN artificial neural network CSIRO.Using FLUXNET data to evaluate land surface models

10. Model comparisons - average seasonal cycle NEE λE Global default parameters for each PFT used H CSIRO.Using FLUXNET data to evaluate land surface models

11. Model comparisons - average daily cycle NEE λE Global default parameters for each PFT used H CSIRO.Using FLUXNET data to evaluate land surface models

12. PDF’s for NEE, λE & H across 6 sites CSIRO.Using FLUXNET data to evaluate land surface models

13. NEE Perturbed-parameter ensemble simulations Average diurnal cycle Monthly averages CSIRO.Using FLUXNET data to evaluate land surface models

14. λE Perturbed-parameter ensemble simulations Average diurnal cycle Monthly averages CSIRO.Using FLUXNET data to evaluate land surface models

15. H Perturbed-parameter ensemble simulations Average diurnal cycle Monthly averages CSIRO.Using FLUXNET data to evaluate land surface models

16. Partitioning climate space into 9 SOM nodes S↓ Tair qair S↓ Tair qair S↓ Tair qair night night night CSIRO.Using FLUXNET data to evaluate land surface models

17. NEE PDFs at nodes 7 -9 at Tumbarumba S↓ Tair qair S↓ Tair qair S↓ Tair qair 9 8 7 night CSIRO.Using FLUXNET data to evaluate land surface models

18. Suggested set of discussion topics • Primary objectives • Establish a framework that provides standardised data sets and an agreed set of analytical tools for LSM evaluation • Analytical tools should provide a wide range of diagnostic information about LSM performance • Datasets specifically formatted for LSM execution and evaluation • Specific objectives • To detect and eliminate systematic biases in several LSMs in current use • To obtain optimal parameter values for LSMs after biases have been diminished or eliminated • To evaluate the correlation between key model parameters and bioclimatic space CSIRO.Using FLUXNET data to evaluate land surface models

19. Tasks for meeting 1 • Discuss what form the LSM evaluation framework should take • PILPS style? • What will be asked of data providers? • What will be asked of LS modellers? • Agree on a minimal set of LSM flux performance measures (model vs observations vs benchmark): • Average diurnal cycle? • Average annual cycle (monthly means)? • Some type of frequency analysis (wavelet, power spectrum etc)? • Conditional analysis (SOM node analysis): • Overlap of pdfs • Multiple criteria cost function set (mean, rmse, rsq, regression gradient and intercept) • Discuss other LSM outputs and datasets useful for process evaluation • Discuss ways to include parameter uncertainty in LSM evaluation (c.f. Abramowitz et al., 2008) CSIRO.Using FLUXNET data to evaluate land surface models

20. Tasks for meeting 2 • Discuss options for the most effective way to provide these services • Will individual groups do benchmarking, evaluation of model states? • Preference for an automated web-based interface and data server • Automatic processing through a website? • Abramowitz suggests automation of basic LSM performance measure plots, including benchmarking (as in Abramowitz, 2005). • Uploaded output from LSM runs in ALMA format netcdf could return standard plots to the user and/or post on website. • Model detective work and improvement to be done by individual groups CSIRO.Using FLUXNET data to evaluate land surface models

21. Data analysis will use: • Several current LSMs • Quality controlled Fluxnet datasets • SOFM (Self-organizing feature maps) analysis • to classify bioclimatic data into n2 nodes • to evaluate model biases for each node to help the ‘detective work’ of identifying areas of model weaknesses • to identify upper-boundary surfaces for stocks of C and N and P in global ecosystems as a function of the n2 climate nodes • Benchmarking • to compare model predictions at each climate node against multiple linear regression (MLR) estimates CSIRO.Using FLUXNET data to evaluate land surface models

22. Tools currently available from Abramowitz • SOLO (SOFM + MLR) software (Fortran) • LSMs ‘Model Farm’ of Reto Stöckli plus CABLE • CSV to ALMA netcdf conversion routine (Fortran) • Plotting routines in R • Fluxnet database in CSV and netcdf formats CSIRO.Using FLUXNET data to evaluate land surface models