1 / 27

Model Performance Evaluation Data Base and Software - Application to CENRAP

Model Performance Evaluation Data Base and Software - Application to CENRAP. Betty K. Pun, Shu-Yun Chen, Kristen Lohman, Christian Seigneur PM Model Performance Workshop Chapel Hill, NC 11 February 2004. Acknowledgements.

havyn
Download Presentation

Model Performance Evaluation Data Base and Software - Application to CENRAP

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Model Performance Evaluation Data Base and Software - Application to CENRAP Betty K. Pun, Shu-Yun Chen, Kristen Lohman, Christian Seigneur PM Model Performance Workshop Chapel Hill, NC 11 February 2004

  2. Acknowledgements • Funding for the MPE software is provided by CENRAP under Modeling Umbrella Contract 02-00307-RP-005 Work Order 1; PM modeling and evaluation are funded under Work Order 3 • CENRAP Modeling Workgroup and outside reviewers provided feedback on work plan and suggestions on useful features in the MPE database and software

  3. The Role of Model Performance Evaluation Model Application Model Evaluation Regulatory Application Model/Data Improvement The modeling cycle iterates until performance is good enough for use in strategy design; hence the need to streamline and automate model performance evaluation

  4. MPE Database and Software Ambient Database Model or Preprocessor Formatted ambient data NetCDF data Model/measurement cross reference tables Model Performance Evaluation Software Processing Component User input (control file): - model - species - subdomain - temporal processing - spatial processing - statistics ... Statistics Component Data output Performance Statistics Paired, unpaired peak error Gross error, bias Normalized error, bias Root mean square error Coefficient of determination ... Graphics

  5. What we have: Many data sources (IMPROVE, CASTNet, AQS, special studies) Different formats Supporting information sometimes separate from data and sometimes difficult to find What we need: Preferably one data source Consistent format Supporting information site location sample start time including time zone sample duration units How to Consolidate, Store, and Retrieve Ambient Data for MPE? • MySQL database • upload data • (measurement, site) • compile other • relevant information • use query to • retrieve data in • consistent format

  6. Querying the MPE Database for Monitoring Sites and Observations • Site query BIBE1,IMPROVE,29.3207,-103.178 • Observation query BIBE1,2002,10,5,0,CT,24,3.619,”ug/m3”

  7. Processing Model Results • Two common formats of output files: binary and NetCDF • Platform-independent NetCDF format selected as standard • CMAQ files require no conversion • Fortran binary to NetCDF converters developed for CAMx

  8. MPE Software Processing Component • Read observations • perform time zone changes • average to longer periods • perform unit conversions • Extract modeling data • calculate grid cell corresponding to latitude/longitude • extract model data at grid cell(s) • sum components of species • perform unit conversions

  9. Cross Reference TablesCAMx4.0, PMCAMx, CMAQ • Used to look up what model species to extract based on model name and species evaluated • Selected entries

  10. Species Options • PM2.5 and PM10 mass • PM2.5 components: sulfate, nitrate, ammonium, organic material, black carbon • PM fraction • MPE software extracts relevant PM mass and species concentration to calculate PM fraction • MySQL query to calculate PM fraction at sites with co-located PM2.5 and speciated PM measurements • Gases: O3, VOC (ppbC and ppb options), NOx, NOy, SO2 • Wet deposition • Option to evaluate concentration in precipitation or deposition flux

  11. All sites included in monitoring site and data files User to provide a list in a file (e.g., a metropolitan statistical area) User to specify minimum and maximum latitude and longitude User to specify minimum and maximum cell numbers in the x and y direction An RPO performance evaluation zone (J. Underhill/D. Watson, 2003) Subdomain Options

  12. Temporal Processing Options • User can select sub-period within simulation period for evaluation • User can specify a comparison interval that is greater than or equal to the measurement interval • MPE software will temporally average both measurements and model results based on the comparison interval • A completeness criterion is needed when performing temporal averaging (>75% recommended)

  13. Spatial Processing Options • Extract model value at grid cell corresponding to site location • Linear interpolation using 4 closest grid cell • Average of all cells within a user specified window • Best estimate with a user specified window • Distance-weighing method within a user specified window

  14. Statistical Options • Threshold to be selected by user • Default output metrics • accuracy of peak (unpaired in time): paired and unpaired in space • mean observed and modeled value • Gross and normalized bias and error • Coefficient of correlation • Normalized root mean square error • Optional output metrics • ratio of means, fractional bias and error, r2, index of agreement, site specific root mean square error, normalized mean bias and error

  15. Outputs .stat header (model, period, species, units data files, options) mean observed value 39.3 mean modeled value 31.6 normalized bias -16.7% ... .tser Site I J Year Month Day CST Obs. Modeled ADPI1 82 67 2002 1 5 5 2.8041 1.3022 ADPI1 82 67 2002 1 8 5 2.5923 3.0811 … BOWA1 49 80 2002 111 6 1.0766 2.4116 BOWA1 49 80 2002 114 6 1.0805 1.0601 … Graphics package e.g., excel

  16. A Basic PM Model Evaluation set inpdir = /usr2/cp179/cmaq/jan2002/out set inpfile = PM2.5.nc foreach spc (PM2.5_Sulfate PM2.5_Nitrate PM2.5_Organic_Material \ PM2.5_Black_Carbon PM2.5) ./statp<<ieof … sitefile | /usr2/cp179/mpe/sitedata/SPECIATION.site datafile | /usr2/cp179/mpe/jan2002data/SPECIATION.$spc.200201.dat … species | $spc … ieof

  17. Model Performance against Urban and Rural AQS Data (119 Speciation Sites) CMAQ (Jan 2002)PM2.5 SO4 OM BC NO3 NH4 Mean Observed Value 12.657 2.3875 5.6490 0.6401 2.9009 1.3597 Mean Modeled Value 20.340 2.7594 2.2889 0.9324 4.7919 2.2454 Gross Bias 7.6834 0.3719 -3.3602 0.2923 1.8910 0.8857 Normalized Bias 0.7669 0.5686 -0.4210 0.7984 1.1893 1.3117 Fractional Bias 0.3862 0.1677 -0.8272 0.2595 0.3567 0.4954 Gross Error 7.4505 1.0305 3.5035 0.5421 2.6451 1.0947 Normalized Error 0.8946 0.7927 0.7068 1.0389 1.4402 1.4397 Fractional Error 0.5598 0.4370 0.8778 0.6167 0.7496 0.6756 Coef. Determination (r2) 0.2842 0.3482 0.1804 0.0789 0.2031 0.3266

  18. Subset of Sites Selected by Grid Cell Range foreach spc (PM2.5_Sulfate PM2.5_Nitrate PM2.5_Organic_Material \ PM2.5_Black_Carbon PM2.5) ./statp<<ieof … sitefile | /usr2/cp179/mpe/sitedata/IMPROVE.site datafile | /usr2/cp179/mpe/jan2002data/IMPROVE.$spc.200201.dat … species | $spc … model | CAMX … listflag | 3 … minx, miny | 2, 2 maxx, maxy | 100, 94 ... ieof Internal boundary excluded from evaluation

  19. Model Performance against Urban and Rural AQS Data (119 Speciation Sites) CAMX (Jan 2002) PM2.5 SO4 NO3 NH4 BC OM Mean Observed Value 12.657 2.3875 2.9009 1.3597 0.6401 5.6490 Mean Modeled Value 22.789 2.8278 6.2119 2.6654 0.9601 2.4260 Gross Bias 10.132 0.4402 3.3110 1.3057 0.3201 -3.2230 Normalized Bias 0.9938 0.7139 1.6849 1.6292 0.9272 -0.3595 Fractional Bias 0.5010 0.1499 0.6087 0.6476 0.3477 -0.7559 Gross Error 11.443 1.1472 3.6841 1.3976 0.5611 3.3733 Normalized Error 1.0820 0.9435 1.8384 1.6917 1.1285 0.6944 Fractional Error 0.6214 0.4413 0.8527 0.7352 0.6521 0.8132 Coef. Determination (r2) 0.2254 0.2888 0.3390 0.3766 0.0574 0.0950

  20. Comparison of Selected Performance Statistics at Speciation Sites A color box is shown under the model with better performance • Gross bias • Normalized bias • Gross error • Normalized error • r2 CMAQ CAMx Key: PM2.5 Sulfate Nitrate Ammonium Organic material Black carbon

  21. Comparison of Selected Performance Statistics at IMPROVE Sites A color box is shown under the model with better performance CMAQ CAMx • Gross bias • Normalized bias • Gross error • Normalized error • r2 Key: PM2.5 Sulfate Nitrate Organic material Black carbon

  22. Subset of Sites Selected by User Input File foreach spc (PM2.5_Sulfate PM2.5_Nitrate PM2.5_Organic_Material \ PM2.5_Black_Carbon PM2.5) ./statp<<ieof … sitefile | /usr2/cp179/mpe/sitedata/IMPROVE.site datafile | /usr2/cp179/mpe/jan2002data/IMPROVE.$spc.200201.dat … species | $spc … compintvl | 24 ... listflag | 1 listfile | BOWA.txt ... ieof BOWA.txt 1 BOWA1

  23. Time Series at Boundary Waters Canoe Area

  24. Composition of PM2.5 at Boundary Waters Canoe Area mg/m3 Observed CMAQ CAMx PM2.5 Mass = 4.0 mg/m3 Predicted PM2.5 = 5.4 mg/m3 Predicted PM2.5 = 4.8 mg/m3

  25. Ambient Variability vs. Model Variability for Sulfate and Nitrate

  26. An MPE Database and Software Designed for Community Use • Comprehensive Processor • data base; binary to NetCDF converter; software • output compatible with common graphics software • Versatility • PM, PM components, gases, deposition fluxes • User-friendly Design • CENRAP review and community input for software features • Software engineering standard • Based on Fortran 90 and MySQL (free!) • Documentation

  27. Comparison with Ambient Data Does Not Always Tell How Good Model Is • PM2.5 • Model overprediction (e.g., nitrate, other) • Sampling losses of volatile species • Organic mass • Model uncertainties in SOA formation • Factor used to convert organic carbon to organic mass • OC vs. BC: still an operational definition based on measurements • Ammonium • Model overprediction (due to nitrate overprediction) • Sampling losses on nylon filter

More Related