Calculating Statistics: Concentration Related Performance Goals

1. Calculating Statistics: Concentration Related Performance Goals James W. Boylan Georgia Department of Natural Resources PM Model Performance Workshop Chapel Hill, NC February 11, 2004

2. Outline • Performance Statistic • Standard Bias and Error Calculations • Model Performance Goals for PM • Speciated Bias and Error Goals • Relative Proportions Goals

4. Example North Carolina 77 Georgia Tech 88 GT showed a positive bias of 11 points NB = 14.3% FB = 13.3%

5. Performance Metrics • Mean Normalized Bias and Error • Usually associated with observation-based minimum threshold • Some components of PM can be very small making it difficult to set a reasonable minimum threshold value without excluding a majority of the data points • Without a minimum threshold, very large normalized biases and errors can result when observations are close to zero even though the absolute biases and errors are very small • A few data points can dominate the metric • Overestimations are weighted more than equivalent underestimations

6. Performance Metrics • Normalized Mean Bias and Error • Biased towards overestimations • Mean Fractional Bias and Error • Bounds maximum bias and error • Gives additional weight to underestimations and less weight to overestimations

7. Example Calculations • Mean Normalized Bias and Error • Most biased and least useful of the three metrics • Normalized Mean Bias and Error • Mean Fractional Bias and Error • Least biased and most useful of the three metrics

8. SAMI Model Performance Summary

9. Proposed Performance Goals • Based on Mean Fractional Error (MFE) and Mean Fractional Bias (MFB) calculations • Performance goals should vary as a function of species concentrations • More abundant species should have a MFE +50% and MFB ±30% • Less abundant species should have less stringent performance goals • Goals should be continuous functions with the features of: • Asymptotically approaching +50% MFE and ±30% MFB when the concentrations (mean of the observed and modeled concentrations) are greater than 2.5 mg/m3 • Approaching +200% MFE and ±200% MFB when the concentrations (mean of the observed and modeled concentrations) are extremely small

10. Proposed Mean Fractional Error and Bias Goals

11. Example Calculations Average CO + CM = 0.5*(1.125 + 1.225) = 1.175 MFE performance goal for “Species X” = 81.3% MFB performance goal for “Species X” = ±46.2%

12. Mean Fractional Error Goal

13. Mean Fractional Bias Goal

14. SAMI – 6 Episodes

15. SAMI – 6 Episodes

16. VISTAS – July 1999 Episode

17. VISTAS – July 1999 Episode

18. VISTAS – January 2002 Episode

19. VISTAS – January 2002 Episode

20. æ ö æ ö C C N N 1 1 å å ç ÷ ç ÷ m o = - £ ± 0.2 RP% (5%) Bias component component ç ÷ ç ÷ N C N Co è ø è ø = = 1 1 i i m Total Total Relative Proportions (RP) PERF Goals • EPA draft guidance (2001) • “For major components (i.e., those observed to comprise at least 30% of measured PM2.5), we propose that the relative proportion predicted for each component averaged over modeled days with monitored data agrees within about 20% of the averaged observed proportion. For minor observed components of PM, we suggest a goal that the observed and modeled absolute proportion of each minor component agree within 5%.”

21. Example Calculation • Calculating component proportions based on concentrations averaged over multiple days can hide poor model performance

22. Relative Proportions for SAMI Observed Simulated

23. Relative Proportions for SAMI

24. Proposed Relative Proportions Performance Goals • Propose to use an equation that accounts for the day-to-day variability of species relative proportions: RP  30%, Error 10% RP  15%, Error 5% RP 15% - 30%, Error[RP]/3

25. Proposed Relative Proportions Performance Goals

26. Concluding Remarks • Recommended performance values are model goals, not model criteria • Failure to meet proposed performance goals should not prohibit the modeling from being used for regulatory purposes • Help identify areas that can be improved upon in future modeling • If performing episodic modeling, performance evaluation should be done on an episode-by-episode basis • If performing annual modeling, performance evaluation should be done on a month-by-month basis

27. Concluding Remarks (cont.) • As models mature, performance goals can be made more restrictive by simply: • Adjusting the coefficients in the MFE and MFB goal equations • Lowering the relative proportion error goals • Q: Is there a need for performance goals for gaseous precursors or wet deposition species? • “One-atmosphere” modeling system • If not, still should be evaluated to help identify potential problems with PM model performance

28. Questions?