Model Predictive Uncertainty: Why, How, and Three Case Studies

1. Model Predictive Uncertainty: Why, How, and Three Case Studies Latif Kalin, Ph.D. School of Forestry and Wildlife Sciences, Auburn University Auburn, AL 2007 ALABAMA WATER RESOURCES CONFERENCE and ALABAMA SECTION OF AWRA SYMPOSIUM Perdido Beach Resort, Orange Beach, Alabama September 5 - 7, 2007

2. Motivation • Hydrologic and water quality models are extensively used • by water resources planners, water quality managers, engineers and scientists • to make future predictions, find answers to what if scenarios and to evaluate the effectiveness of various control strategies. • Models involve many assumptions made by • model creators who develop the relationships and define the processes • model programmers who carry the model into computer platforms • Model users who generate/gather input data • Models are only approximate representations of the complex natural processes and therefore, in real world problems, uncertainties are unavoidable and should be rigorously addressed in the development and application of models.

3. Motivation • Estimating model predictive uncertainty is imperative to informed environmental decision making and management of water resources • In a state-of-the-science review of the role of research in confronting the nations’ water problems, NRC (2004) called for the explicit recognition of uncertainty occurrence, measuring its importance, and incorporating it into decision making. • Estimating model predictive uncertainty provides environmental managers a basis for selecting among alternative actions and for deciding whether or not additional experimental/field data are needed (Reckhow, 1994).

4. What is Uncertainty? • “The lack of certainty, A state of having limited knowledge where it is impossible to exactly describe existing state or future outcome, more than one possible outcome” • Aleatory (stochastic) uncertainty - data based • Associated with inherent variability • Irreducible • Best represented by probability dist. • Epistemic (imprecision) uncertainty - knowledge based • Imperfect models (knowledge) of real world • Can be reduced (improved models/experiments) • Best represented by intervals • Transboundary uncertainty (communicating information from models to decision makers or other stakeholders) • Decision uncertainty (ambiguities in quantifying social values) • Linguistic uncertainty (vagueness of communicating information)

5. Model Predictive Uncertainty • Analysis of Variance (First/second order analysis): Mean and variance of output expressed in terms of means and variances of input random variables • Sampling-based methods (Monte Carlo - MC): Distribution of model output(s) are obtained by sampling model parameters from prioriprobability distributions derived from literature or new knowledge gained from experience and model calibration • Bayesian uncertainty estimation: recasts a deterministic model into a standard regression form and conducts model simulations based on Bayesian statistics to estimate uncertainties assuming zero-mean and normally-distributed residual errors • Bayesian MC: Combines MC simulations with observations in a Bayesian framework • Generalized Likelihood Uncertainty Estimator (GLUE) • Markov Chain Sampling • Pareto optimality: Similar to GLUE. Inherently deterministic and multi-objective in nature. • Stochastic analysis of model residuals

6. IA N W - 2 Case Study -1 (MC based) • Modeling Runoff and Sediment Yield Uncertainty • Treynor watershed: USDA operated (corn) • KINEROS-2 model • Monte Carlo sampling and simulations i (cm/hr)

7. Case Study -1 (con’t)

8. Case Study -1 (con’t) • KINEROS-2 can be calibrated with soil parameters values consistent with national statistical soil data • Comparison of medians from MC simulations and simulations by direct substitution of average parameters with observed flow rates and sediment discharges indicates that KINEROS2 can be applied to un-gagged watersheds and still produce runoff and sediment yield predictions within order of magnitude of accuracy • Model predictive uncertainty measured by the coefficient of variation decreased with rainfall intensity, thus, implying improved model reliability for larger rainfall events • Physically-based models can be used in ungauged watersheds but not empirical conceptual models

9. Case Study -2 (stochastic analy.) • Hydrologic Modeling of Pocono Creek Watershed • Uncertainty in forecasting.. • Time series model (ARIMA) for t = ot - pt

10. Case Study -2 (con’t) Example: with 95% confidence, 100-day flow, Q(Tr=100-day) 11.5<Q<15 m3/s

11. Case Study -2 (con’t) • The seminonparametric model offers the added advantage of relaxing the normality requirement for the random noise as a condition for the application of the relatively simple time series models. • Ensemble of streamflows generated through Latin-Hypercube Monte Carlo simulations showed that long-term annual maximum daily flows (relevant to storm runoff management) had higher uncertainty than long-term daily, monthly median of daily flows (ecologically relevant metric) • Simulated ensemble of flow duration curves showed that low flows had higher uncertainty than flows in the medium and high range

12. Case Study-3 • Physically-based model of Hantush (2007) • Applied to Chesapeake Bay data in Di Toro (2001) • Generalized Likelihood Uncertainty Estimator (GLUE) • Initial parameter distributions from literature • P’(Qi) = ci * P(Qi) * L(Qi) • SOD, ammonia and nitrate fluxes Hantush, M.M. (2007). “Modeling nitrogen-carbon cycling and oxygen consumption in bottom sediments.” Adv. Water Resour. 30, 59-79 Di Toro, D.M. (2001). Sediment Flux Modeling. John Wiley, New York.

13. Case Study-3 (con’t)

14. Case Study-3 (con’t) • The significant number of observations positioned outside the 90% confidence bands is an indication of either or combinations of inadequate prior parameter distributions, sparse measurements, measurement errors, and inadequate model • Results remain preliminary and a more thorough analysis is needed: • Further analysis to revise prior parameter distributions • Increase the number of behavioral parameter sets for improved posterior cumulative distributions of the model outputs • Better inferences of the distribution of depositional flux of organic matter and refined relationship for the estimation of the thickness of the aerobic layer, may lead to more realistic predictive uncertainty estimates

15. Summary & Conclusions • Importance of uncertainty analysis • Three case studies • Flow time series in an rapidly urbanizing forested watershed • Runoff and sediment yield modeling in an agricultural watershed • Sediment-nutrient flux in Chesapeake Bay • Computed uncertainties of predicted water quality and quantity attributes provide basis for communicating the risk to water resources managers and decision makers