Realistic Uncertainty Bounds for Complex Dynamic Models Andrew Packard, Michael Frenklach CTS-0113985 January 2004

1. Realistic Uncertainty Bounds for Complex Dynamic Models Andrew Packard, Michael Frenklach CTS-0113985 January 2004 • Reasoning on Collections of assertions • test for consistency • inconsistency falsifies at least one • sensitivity of consistency to data • which are likely false? • infer additional implications from assertions • sensitivity of inferred conclusions to data • which assertions have the most impact? • We have developed a formalism involving assertions expressed as polynomial inequalities on a parameter space. We use global optimization methods, developed in control systems analysis, with origins in algebraic geometry. The GRI-Mech DataSet fits in this framework. We have carried out extensive novel analysis described here. Our research focuses on the benefits of treating models/data pairs as assertions, that can be shared and reasoned with using automated algorithms. Message: Use collaboration through model/data sharing and automated reasoning to extract the totality of information in the community data sets. Michael Frenklach, Andrew Packard, Pete Seiler and Ryan Feeley, “Collaborative data processing in developing predictive models of complex reaction systems,” International Journal of Chemical Kinetics, vol. 36, issue 1, pp. 57-66, 2004. Michael Frenklach, Andy Packard and Pete Seiler, “Prediction uncertainty from models and data,” 2002 American Control Conference, pp. 4135-4140, Anchorage, Alaska, May 8-10, 2002. Pete Seiler, Michael Frenklach, Andrew Packard and Ryan Feeley, “Numerical approaches for developing predictive models,” submitted to Engineering Optimization, Kluwer December 2003. Ryan Feeley, Pete Seiler, Andy Packard and Michael Frenklach, “Consistency of a reaction data set,” in preparation. Copyright 2004, Packard, Frenklach. This work is licensed under the Creative Commons Attribution-ShareAlike License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/2.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.

2. S1(r) S2(r) Chemistry(r) Chemistry(r) Transport 1 Transport 2 S77(r) Chemistry(r) Transport 77 GRI DataSet • The GRIMech (www.me.berkeley.edu/gri_mech) DataSet is collection of 77 experimental reports, consisting of models and ``raw'' measurement data, compiled/arranged towards obtaining a complete mechanism for CH4 + 2O2→ 2H2O + CO2 capable of accurately predicting pollutant formation. The DataSet consists of: • Reaction model: 53 chemical species, 325 reactions. • Processes (Pi): 77 widely trusted, high-quality laboratory experiments, all involving methane combustion, but under different • physical manifestations, and different conditions. • Measured Data (di,ui) data and measurement uncertainty from77 peer-reviewed papers reporting above experiments. • Unknown parameters (): 102 normalized parameters, typically derived from rate constants. • Prior Information: Each normalized parameter is known to lie between -1 and 1. • Process Models: 77 1-d and 2-d numerical PDE models, coupled with the reaction model. • Surrogate (reduced) Models (Si): 77 polynomials in 102 variables. d1  u1 ProcessP1 300+ Reactions, 53 Species, and 102 “Active” Parameters CH4 + 2O2 ⇅ 2H20 + C02 ⋮ Process P77 Process P2 d2  u2 d77  u77 The prior information, models and measured data are assertions about possible parameter values. • k’th assertion associated with prior info: • Assertions associated with i’th dataset unit:

3. Consistency of GRI-Mech DataSet Is there a value of  which satisfies all of the assertions? This question constitutes a “consistency” analysis. For simplicity and in light of insufficient records of experimental uncertainties (ui) even for such a well-documented case as GRI-Mech, artificial but realistic uniform levels of experimental uncertainties were used. • At ui=0.1, the GRI-Mech Data Set is consistent, in that local optimization readily generates 102-dimensional  satisfying all assertions. In fact, this can be accomplished for ui as low as 0.087. • At ui=0.083, the data set is inconsistent, in that no exists to satisfy all assertions! How/why? Using semidefinite programming, we find nonnegative scalars i, i, and k such that the polynomial function is the negative of the sum of squares of other polynomials. Hence, the quantities inside the bracketed terms (which are asserted to be nonnegative) cannot all be made nonnegative by any one value of . This certifies that for every value of , at least one assertion is violated. What is the sensitivity of the above conclusions to the numeric values within the assertions (e.g., measured data and prior bounds on parameters)?

4. Sensitivity of DataSet Consistency to Assertions The top two panels of the accompanying figure show the sensitivity of the consistency measure to the bounds on the 102 active parameters. The numbers indicate the consistency of the dataset is almost unaffected by the prior assumptions on the unknown parameters (i.e., the -1 and 1 values for the bounds). Note units (and compare below). The consistency measure is most sensitive to two specific experimental reports (57 and 58). Panel lower-left shows the sensitivity of the consistency measure to reported lower bound of measurement value. Panel lower-right shows the sensitivity of the consistency measure to reported upper bound of measurement value. Note the units -- nonzero bars in the lower panels are 2 orders of magnitude larger than those in top panels. Large values are suggestive that some data should be reexamined… Upon notification (but no other details) that our automatic tools raised flags, the scientists involved in (57) and (58) rechecked calculations, and concluded that reporting errors had been made. Both reports were updated -- one measurement value increased, one decreased -- exactly what the consistency analysis had suggested.

5. High price of low cost, noncollaborative data processing Computational exercise: assess capability of 76 assertions in predicting the outcome of the 77th model. Quantify information “lost” by doing more conservative analysis without the benefit of collaboration at the raw data level. Select one of the 77 models, “treat” it as a process whose outcome is to be predicted, using the other 76 model/measured data pairs (assertions) as information. The prediction is an interval -- the min and max values that the predicted process's outcome could take on, inferred from the 76 assertions. Method P: Use only the prior information on parameters; gives an interval whose length is denoted P (meaning “prior”). Method C: Community “pools” prior information and 76 assertions, but for simplicity, chooses only to reduce parameter uncertainty, maintaining a cube description (i.e., intervals for each parameter -- easy to publish, share and think about). That computation requires a one-time collaboration with the assertions. The new, improved (i.e., smaller) parameter cube will allow future predictions to be more constrained. The length of the prediction interval on the 77th model using the new improved cube is denoted C (community). Method A: The prediction directly uses the raw model/data pairs from all 76 experiments, as well as the prior information. The length of the prediction interval (exactly what we want to compute) is denoted A. Repeat computation 77 times, interchanging which GRI-Mech model plays the role of the ``to-be-predicted-process'' while the remaining 76 assertions are used as the information. How much information is lost when one resorts to method C instead of A? Define the ``loss in using method C'' as No loss (LC=0) occurs if prediction by C is as tight as that achieved by A. Complete loss (LC=1) occurs if prediction by C is no better than method P (only using prior info). In such case, the experimental results are effectively wasted. The statistics of the loss variables (over 77 cases) are Method C pays a significant price for its lack of collaboration and crude representation of the information in the assertions. This illustrates the need (and payoff) for fully collaborative environments in which models and data can be shared, allowing sophisticated global optimization-based tools to reason quantitatively with the community information.