The Science/Policy Interface in Logic‑based Evaluation of Forest Ecosystem Sustainability

1. The Science/Policy Interface in Logic‑based Evaluation of Forest Ecosystem Sustainability Keith M. Reynolds, USDA Forest Service K. Norman Johnson, Oregon State University Sean N. Gordon, Oregon State University

2. Acknowledgments • USDA Forest Service Washington Office • National Forest System, Ecosystem Management • Pacific Northwest Research Station • Social and Economic Values RD&A Program

3. Overview • Introduction • Knowledge bases and logic modeling • Analysis • Model design issues • Lessons learned • Recommendations

4. Introduction: indicators An indicator is any variable or component of the forest ecosystem … used to infer attributes of the sustainability of the resource and its utilization. Indicators should convey a ‘single meaningful message.’ This ‘single message’ is termed information. It represents an aggregate of one or more data elements with certain established relationships (Prabhu et al. 2001).

5. Introduction: criterion • A standard that a thing is judged by (Prabhu et al. 2001). • Criteria are the intermediate points to which the information provided by the indicators can be integrated and where an interpretable assessment crystallizes. • Principles [e.g., sustainability]form the final point of integration. • A criterion should be treated as a reflection of knowledge. • It can be viewed as a large‑scale selective combination … of related pieces of information.

6. Introduction: measurement endpoint • Some indicators are simple. • Their definition suggests an obvious one‑to‑one correspondence between an indicator and a metric for that indicator. • Definitions of some indicators are more complex. • They represent a synthesis of two or more data elements, which we refer to as measurement endpoints.

7. Introduction: scales of application • Purpose varies with scale (Castañeda 2001) • National and regional • Policy instruments to evaluate laws, policy, regulations • E.g., Montreal Process, NWFP, ICBEMP • Management unit • Evaluation and adjustment of management practices • E.g., CIFOR, USDA FS IMI

8. Introduction: objectives • Illustrate the value of a logic-based approach in designing a formal specification to evaluate the Montreal criteria and indicators. • Identify the roles of science and policy in this effort. • Highlight lessons learned from this process. • Suggest some general recommendations.

9. Knowledge bases • A form of meta database • A formal logical representation of how to evaluate information • Networks of interrelated topics • Mental map • Advantages • Interactive, graphic design (modularity) • Numerous & diverse topics can be analyzed within a single integrated analysis

10. Knowledge bases: forms of uncertainty • Probabilistic • Uncertainty of events • Linguistic (or lexical) • Lofti Zadeh, 1966 • Uncertainty about the definition of the event • A proposition is the smallest unit of thought to which one can assign a measure of truth • Truth value metrics • Indices that quantify the degree of support for a proposition provided by its premises

11. Knowledge bases: networks of topics = topic Concern 1 Concern 2 Ecostate A Ecostate B Etc. Ecostate C Ecostate D Data link Data link Data link Data link Data link Data

12. Knowledge bases: topics • Each typically evaluates a proposition • Attributes of topics • Name • Proposition • Truth value: a measure of support for the proposition • Documentation • Explanation, source, citations

13. Knowledge bases: evaluation Concern 1 Ecostate A Get data requirements Evaluate data Ecostate B Ecostate C Data link Data link Data link

14. Knowledge bases: fuzzy logic

15. Analysis: Montreal C&I • The Montreal specifications provide relatively clear definitions of biophysical, socioeconomic, and framework attributes requiring evaluation (WGCICSMTBF 1995) ... • But, design of evaluation procedures that allow interpretation of the Montreal C&I is one of the major technical issues that remain to be resolved (Raison et al. 2001).

16. Analysis: conceptual framework • Specified conditions or outcomes to be sustained (the indicators). • A measure for each condition or outcome. • Calculation of the level of the indicator over some time period using the selected measure. • A frame of reference for gauging sustainability. • Rules for deciding when sustainability has been achieved (sustainability check). • A monitoring program. • A formalism that supports requirements 1 to 6.

17. Analysis: logic models as design frameworks • Logic models (knowledge bases) provide a formal specification for organizing and interpreting information. • NetWeaver kb developer system • Problem represented in terms of propositions about topics of interest and their interdependencies. • Topics translated into propositions. • Fuzzy logic to accommodate lexical uncertainty.

18. Analysis: logic models as design frameworks (continued) • Need for transparency (Prabhu et al. 2001) • Models embody important policy decisions. • Models depend on value judgments and critical assumptions that need clear documentation. • Model development • Graphic representation is an effective basis for organizing discussion and for evolution of design. • Communication • Between scientists and policy makers. • With interested publics.

19. Design issues: model organization Basic organization of topics. For example, evaluation of criteria in the current prototype.

20. Design issues: model organization An alternative organization with very different emphasis on criteria.

21. Design issues: synthesis • ADD operator: arguments evaluated as limiting factors. • SUM operator: arguments contribute incrementally to evaluation and can compensate.

22. Design issues: synthesis Another example, including the OR operator.

23. Design issues: weighting • Intrinsic weights • Each topic in a NetWeaver logic model has an intrinsic weight attribute. • E.g., set weight attribute on any topic to adjust its contribution of evidence to a proposition. • Bad idea: part of specification, but not obvious. • Explicit weights • Better, but add another layer of subjectivity. • Some valid purposes, however.

24. Design issues: reference conditions • Each fuzzy node evaluates a measurement endpoint against reference conditions. • Lack of reference conditions is a basic problem for most measurement endpoints.

25. Design issues: reference conditions Implementation of a fuzzy node to evaluate measurement endpoint against reference conditions.

26. Design issues: qualitative measures Outcomes evaluated on an ordinal scale.

27. Design issues: reliability of data • Reliability of data for evaluation of Montreal C&I. • Stochastic, rather than lexical, uncertainty • Formal representation of stochastic uncertainty is problematic in the context of a logic model. • Not addressed in the current Montreal C&I prototype. • Possible solution • Adjusting topic weights with a normalized metric such as standard error of the mean. • Problems: availability, unknown error correlations

28. Design issues: precision of knowledge Sequential OR (SOR) to specify multiple alternative pathways in order of preference.

29. Lessons learned • Lexical uncertainty is an important issue in evaluation of Montreal criteria and indicators. • Many aspects of evaluating sustainability cannot be answered by science alone. • Acquiring data on sustainability is necessary but not sufficient for setting policy. • Evaluating the state of sustainability and deciding how to respond are separate but interdependent decision processes. • Evaluating sustainability is not the same as defining desired future conditions.

30. Recommendations • Assess the policy role in sustainability evaluation, and undertake a policy review of model organization and strategies for integrating sustainability information. • The clearest, and most critical, role of science is in development of reference conditions. • A major effort is needed to identify measurement endpoints for indicators of the institutional framework (criterion 7).

31. Authors • Keith M. Reynolds • USDA Forest Service, Pacific Northwest Research Stn. • kreynolds@fs.fed.us • K. Norman Johnson • Oregon State University, College of Forestry • norm.johnson@orst.edu • Sean N. Gordon • Oregon State University, College of Forestry • sean.gordon@orst.edu