Defining and Evaluating ‘Science for Sustainability’

1. Defining and Evaluating ‘Science for Sustainability’ AIRP-SD Project Paul Weaver & Leo Jansen pweaver@noos.fr jansenleo@hetnet.nl NZSSES/ICSER Conference

2. AIRP-SD Goals • Develop an approach for ex ante, concomitant and ex post evaluation of sustainability-oriented RTD programmes • Describe innovative RTD approaches • Identify good and bad practices • Assess transferability of practice • Establish programme design guidelines NZSSES/ICSER Conference

3. Toward an evaluation architecture • Lack of theory and practice suggests need to develop both through an iterative, heuristic procedure • begin with tentative hypotheses about the nature of the research processes that might support SD; test/refine these through the evaluation process itself • Need for externally-specified evaluation criteria • Depends on establishing a conceptual model of sustainability-oriented RTD NZSSES/ICSER Conference

4. AIRP-SD Procedure Identify good practice Relate outcomes to context Revise conceptual model Recommendations for research management and design Relate design to context Relate research outcomes to design Evaluate outcomes, design, context separately Describe research outcomes, designs, contexts Select case study research programmes/projects Establish conceptual model of sustainability research NZSSES/ICSER Conference

5. Derive conceptual model from: • the nature and concept of sustainable development • the nature of the socio-ecological systems of interest • the nature of SD challenges to science and society NZSSES/ICSER Conference

6. SD challenges • Orientation and ambition - paradigm change - transition - prescriptive - normative - vision-led • Complexity of socio-ecological systems of interest, indeterminacy, irreversibility, risk, precaution • Multiple stakeholders, high stakes, uncertainty, non-market values • Need to communicate with non-scientists NZSSES/ICSER Conference

7. Evaluation objects (outcomes) Contribution to strategic orientation Contribution to solution set (quantity/quality) Contribution to strategic management Outcomes Contribution to scientific capital and capacities Contribution to influencing society Contribution to social capital NZSSES/ICSER Conference

8. Evaluation objects (D&P) Management and finance model Procedure for initiating/defining programme Research model (methods, tools, procedures) Design and Process Procedure and basis for decision making Procedure for communication & dissemination Procedure for handling risk and uncertainty NZSSES/ICSER Conference

9. Evaluation objects (context) Established social capital Established RTD management capital Established scientific/research capital Context Extent & urgency of unsustainability or vulnerability Understanding & awareness of (un) sustainability across society NZSSES/ICSER Conference

10. Sustainability principles • Global and systemic approaches • Knowledge sharing and mutual learning • Broad participation • Environmental heritage • Precaution / Action to prevent risk • Resilience NZSSES/ICSER Conference

11. Sustainability principles (cont.) • Transparency and justification of decision making • Appropriate scale-issue relation • Inter-generational equity • Intra-generational equity NZSSES/ICSER Conference

12. AIRP-SD working hypotheses Hypothesis I A wide set of generic research outcomes can be identified and evaluated on a set of externally specified sustainability goals and criteria Hypothesis II Research designs and processes that conform strongly with the principles of sustainable development are likely to contribute to strong research outcomes Hypothesis III Contexts where sustainability problems or awareness is high are likely to facilitate SD research. Strong scientific/social capital in relation to SD challenges will facilitate SD research and strong outcomes NZSSES/ICSER Conference

13. Hypothesis IV Research context, design/process and outcomes are connected in a loop CONTEXT Research outcomes Research design and process NZSSES/ICSER Conference

14. Nine case studies • ALR • ATSD • CUTS • ECO-CYCLE • HABIFORUM • MAHRE • SEP • STD • ZERI NZSSES/ICSER Conference

15. Fact finding versus evaluation • Separate tasks • Data collection – factual information – What? How? Who? • Evaluation – analytical, judgmental – Whether? How well? • Evaluative questions shape the mindset and background for fact-finding NZSSES/ICSER Conference

16. Methodological issues • Restricted time-frame for analysis; some outcomes yet to emerge: interim outcomes, process-orientation • Ambiguities; e.g., financing model: detailed specification and differentiation • Subjectivity: consistent definitions, inter-subjectivity and cross-checking • Depth vs breadth: encourage evaluation to enable validation in due course NZSSES/ICSER Conference

17. Multi-level approach (outcomes) NZSSES/ICSER Conference

18. Evaluation of each object • Evaluate each outcome using SD goals • Evaluate eachD&P feature using SD principles • Evaluate each contextfeature using SD challenges • Strength (strong, weak, neutral) • Direction (positive, negative) NZSSES/ICSER Conference

19. Evaluation of interrelations • Evaluate influence of each D&P feature on outcomes • Evaluate influence of each D&P feature on each other (synergies) • Evaluate influence of context (context dependency, transferability) of each D&P feature • Evaluate any RTD induced change in context NZSSES/ICSER Conference

20. Data analysis NZSSES/ICSER Conference

21. Results: good practices • Initiation • stakeholder inventory/analysis • broad participation • re-conceptualisation, scope/goals • Finance • co-funding arrangements • dedicated funds for re-conceptualisation, stakeholder participation, communication • phased funding arrangements NZSSES/ICSER Conference

22. Results: good practices (cont.) • Management • internal learning (iterative, adaptive) • openness, transparency, accountability • continuous monitoring and quality assurance • re-defined set of success criteria • Research • vision-led, system-based, long-term oriented, inter/trans-disciplinary, iterative • multi-functional approaches/tools NZSSES/ICSER Conference

23. Conclusions • Importance of context • Importance of structure and process • Importance of goal setting • Importance of SD principles • Synergy among design features - need for thorough integration • Feedback - practice makes perfect NZSSES/ICSER Conference

24. Defining and Evaluating ‘Science for Sustainability’ AIRP-SD Project Paul Weaver & Leo Jansen pweaver@noos.fr jansenleo@hetnet.nl NZSSES/ICSER Conference