Management of assessments and decision making: execution, facilitation, evaluation

1. Mikko V. Pohjola, Nordem Oy (THL) Management of assessments and decision making: execution, facilitation, evaluation

2. Basic problem • Lankinen, et al.: State research institutes and research funding: proposal on a comprehensive reform: • “Preparation, decision-making and implementation of policy should be based on researched information and evidence.…the use of research-based information in social policy decision-making, must be strengthened” • If we want policy to be ”knowledge-based” and ”effective”… • …what constitutes effective knowledge-based policy? • and effective knowledge support to policy making?

3. Policy Outcomes (vs. objectives) Behavioral responses + CBA Monitoring Ex ante – ex post evaluation Factors limiting implementation Context Modelling/assessment Enablers Output Process Use Outcomes Constraints Common perspectives to effectiveness • Assessment (and modeling) • Quality assurance / control • Uncertainty analysis • Technical assessment of models • ”Effectiveness” • Other

4. Concepts • Assessment – knowledge support for practical needs • Policy – decisions and actions with societal relevance • Outcomes – changes in the world resulting from decision and actions • Practices – (common) ways of dealing with things • Effectiveness – capability (or likelihood) of achieving intended outcomes • Evaluation – analysis of the capability (or likelihood) of achieving intended outcomes

5. Framework for knowledge-based policy (Knowledge) practices Policy making Assessment Question Implementation Outcomes Answer Interpretation • Process • Product • Use • Interaction • Design • Execution • Follow-up Evaluation & management

6. Framework for knowledge-based policy • Focus on knowledge support to decision making • Decision makers' needs? • Societal needs? • Other needs? • Expert knowledge • Local knowledge • Opinions, views, perspectives, values • → collaborative search for questions and answers • → good decisions, desired changes in the world • Evaluation and management of effectiveness -> proactive guidance of decisions and actions both now and in the future

7. Management of assessments and decision making Execution Evaluation Assessment Decision Outcomes Assessment Decision Outcomes Question • Explicit purposes, aims • Common practices • … • Properties of good assessment • Dimensions of openness • Assessment-policy interaction • Explicit purposes, aims • … • Monitoring • Policy analysis • … Answer Interpretation Facilitation • Open assessment • Openness • Discussion • Collaboration • Shared understanding • Information structure • …

8. Evaluation of decision support • Five ”knowledge-based” approaches to effectiveness (cf. the Puijo-case): • Dimensions of openness (DO)(Pohjola and Tuomisto 2011) • Knowledge-policy interaction (KPI)(Pohjola manuscript) • Properties of good assessment (PoGA)(Sandström et al. manuscript, http://en.opasnet.org/w/Properties_of_good_assessment) • Assessment effectiveness (Kauppinen et al. 2006) • Relational evaluation approach (Koivisto and Pohjola 2011) • Evaluation guides (the management of) execution! • Not only afterwards, but also during and even before the making of an assessment

9. Evaluation • Evaluation is possible only in relation to purpose! • Taking account of (intended) use is essential • Instrumental value of assessment • Cf. absolute value (a common science view) • Cf. Ad hoc solutions (a common practice view) • => Assessment is a tool for policy-support (a means to an end) • => A model is a tool for making assessments

10. Evaluation • Why evaluation of decisions / decision support? • Efficient use of resources? • Value of work done? • Importance/meaning of information? • Implications of information? • Actual impacts of information? • … • …because funder, customer, user, boss, peer, stakeholder etc. wants/needs to know!

11. Evaluation Assessment is business of creating understanding about reality Making right questions Providing good answers Delivering the answers (and questions) to use Dual purpose of assessments: Meeting the needs of use (decision making, policy) Striving for truth (science)

12. KPI: how well does assessment address decision needs? • Impacts • Causes • Problem owner • Target • Interaction

14. DO: how well can a decision process (or assessment) accept contributions? • Scope of participation: who are allowed to participate in the process? • Access to information: what information regarding the issue at hand is made available to participants • Timing of openness: when are participants invited or allowed to participate? • Scope of contribution: which aspects of the issue at hand participants are invited or allowed to contribute to? • Impact of contribution: to what extent are participant contributions allowed to have influence on the outcomes, i.e. how much weight is given to participant contributions?

15. PoGA: how well can an assessment influence decisions and actions? • Quality of content • Informativeness • Calibration • Coherence • Applicability • Relevance • Availability • Usability • Acceptability • Efficiency • Intra-assessment efficiency • Inter-assessment efficiencý

16. Properties of good assessment

18. Example: what makes a good hammer?

19. Example: what makes a good hammer? • How is the hammer made? By who? • What properties does the hammer have? • What do you want to do with the hammer? • How does the hammer help you do it?

20. Summary: evaluating decision support Purpose: Identify and explicate decision maker aims & decision criteria Identify and explicate other relevant needs Decision process: DO: Making use of assessments Decision support (assessment): PoGA & KPI: Serving decision needs DO: Making use of available knowledge

21. Homework 9 Look into two (other than your own) homework 3 draft assessments Characterize the assessment drafts according to Knowledge-policy interaction & Dimensions of openness Based on the characterization, evaluate the assessment drafts according to Properties of good assessment (modified) Write your development suggestions to the corresponding user pages Homework 3 deadline is Monday 4.2.! -> do your homework 9 after that!