Shan Shan Li & Ling-Chu Lee Science & Technology Policy Research and Information Center

1. Skill-building Workshop Using Logical Framework to Identify Outcomes and Develop Performance Indicators in S&T Program Proposals Shan Shan Li & Ling-Chu Lee Science & Technology Policy Research and Information Center AEA annual conference 2011 2011/11/03

2. Two parts of the workshop • Introduction on the steps of the simplified LFA by explaining the case (15 mins.) • Hand-on practice (30 mins.)

3. Importance of Ex-ante evaluation • The influence of evaluation is to make governments’ resources allocation effective and efficient. • How to present the “significant” outcomes and develop the “suitable” indicators is very important for resources allocation. • The aim of a “good ex-ante evaluation” is to request the well-built framework and present the kind of logical thinking between the goals and indicators in programs’ proposals.

4. Taiwan’s situation In the 2010 Survey of Government Strategy for Technology Development finds that 31.9% of respondents believe that the objectives and indicators in government plans are not specific enough (National Science Council, 2010). Government departments are not explicitly defining objectives in their plans, or specifying suitable outputs and outcome indicators so as to reflect the return on the resources invested.

5. The simplified LFA To present the logical relationship between problems, goals/objectives, and indicators Simplified LFA Fishbone diagram SWOT analysis Taiwan’s S&T program evaluation framework (NORAD) Logical Framework Approach (LFA)

6. Objectives of the workshop 1 How to identify the problem; 2 • How to develop the problem tree and objective tree by using the concept of fishbone diagram; 3 To understand the importance of SWOT analysis in the application of simplified of logical framework; 4 How to develop the logframe matrix through the above steps; To understand how to identify outcomes and develop performance indicators. 5

7. Steps of the simplified LFA Thinking spindle: clarify problem→define the resources→develop solutions→design indicators 1 2 3 4 Problem Analysis Strategy Analysis Objective Analysis Development of LFA matrix Logical thinking of S&T programs

8. CASE ：an agency’ program of MOEA(Ministry of Economics Affaires) Thinking spindle: clarify problem→define the resources→develop solutions→design indicators *The content is only provided for the demonstration of the simplified LFA, not representing the position and implementation of Taiwan government agencies’ programs.

9. The agency’s mission • full support to the development of a first-class investment environment; • providing comprehensive assistance to business enterprises to overcome investment obstacles; • providing guidance for the development of new industries as well as the upgrading and transformation of existing industries, so as to lay a firm foundation for long-term national development. • will continue to work closely with the industry to jointly create a new chapter for Taiwan’s industrial development.

10. STEP 1Problem analysis Identification of the major problems faced by beneficiaries; and Development of a problem tree to establish causes and effects.

11. Ideal deviation Status past present Definition of the core problem Problem

12. TIPS The classification of problems in S&T programs • Necessary problems (Must) • it is not possible to measure the importance or priority of this kind of problem, its primary goal is its own and not negotiable. Therefore, in discussing this kindofproblems, resources are not used to decide strategies, but first to decide the best strategy. • Problems available to be solved with discretion (Desirable) • When some resources remain available after solving the necessary problems, such problems can be analyzed and further sorted for coming up with the one with the highest priority among them which needs to be solved. • Problems available to be solved incidentally (Feasible) • With not much effort needed to solve the problem, which isn’t necessarily related to the objective, but just like taking a free ride.

13. Problem tree To construct the relationship between cause and effect Sub problem 3 Sub problem 1 Core Problem Itemized Problem A Itemized Problem B Detail of the Itemized Problem B Sub problem 4 Sub problem 2 It is required to follow the principles of “mutual Exclusiveness” and “collective exhaustiveness”

14. Decomposition of problems—the principle of MECE • Principle of MECE in problem decomposition: The use of McKinsey & Company Minto Pyramid Principle in the principle of MECE • MECE has stressed that when facing a conceptual things or idea that needs to be split, sequence in line with structured logic should be used for the classification. • Two terms are required, namely • Mutual exclusiveness • Collective exhaustiveness

15. TIPS Reference on dimension of sub problems

16. TIPS Narrative approach of problems • The narrative of the core problem of is negative thinking. • Narrative form: adjective + noun for the sentence structure, and ought to be a negative narrative. • Every dimension of a Problems Tree is likely to become a single Objectives Tree.

17. Construction of the Problems Tree Lack of Innovation Production cost too high Low R & D investment • Increasing of labor hours, labor-intensive Requirement of a larger area Fixedproduction line • Technology-based thinking • Declining competitiveness of the traditional industries • Long product life cycle Low degree of internationalization WTOopening mechanism 1.Select the dimensions of sub problems 2.Think itemized problem 3.Narrative form: adjective + noun for the sentence structure, and ought to be a negative narrative 4.Clarify the cause-effect relationship • Lack of industrial competitiveness

18. STEP 2Strategy analysis

19. SWOT analysis The SWOT analysis provides information on positive and negative aspects both within and outside the project. Favorable to the achievement of objectives Unfavorable to the achievement of objectives Internal environment S W Strengths Weaknesses External environment O T Opportunities Threats

20. TIPS SWOT analysis 1. SWOT analysis responding to core problem 2. To make an inventory of organization or programs themselves internal subjective factors controllable by the organizations • Organization structure • Internal coordination • Financial management external objective factors not controllable by the organizations Source: NLR. (2007).

21. SWOT matrix Strategic option: First explain that the focus of the group is now changing from analysisinto action and that for effective actions we will need to have a visionabout where we wish to go. Inner Analysis Strength（S） Weakness（W） So Strategy (Max-Max) Strengthen advantages Exploit opportunities WO strategy (Min-Mix) Reduce threats, Exploit opportunities Opportunity (O) Outer Analysis Threat (T) ST Strategy (MAX-Min) Strengthen advantages Reduce threats WT Strategy (Min-Min) Lower threats, Reduce disadvantages source：Weihrich, 1982

22. TIPS SWOT matrix—strategic thinking Successful strategies build upon strengths and takes advantage ofopportunities, while it overcomes or minimizes the effects of weaknessesand threats. Inner Analysis Opportunity Threat Strength Will this strength help us to make use of this opportunity? Will this strength help us to ward off this threat? Outer Analysis Weaknesses Does this weakness prohibit us to make use of this opportunity? Does this weakness prohibit us to ward off this threat? Source: NLR. (2007).

23. SWOT Analysis

24. SWOT Matrix

25. STEP 3Objective analysis

26. Objective tree To determine the relationship of means and methods. Objective 1 Objective 3 Goal SubObjective A SubObjective B Itemed Objective Objective 4 Objective 2

27. TIPS Narrative approach of objectives • Method of objective presentation: statement structure of the Objective Tree: is almost equivalent to that of the Problems Tree, except converting the opposite viewpoints of the Problems Tree into positive objective statements. • Narrative: verb + object (person / thing / object) for sentence structure, and use a positive statement. • Statement of principles: corresponding indicators can be developed for measuring the objectives.

28. Construction of the Objectives Tree • Global division of labor • Independent • R & D • SO2 Industry-university-academic joint development Establishment of Joint Development Model • Transfer of high-tech industry experience Development led by the key industries • Innovation of Industrial production technology • Establishment of intellectual property and accounting systems Vertical Integration • Improve the competitiveness of the traditional industries • SO3 R & D subsidies provision • Enhancement of product uniqueness & differentiation Combination with design services 1.Goal and objectives are transferred from the problem tree 2.Itemized objectives come from SWOT matrix 3. Narrative: verb + object (person / thing / object) for sentence structure, and use a positive statement. Improvement of product competitiveness

29. STEP 4Development of LFA matrix

30. LFA matrix

31. TIPS Assumptions Assumptions are statements about external factors influencing the project positively or negatively, but are outside the project’s control. These project risks are to be formulated during project planning and monitored throughout implementation. Assumptions are stated as positive conditions. Source: NLR. (2007).

32. Development of LFA matrix -vertical logic + horizontal logic- 14 15 1 13 14 15 13 1 10 11 2 9 10 11 9 2 7 8 3 6 3 7 8 6 4 5 4 5 Vertical logic Horizontal Activity Resource

33. TIPS Transformation of outputs to outcomes • Outputs: quantitative items of programs • Outcomes: must consider “anticipated changes of target population” • “anticipated changes of target population”: knowledge, skills, attitudes, and behaviors • Example: Output: papers Target: researchers Outcomes: citation of papers

34. TIPS Types of indicators Output: materials or things created by project activities Direct outcomes: change of behavior or concept of a target objective due to output • Paper citation • Patent purchase • Technology application • material application • database application • Knowledge diffusion • Decision making • Tool Application • Business model • Increased investment • Industry-University Cooperation • R & D team • New Product • New Service • Paper • Technology • Patent • Teaching material • Database • Information System • Academic Activities • Technical Service • Utility method • Investigation report • analysis report • Benchmark/norm • Policy recommendation Indirect outcomes : Changes created by the diffusion, extension and integration of direct benefits Socio-economic impact: social and economic side effects arising from the proliferation of indirect benefits • Theoretical applications • Financial intermediation • Cost reduction • Industry-University mode • Market development • Talent Training • Knowledge diffusion • Competitiveness enhancement • Extended value • Service Innovation • Knowledge Fusion • Decision-making results • Value change • Improved talent quality • Increase in employment • Improved efficiency • Development of new industries • Improved quality of life • Efficient use of resources • Balanced distribution of benefits • Implementation of environmental safeguard • Improvements in global issues • Benefits of the policy implementation

35. LFA matrix 1.Goals come from goal in objective tree 2.Objectives come from objectives in objective tree 3.Outputs come from itemized objectives in objective tree 4.Assumption is the necessary condition of successful program

36. Hand-on practice The procedures of the session are: 1 • The problem identification (5 mins); 2 The development of the problem tree (5 mins); 3 • The application of SWOT analysis (5 mins); 4 • The identification of the objective tree (5 mins); 5 The development of the logframe matrix (5 mins);

37. Requirements • Open atmosphere for communication • Free talk without any rules or disturbances

38. Any issues you can choose… • The topics on aging population • health issues:These include hearing and sight loss, memory loss and increased health problems. • elderly person's income and economic welfare: No longer being in the workforce, these individuals will need to rely on their pensions and Social Security. • Loneliness:While some children are a short distance from older parents, some live too far a distance away to provide the proper support in emergencies. • ……OR you can choose any issues which happen in your life or your job

39. Template for problem tree 2 cluture difference Young children are work outside Middle class don’t have much money to pay insurance

40. Template for SWOT analysis 3

41. Template for SWOT matrix

42. Template for objective tree 4 Increase vareities of insurance Increase The salary of middle class

43. Template for LFA matrix 5 Increase The salary of middle class

44. References • NORAD (1999). The logical framework Approach: handbook for objectives-oriented planning. • Netherlands Leprosy Relief (NLR). (2007). Guidelines for Logical Framework Planning Workshops. • Thailand Bureau of the Budget. (2004). The Logical Framework Approach - a Step-by-Step Guide to Objective-Oriented Project Design. PPD Refreshers Workshop. • Y. C. Shiue. (2008). Strategy Planning and Management. Taipei: YehYeh Book Gallery.

45. Thank you for Attention! Shan Shan Li, ssli@stpi.narl.org.tw Ling-Chu Lee, lclee@stpi.narl.org.tw