The CZ example of evaluating and drafting of drug strategy

1. The CZ example of evaluating and drafting of drug strategy Lucia KiššováJindřich Vobořil

2. Drug policy formulated in strategic documents 1990 – 1992 Commission of CSFR for Drugs December 1992 – Christmas memorandum to the Government 1993 – set up of the Interministerial Drug Commission 1993 – 1996 – First Drug Policy Program 1998 – 2000 – Second Drug Policy Program 2001 – 2004 – National Drug Strategy 2005 – 2009 – National Drug Strategy and 2 Action plans 2005-2006 action plan 2007-2009 action plan 2010 – 2018 – National Drug Strategy and 3 Action Plans 2010-2012 action plan 2013-2015 action plan 2016-2018 action plan

3. National strategy: More general, defines main goals and basic principles of drug policy for the covered period, drug policy intervention areas Action plans: More specific, define activities, responsible institutions and deadlines

4. Evaluation of drug policy 2005-2009 Subject of evaluation National strategy 2005-2009 Action plan 2007-2009 Time frame 05/2009 – 03/2010 Internal evaluation – National Drug Committee secretariat – body responsible for drug policy coordination Aim of the evaluation – to learn: The extend to which strategic goals where achieved (How has the drug situation changed) What were the main achievements and failures of drug policy in 5 years

5. Concept of evaluation Based on the (simplified) policy cycle model

6. Findings and recommendations What have we learnt in connection to drafting new drug strategy

7. Formal structure of strategic documents is convenient, most of the strategic goals are relevant for the future The strategy and action plan were not fully cohesive/linked-up One strategic goal had no activity that could lead (directly or indirectly) to its fulfillment Even though strategic goals were mostly focused on changing the drug situation, the majority of activities were focused on coordination and organizational framework In some areas majority of planed activities had limited potential to change the situation, so even if implemented the strategic goal could not be achieved Only small no. of activities were essential to achievestrategic goals The character and potential of planned activities defined in Action plan 2007-2009 and also the level of successful implementation corresponds with the success in achieving strategic goals and areas

8. Incidention matrix – all activities that contribute to the achievement of goals

9. Incidention matrix – activities that are essential to achieve goals

10. eg. - prevention Strategy goal – decrease the level of experimental use Not achieved: high level of experimental use – cannabis, alcohol … The consumption of illegal drugs increased and of legal drug stabilized on a high level Prevention in Action plan 2007-2009 No activity leaded directly to specific strategic goals High % (70) of activities focused on organizational-coordination frame – only 30 % of activities were focused on solving a problem Only 35 % - implemented activities in 2007-2009 and 50 % in 2005-2006 No improvement was noticed in this area (experts opinion); number of strong points from 2004 were weakened (SWOT)

11. Strategy goal –Decrease the risks of drug use Achieved: Infection diseases and other health consequences on a low level Stabilization of problem drug users Relatively stable network of drug services Harm-reduction in Action plan 2007-2009 Had the highest no. of activities leading directly to achieving most of strategic goals Highest % (65) activities, which were specifically focused on problem solving, Fulfilled/implemented 78 % of activities 2005-2006 (most from all 4 pillars), 41 % in 2007-2009

12. Therefore more attention should be paid to better link action plan and strategy – planned activities with strategic goals While planning check weather planed activities contribute to the achievement of goals and weather the activities are realistic Define limited no. of priorities for specific time period in order to concentrate attention, personal and financial resources Formulate activities clearly – unclear vague definition causes variety of interpretation, it complicated implementation and evaluation Positive correlation was find between clear formulation of activity and the level of achievement

13. Czech drug policy can rely on the professional potential of experts, availability of relevant information and on the institutional framework (network of services for drug users, institutions responsible for drug policy implementation. We should rely on this factors in implementing future drug policy The level of coordination of relevant institutions plays positive but also a negative role in drug policy implementation. The future drug policy should pay attention to strengthen the area of cooperation and coordination Negative factors that complicate the goal achievement is formality in activity fulfillment, not realistic activities defined in action plan. In the future this should be eliminated by paying attention to set realistic activities, that is possible to implement and more accurate definition of how the activity could be achieved The Czech drug policy is negatively influenced by limited financial resources, given priority to other public policies, low interest in drug field by politicians and finally high level of social tolerance to use of legal drugs and underestimation of risks related to drug use.

14. Drafting the new drug strategy and action plan

15. The process of drafting new strategic documents and the process of evaluation were conducted in parallel, involved the same people therefore the transfer of knowledge was easier The process of drafting new documents and evaluation was coordinated by National Drug Committee Secretariat Strategy: We decided to keep the strategy and its structure more-less the same According to findings from drug situation monitoring we updated strategic goals Action plan: Expert working groups were established Priorities of drug policy for the period covered by action plan (3 years) were set by the secretariat of NDC

16. Establishment of working groups Working group for each intervention drug policy areas (7 areas) 7 areas – Prevention, Treatment, Harm reduction, Law enforcement, Coordination and financing, Information, research, evaluation and International cooperation Identify all relevant stakeholders for each area Each working group had to ideally consist of: ministries, NGOs, professional organization, individual experts, university (Centre of adictology) 15-20 people per one group 60 experts all together involved in the process

17. Work of working groups Participated in the evaluation and worked with outputs of evaluation Met cca. 3x Conducted SWOT analysis of situation in relevant drug policy area Identified main problems (W) and strengths (S) – experts ranked W,S according to their importance Experts than tried to find solutions to identified main problems by method of brainstorming There was a possibility to send ideas also by e-mail after the meeting of working group There was a criterion that proposed activities should cover defined priorities of drug policy for the period of three years. Proposed solutions/ideas created the basis for defining activities for action plan Secretariat of National Drug Committee made the selection of proposed activities/ideas and drafted the new action plan.

18. The process of negotiation/commenting on action plan Draft of action plan was discussed: in the working groups that participated in the process of evaluation and drafting in the Committee of regions Committee of Department representatives (ministries, institution) Then in the National Drug Committee (members are ministers and other high level representatives of other institution) After that the draft action plan was submitted to the Government

19. Drug Policy Coordination

20. Pros and cons of this process (-) More work and more effort needed (-) More time consuming (-) Demands good negotiations and facilitation skills (+) Involves all relevant stakeholders and experts from the low level to the high level management (+) Helps to steer the discussion, exchange views and therefore helps to find commonly acceptable drug policy (+) Supports the feeling of shared responsibility for drug policy Still keep in mind: Drug policy is still a public policy Even if you try to be it as evidence based and objective as possible there will still be different points of views, individual interests of ministries or individuals, political influence

22. Thank you and Kissova.lucia@vlada.cz Voboril.jindrich@vlada.cz

23. Detailed information on the evaluation of CZ drug strategy and action plan Background information for BC participants

24. National Drug Strategy 2005-2009 Specific goals: I. Stabilize / decrease the no. of problem drug users II. Stop the increase of experimental drug use III. Stabilize / decrease drug consumption IV. Decrease the risks of drug use Guarantee availability of drug services VI. Decrease the accessibility of drugs Technical-organizational goals: VII. Effective financing VIII. Coordination IX. Information to general public X. International cooperation XI. Evaluation of activities Main goals • Decrease the level of any drug use prevalence and the potential risks • To combat organized crime

25. Action plan 2007-2009 7 drug areas: Prevention Treatment Harm reduction Law enforcement Action plan 2007-2009 had 43 objectives and 170 activities • Coordination and financing • Information, research, evaluation • International cooperation Example of a structure of action plan

26. Aim of evaluation To learn : The extend to which strategic goals where achieved (How has the drug situation changed) What were the main achievements and failures of drug policy in 5 years Organizational aspects: internal evaluation – National Drug Committee secretariat (a body responsible for coordination and implementation of drug policy) 1,5 employee, no budget The Board of Drug Strategy Evaluation (5 experts) was established: Provided methodological guidance of the evaluation process and evaluation tools Supervision of final outcome/findings and final version of evaluation report

27. Concept of evaluation Based on the (simplified) policy cycle model

28. Stakeholders perception on the structure and formulation of evaluated drug strategy and action plan

29. We were about to draft new strategic documents so in the frame of evaluation we asked stakeholders on their perception of current strategy and action plan (We wanted to find out weather we should keep the structure for the new strategy or change it) Method used Online questionnaire Target group: Representants of relevant ministries (33 people) Regions (regional drug coordinators - 14) Members of professional organizations (8 people)

30. Questions asked in the questionnaire: Were the strategic goals realistic? Did strategic documents clearly define a way (measurements) for reaching strategic goals? Were the priorities clearly defined? Do you perceive a current action plan as an effective tool for reaching the strategic goals? How well did the action plan reflected current drug situation and trends? How useful you find for your work the drug strategy and action plan? Do you find the structure of current strategy and action plan convenient for you? Example

31. Evaluation of the process of formulation How the strategic documents were formulated

32. During the process of drafting and adopting strategic documents in 2004 there was a personal change in the position of the main actor/stakeholder responsible for drafting strategic documents. National strategy 2005-2009 was drafted by the previous stakeholder, the following action plan 2005-2007 by new one. In those two documents some inconsistencies were found – drug policy areas in the action plan and its activities did not fully corresponded to the National Strategy. With the change of the main responsible stakeholder original purpose/structure and content of action plan has been affected/changed. Therefore we decided to focus our evaluation also on the process of formulation of strategic documents.

33. Q1: Have changes occurred within the formulation process? Q2: To what extend is the Action plan and Strategy cohesive/linked-up? Q3: What is the character/type of activities planed in Action plan? Formulation Evaluation questions

34. Have strategic documents changed compared to their planed design? Methods used Comparison of official (and internal) documents – what structure and contend was planed in 2004 what was approved in 2005 Semi-structure interview with the former stakeholder Main finding There was a change in the original purpose and content of action plan Originally the action plan should contain the area Use of legal drugs, but it was abandoned – alcohol and tobacco is still not integrated part of drug policy (a failure of drug policy) There was also a change in the originally planed area Public relation for Information, research, evaluation – therefore no links existed between activities in the action plan and a specific strategic goal Formulation Q1: Have changes occurred within the formulation process?

35. For Q2 we wanted to know how many activities really lead to strategic goals. For Q3 we assessed the activities from the point of their formulation, their potential to solve a problem and their significance for drug situation Methods used: Analysis of documents based on the idea of logframe analysis – it helps to link and clarify mutual relationship among goals, objectives, outputs and activities Internal working group was set up (5 people) for this purpose – the working group assessed each activity of action plan in 5 different criteria Q2: To what extend is the Action plan and Strategy cohesive/linked up?Q3: What is the character/type of activities planed in Action plan?

36. Criteria assessed by the working group for each activity: formulation – was the activity clear and understandable? (yeas/no) context of problem solving: is the activity focused on solving a problem? (eg. Implementation of specific prevention program) or is the activity focused on creating conditions/framework for solving the problem? (e.g. Regular meetings of prevention committee) significance for drug situation: Activity with high potential to change the drug situation (eg. To adopt stricter regulations for free selling of medicine containing pseudoefedrine) Activity with partial potential to change the drug situation (eg. To publish a methodological manual for prevention) Activity with no potential to change the drug situation (eg. To realize a questionnaire survey among teachers) relevance to strategic goals: Each activity was then assessed from the point of their relevance to individual strategic goals (we had 11 strategic goals). The group assessed weather the activity was crucial for reaching the strategic goal or contributed to the strat. goal only partially or did not contribute at all. This way we found out that activities formulated in the area of prevention were rather formal, had no direct link with strategic goals and very low potential to change the situation. Example

37. Process of Implementation evaluation

38. Q1: How successful was the implementation of Action Plan 2007-2009? Q2: What factors influenced the process of implementation (positively and negatively)? Q3: What are the conditions for implementation of drug policy on the regional level? Evaluation questions

39. Methods used: Top-down approach - Objectives set on the central level are assessed from the point of – how these have been achieved, what hindered their achievement, and what is the consistency of outputs and outcomes with set objectives. A questionnaire was sent to relevant stakeholders/actors – (16 ministries and institutions directly responsible for activity implementation, 6 collaborating subjects – professional organizations, 14 regions) An internal working group of the body responsible for evaluation assessed the information submitted Q1: How successful was the implementation of Action Plan? (monitoring of implementation)

40. Structure of the questionnaire – asked for each activity (eg.): Was the activity implemented? (yeas-no-partially) Was it implemented on time? (yeas-no) Describe the process of activity implementation (how it was implemented) Describe problems in implementation/reasons for not implementing the activity Describe the outcome of activity implementation (weather the activity met its original purpose) If applicable, describe alternative outcomes of activity implementation or non intended outcomes. Provide data for defined indicators (indicators were set for each activity) If possible, provide data of financial expenditures needed for activity implementation Do you plan to continue with this activity in the future action plan? Implementation Example

41. Working group than assessed the information submitted and compared it with real outputs/objectives planed in Action plan Eg. A stakeholder informed us that data on alcohol are regularly collected and published in annual report therefore the implementation of an activity is fulfilled. However the working group found that in reality there is no such annual report and therefore marked the activity as not implemented. This way we decreased the formality in fulfillment of activities and made the evaluation more objective (eg. In the area of prevention declared implemented activities decreased from cca 45 % down to 35 %)

42. Methods used: expert groups for 7 drug areas – according to areas of action plan (cca. 60 people) (15-20 people per one expert group). Expert groups conducted SWOT analysis for each area of action plan (in 2009). Comparison of SWOT analysis conducted in 2004 and 2009 (comparison of opportunities and threads). Experts assessed weather the opportunity identified in 2004 was taken or not and weather the thread identified in 2004 occurred within 5 years or not) On-line anonymous questionnaire (35 respondents) – state organizations, local government, professional organizations – asking respondents to mark the level of influence of selected factors (positively, negatively) Implementation Q2: What factors influenced the process of implementation (positively and negatively)?

43. Example The questionnaire asked stakeholders to mark the level of influence to the implementation of factors listed bellow (-3 very negatively, +3 very positively): Formulation of activity Realistic planning Allocated financial resources Institutional framework Personal resources Available information needed for activity achievement Level of motivation of professionals and institutions in drug field Level of professional knowledge of people responsible for activity implementation Level of cooperation of relevant stakeholders/institutions Level of coordination Level of flexibility of responsible actors to act and react on new trends in drug situation Other – please mention

44. Methods used: On-line questionnaire among regional drug coordinators (14 regions) Questions focused on the area of: Extend to which goals of national drug strategy has been introduced into regional strategic documents (how many and which goals became a part of regional strategic documents) Cooperation with state organizations Position of regional drug coordinator/possibilities to influence drug policy implementation (where is the regional coordinator situated within organization structure, what other areas except drug policy is drug coordinator responsible for, weather regional drug coordinator is a member of all relevant regional coordination bodies and working groups/committees etc.) Development of conditions for implementation of regional drug policy in the last 5 years- personal, political, financial, institutional (indicate weather the situation in listed conditions worsened or got better) Implementation Q3: What are the conditions for implementation of drug policy on the regional level?

45. Outcome evaluation of CZ drug policy 2005-2009

46. Evaluation questions Q2: What was the development in each drug area in the last 5 years? Q1: What is the level of achievement of specific strategic goals? Q3: What is the level of achievement of technical-organizational strategic goals?

47. 7 drug areas – Prevention, Treatment, Harm reduction, Law enforcement, Coordination and financing, Information, research, evaluation and International cooperation Methods used: Expert working groups for each drug area (7 groups), 60 experts (15-20 experts in each working group) Each expert group assesses elements of SWOT analysis from 2004 Is the strength still strong point or has it become week point? have the week points been eliminated? have we used identified opportunities or have the threads occurred or not…? Expert groups than identified new elements of SWOT for 2009 Outcome evaluation Q1: What was the development in each drug area in the last 5 years?

48. Q2: What is the level of achievement of specific strategic goals? 6 specific strategic goals - focused on the change of drug situation: I. Stabilize / decrease the no. of problem drug users II. Stop the increase of experimental drug use III. Stabilize / decrease drug consumption IV. Decrease the risks of drug use V. Guarantee availability of drug services VI. Decrease the accessibility of drugs Methods used: Analysis of quantitative data/indicators of drug situation (data provided by NFP)

49. Example We assessed the development in the following indicators: Goal I. – no. of problem drug users and their characteristics (average age, injecting use, primary drug), no.of problem drug users in treatment (in-out-patient, substitution treatment) Goal II. – prevalence of drug use in general population, in school population, development in the institutional framework and support to primary prevention programmers Goal III. – alcohol and tobacco consumption, consumption of other illegal drugs Goal IV. – drug related death (direct/indirect), drug related infection diseases, no. of distributed needles and syringes, no.of tests for infection diseases, and other intervention of low threshold facilities. Goal V. – availability and accessibility of services for drug users, socio-economic characteristics of drug users in treatment Goal VI. – drug seizures, seizures of drug precursors (medicines containing pseudoephedrine), laboratories for pervitin production, drug related law offences, subjectively perceived availably of drugs in the marked

50. 5 Technical-organizational goals: VII. Effective financing VIII. Coordination IX. Information to general public X. International cooperation XI. Evaluation of activities Methods used: Combination of results and findings from all parts of evaluation (evaluation of formulation process, implementation and from expert working groups) Q3: What is the level of achievement of technical-organizational strategic goals?