MSPF Evaluation Meeting

1. MSPF Evaluation Meeting Wednesday, September 18, 2013 at The University of Maryland School of Pharmacy

2. Objectives • To clarify and discuss MSPF reporting requirements • To present and discuss examples of data collection implementation plans and data tracking tools

3. Agenda • Reporting Requirements for Strategic Planning • Reporting for the Evaluation Phase • Data Management • Monthly Reports • CLI

4. Reporting Requirements for the Strategic Planning Phase

5. Implementation Plan • Use strategy protocols to create action steps in the Strategic Plan Part II • Protocols are available on the MSPF website at: http://adaa.dhmh.maryland.gov/mspf/SitePages/Implementation.aspx • If you find other protocols you like, please let us know and we can add them to the website

6. Implementation Plan • Strategic Plan Part II annual submissions • The Worksheet #1 Local Implementation Plan table should be updated with new action steps for the coming year. • Changes to the Strategic Plan should be discussed in narrative form in the Strategic Planning Summary.

7. IRB Requirement • An IRB abstract summary is required by the IRB at DHMH • Prevention Coordinators and Health Officers were notified of this in June/July 2012. • Should be completed while working on Strategic Planning Part II and before implementation of strategies. • Upon IRB approval strategies can be implemented

8. IRB Process

9. IRB Lessons Learned • Complete initial abstract summary with just secondary data information. • We can amend your summary later on for any primary data collection needs • If collecting primary data and working with another agency or local entity have their approval before submitting to IRB • We will walk you through this process

10. Reports for the Evaluation Phase

11. Evaluation Plan • An evaluation plan template is available on the MSPF website: http://adaa.dhmh.maryland.gov/mspf/SitePages/Evaluation.aspx • We recommend that an evaluation plan be completed in conjunction with the creation of the implementation plan

12. Evaluation Plan Discussion • How helpful was the evaluation plan when writing your evaluation report?

13. Evaluation Reports - Content 3. Strategy(s) Selected- For each strategy address the following: • Describe the Strategy • What population(s) received this strategy? Describe the intended target population for each selected strategy (i.e., parents, youth, servers…) • How do you know this strategy addressed your prioritized contributing factor (evidence based)? • Briefly describe how the strategy was implemented. Summarize the action steps from your implementation plan. Please include the dosage of the strategy (i.e., how many compliance checks total? How many per month? Percentage of stores targets?) Reference any protocols that you are implementing. • Describe the level of effort (who got how much of what?) What resources did you plan to use for the strategy? (Monetary resources, human resources…)

14. Evaluation Reports - Content 4. Process Evaluation: For each strategy that your community has selected, please address the following 4b. Results for Process Evaluation This section of the reports should summarize the process evaluation questions listed in the evaluation plan a. As applicable report on the following: • Demographics of population served- Did the strategy target and reach who it was intended to? • Was implementation plan followed? How well? Any Challenges? Any changes to the implementation plan? Did all the action steps from the implementation plan occur? Were they all completed as planned? If no, why? (Fidelity of implementation of strategy) • i.e., check as many stores as planned, train the target number of officers, retailers… • Number of participants served Dependent on strategy selected (i.e., entire community, #servers trained…) • Was Participant’s satisfaction measured? What was their satisfaction with the program/strategy? Strategies examples that this section is applicable: RBS training, Law enforcement training… • Staff satisfaction with the program/strategy It is important to monitor the staff who are implementing the strategy (i.e., police, planning committees…). Did you use satisfaction surveys or interviews? Was the protocol adjusted based on feedback? • What resources were used to implement the strategies? Was this sufficient? Explain whether the resources were used as planned?

15. Evaluation Reports - Content 8. Conclusion a. What are the coalition’s overall successes and challenges? Successes Examples: changes in community awareness/readiness, policy changes, involvement by key community leaders, increased membership… Challenges Examples: Discuss barriers to implementing the strategies selected as planned. Were there enough resources to effectively implement each strategy? Were there other outside factors that influenced the work of the coalition? b. What are your plans to address the challenges you identified? Discuss any changes to your implementation plan. Explain if your coalition wants to drop/add strategies c. How will your work on your selected priority(s) be sustained in the community? d. What key elements of the MSPF will you be able to sustain? (i.e. staff, coalition, strategies, policies, etc.)

16. Evaluation Reports – Formatting • Outline posted on the MSPF website http://adaa.dhmh.maryland.gov/mspf/SitePages/Evaluation.aspx • Follow the outline • Include a cover page with county name • Use headings and subheadings for clarity • Insert page numbers • Do not refer to other sections – please repeat information as needed

17. Evaluation Reports – Lessons Learned • Address all items – when an item is not applicable, indicate why • If strategies or action steps weren’t implemented as planned indicate this in section 8 and address why • Be specific and detailed when addressing the data that have been collected • Describe your tracking tools (e.g. word document, excel spreadsheet, etc.)

18. Data Management

19. Data Collection Plan • A data collection plan details the steps necessary to effectively gather the data of interest with fidelity • Can be used for survey administration, law enforcement data, court watch procedures, focus group interviews or other collection activities

20. Data Collection Plan • Marge Rosensweig – Local Evaluator • Baltimore County • Calvert County • Howard County • St. Mary’s County

21. Data Tracking Tools • Data tracking tools are any type of program or document that is used to keep track of the data collected and to organize it for future analysis • Excel spreadsheets, Word documents, Access data sets, and specific data tracking software are all types of data tracking tools

22. Data Tracking Tools Law Enforcement Tracking - Excel • Tim Kerns – Local Evaluator • Cecil County

23. Data Tracking Tools Multiple Variable Tracking Tables - Word • Erin Artigiani – Local Evaluator • Carroll County • Washington County

24. Data Tracking Tools REACH Software • Linda Walls – Local Evaluator • Caroline County • Dorchester County • Kent County • Queen Anne’s County • Somerset County • Talbot County • Worcester County

25. Monthly Reports

26. Monthly Reports • Submit reports by the 5th of each month • Report sections are designed to help with CLI reporting

27. Monthly Reports • Question 2: Coalition Activities • Include information on coalition meetings, TA trainings, and state site visits • All other coalition activities should be listed in Question 4 • Question 4: Capacity Building • List all awareness activity about alcohol or coalition capacity building in the community • Include meetings and other activities that took place to build or strengthen relationships in the community • Implementation activities should NOT be included in this section!

28. Monthly Reports • Question 6: Implementation • Include all implementation action steps/activities completed in the past month per strategy • Include any measurable outcome updates • Questions 8, 9 & 10: Successes, Problems & Technical Assistance Needs • Complete these sections, as they help us identify common problems or technical needs to which we can respond

29. Community Level Instrument (CLI)

30. CLI Part I • Reminder: • CLI Part I Reporting Period: October 1, 2012-September 30, 2013 • Forms can be completed throughout the reporting period • Submissions due to the state evaluation team the latest by October 15th

31. CLI Part I • Part I collects data about the progress of the subrecipient community

32. CLI Part I • Section 5D: Prevention Interventions Implementation • Review questions in this section to make sure you are collecting the data needed to complete the section. • Section asks specific questions on the environmental strategies related to policy change, enforcement, communication and the activities affecting the implementation of those strategies

33. Section 5D: Prevention Interventions Implementation Environmental strategies related to enforcement worked to implement or implemented during reporting period. CLI #94: Conducted compliance checks? CLI #95: How many compliance checks? CLI #96: Establish sobriety checkpoints? CLI #97: How many were established? CLI #98: Frequency of checkpoints during reporting period.

34. Section 5D: Prevention Interventions Implementation Activities to affect the implementation of environmental strategies related to enforcement. CLI #108: Educated law enforcement during this reporting period? CLI #109: How many law enforcement sessions conducted? CLI #110: How many law enforcement officers were educated? CLI #111: Collaborated with law enforcement? CLI #112: How many law enforcement officers were engaged? CLI #113: Conducted citizen patrols? CLI #114: How many citizen patrols were conducted? CLI #115: How many neighborhoods were patrolled?

35. CLI Part I • Question 163 • Evaluation Team will fill out this question in the MRT • Please submit your completed Question #163 handout based on the strategies being implemented (April 1-September 30) to the evaluation team by September 30th or sooner. • Question 163 needs to be completed in order to fill out the CLI Part II surveys

36. CLI Part II • Reminder: • CLI Part II Reporting Period: April 1, 2013-September 30, 2013 • Submissions due to the state evaluation team the latest by October 15th

37. CLI Part II • Part II collects information about the specific prevention interventions and strategies being implemented by the subrecipient community • Complete for each prevention strategy: • Sub-form survey • Demographic survey

38. CLI Part II • CLI Part II Sub-form survey Demonstration • https://www.pmrts.samhsa.gov/pmrts/ • Environmental Strategy Example

39. Wrap-Up • Additional Evaluation Questions • Comments

40. Resources MSPF Website: http://adaa.dhmh.maryland.gov/mspf/SitePages/Home.aspx MSPF Help & Technical Assistance – Contact: Nicole Sealfon Adam Tate nsealfon@rx.umaryland.edu atate@rx.umaryland.edu 410.706.0178 410.706.7614 IRB Help– Contact: Michelle Campbell mcamp008@umaryland.edu 410.706.1418