Roy M. Gabriel, Ph.D.

1. National Evaluation of the Addiction Technology Transfer Center (ATTC) Network: ATTC Event & Activity Reporting Database Review Roy M. Gabriel, Ph.D. Jeffrey R.W. Knudsen Margaret Gwaltney, M.B.A. Richard Finkbiner, Ph.D. RMC Research Corporation Abt Associates, Inc. MANILA Consulting Group Portland, OR Bethesda, MD McLean, VA Presentation to ATTC DirectorsNovember 5, 2009

2. Agenda • The ATTC Event & Activity Reporting Database: • Review of past action steps • The NO, NET, and the ATTC Network • Implementation to date • Examination of current data • All events vs. GPRA only • Role of these data in overall evaluation • Next steps???

3. Review—Where We’ve Been

4. ATTC Event & Activity Reporting Database and the NET • ATTC Event & Activity Reporting Database was augmented to meet multiple reporting needs: National Office/CSAT, NIDA, and the national evaluation. • In coordination with ATTC National Office, the NET developed a web-based query system to extract and summarize information from the database. Examples: • Query the entire Network, specific region(s), time periods, event types, etc. • Tally number of events, number of participants, etc., as ATTC “outputs” (both GPRA and non-GPRA). • Cross-tab tallies w/many other event characteristics (e.g., funding source, materials source, GPRA/non-GPRA, collaborator type, delivery mode, etc.).

5. The Underlying Premise  . . .   • During the design phase of the ATTC Evaluation contract, ATTC Directors clearly indicated that, from a service provision perspective, GPRA data did not tell the entire story. • It was clear that other very important services were being provided by ATTC centers, but were not being counted by the GPRA system.

6. The Output is Only as Good as the Input . . . Data Quality • The NET presented a series of sample summary tables at the December 2008 ATTC Directors meeting. • It was determined that assuring useful (and accurate) data queries and reports is contingent on the NET and the Network working together: • The ATTC NO and the NET hosted a 2-part webinar series in March 2009 reviewing (a) the purpose of the Event & Activity Reporting Database (from the perspectives of CSAT, NIDA, and the NET) and (b) data entry procedures. • The ATTC Performance Monitoring Subcommittee was working to (a) re-disseminate activity/event definitions and to (b) establish rules of thumb regarding what non-GPRA events should be entered into the system. • The NO provided draft guidelines for review (April 2009).

7. NET Reporting Plans—On Hold • The NET prepared templates for quarterly summaries of events/activities to each regional ATTC. • Reports were to feature regional activity data along with a Network-wide summary of the same information. • The NET had hoped such summary reports would increase data utilization within Network. • Production and dissemination were contingent upon more consistent implementation of data system. • The NET does not want to issue reports if numbers are misleading, skewed, or simply inaccurate.

8. Implementation of Event & Activity Reporting DatabaseJanuary 1–June 30, 2009

9. GPRA vs. Non-GPRA Events

10. Illustrative NET Interpretation • In a 6-month period, the Network conducted nearly 750 events and activities. • An average of about 50 events/activities per region. • Much variability in these numbers from region to region. • Nearly 40% of these are not captured by the GPRA reporting system. • “Natural” follow-up questions: • What kind of events? • Provided to whom? With whom? • On what topics?

11. Variability in Entering Non-GPRA Events • 3 Regional Centers account for 70% of all non-GPRA events entered into database. • These 3 Regional Centers demonstrated a 25% GPRA/75% non-GPRA balance.

12. NET Observations/Questions • Is it accurate that: • 3–4 regions conducted no non-GPRA events in these quarters? • The majority of regions conduct fewer than 5 non-GPRA events in a quarter? • In contrast to the earlier slide that indicated, “Network-wide,” nearly 40% of the events conducted fall in the non-GPRA category. • What is the balance between GPRA and non-GPRA events in the Network: 60%/40%? 25%/75%? Something in between?

13. A Closer Look at Data:Event TypeYR2, Q2–3

14. Event Type (n = 745)

15. Event Type—GPRA Only (n = 443)

16. Event Type—Non GPRA Only (n = 273)

17. Event Types for 3 Centers Consistently Logging Both GPRA & Non GPRA Events (n = 256)

18. A Closer Look at Data:Collaborator TypeYR2, Q2–3

19. Collaborator Type (n = 745 events)

20. Collaborator Type—GPRA Only (n = 443)

21. Collaborator Type—Non GPRA Only (n = 273)

22. Collaborator Types for 3 Centers Consistently Logging Both GPRA & Non GPRA Events (n = 256 events)

23. Many More Examples Available . . .  • Previous slides provided only as illustrative examples. • Many other queries could have also been used to demonstrate the impact of non-GPRA data, for example: • Primary funding source • Tech transfer objective • Delivery mode • Target audience • State-level service saturation

24. What This Means . . .  • The inclusion of non-GPRA data changes the overall service delivery “picture” for the Network. • This is not a revelation, nor unexpected; but nonetheless, from the NET’s perspective, very important to the Network. • The change in picture is even more apparent when looking at those few Centers who seem to be higher implementers (or reporters) of non-GPRA activities. • However, given the variability in reporting, the NET can only confidently put forth the service delivery picture drawn from GPRA. • When 70% of non-GPRA entries are coming from 20% of the Network, the climate for skew exists.

25. Importance of Event & Activity Reporting Database in National Evaluation • The NET has designed a comprehensive series of data collection activities to: • Address and assess multiple objectives of tech transfer in the Network. • Obtain a variety of perspectives on the effectiveness and value of the ATTC Network. • Across all of these data collection activities, the database is the only one that can capture the sheer volume of work produced by the Network (and volume is valued by many key stakeholders).

26. Next Steps • The NET will run all developed queries on GPRA events entered into ATTC Event & Activity Reporting Database to learn more about planning processes, partners, funding, materials, objectives, etc. of ATTC provided services. • If the NET can determine that entry of non-GPRA events/activities has improved, pool of events targeted for inquiry will be expanded.