1 / 41

Role of PI and Evaluator: Resources That Can Help You Manage and Evaluate Your Project

Barbara Estrada, MS GAIN Senior Clinical and Evaluation Consultant, Chestnut Health Systems. Role of PI and Evaluator: Resources That Can Help You Manage and Evaluate Your Project. Current GAIN Data: Brief overview of the current CSAT GAIN data

nayda-cobb
Download Presentation

Role of PI and Evaluator: Resources That Can Help You Manage and Evaluate Your Project

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Barbara Estrada, MS GAIN Senior Clinical and Evaluation Consultant, Chestnut Health Systems Role of PI and Evaluator: Resources That Can Help You Manage and Evaluate Your Project

  2. Current GAIN Data: Brief overview of the current CSAT GAIN data Your GAIN Data and Reports: Review of data and reports you have now Resources for Evaluators: Overview of GCC resources for Evaluators Analytic Issues: Brief discussion of some analytic issues, and possible next steps Overview

  3. Current GAIN Data

  4. CSAT GAIN-I Data (through 12/2010) This is a VERY large dataset you are part of and have access to!

  5. Followup Rates* by Program *Of those due for that followup Source: CSAT 2010 Summary Analytic Dataset

  6. Your GAIN Data and Reports

  7. GAIN Data Reports Reports in ABS used to help projects track participants who are administered GAIN assessments and treatment they have received. Treatment data and reasons for noncompletion of assessments must be entered into ABS Can view these reports anytime Treatment Log Report Engagement rate, continuing care rate, intake status, referral source LOC, intake program LOC, discharge status, discharge LOC. Followup Report Shows recruitment rate, completion rate, followup rates Use beyond grant requirements Provide feedback loop to treatment program staff and researchers Understand referral sources, discharges, transitions in treatment

  8. Treatment Report: Engagement and Continuing Care

  9. Treatment Report: Referral Source LOC

  10. Treatment Report: Discharge Status

  11. Followup Report: Recruitment

  12. Followup Report: Followup Rates

  13. Followup Report: Completion

  14. Followup Rates: Think Across Waves! Reality is somewhere in between

  15. Vertical Data File Data are organized one row per observation

  16. Horizontal Data File Data organized one row per person, all observations included

  17. Validity Concerns: GAIN Edits GAIN edit files are posted to the ftp Web site for your site each month. The data manager at your site will address and correct the edits leading to more accurate data. This is effective when newly trained individuals are using the GAIN; sites maybe collecting information from the clients incorrectly or items are not being administered. The number of GAIN edits will depend on the number of assessments you have completed and the level of experience of the interviewers.

  18. Validity Concerns: GAIN Edits High Low = 0-50 edits, Moderate = 51-100 edits, High = > 100 edits Moderate Low

  19. Using Site Profiles Profiles include: Useful to understand how your site compares to others Profiles are updated quarterly for all programs, posted on GPSS site and e-mailed to each PI Located on the GPSS Web site http://www.chestnut.org/li/GPSS/ and navigate to the ORP program section • Demographics • Substance use data • Comorbidity data • Risk data • Treatment information • Selected outcomes • Individual site graphs • Two-site comparison graphs

  20. Site Profiles: Example

  21. Site Profiles: Example

  22. What will you do with your GAIN data?

  23. GAIN Coordinating Center: Resources for Evaluators

  24. FTP Common Site ftp://data.chestnut.org/ Username: Common Password: public

  25. GAIN Evaluator Manual (GEM) A learning tool for using GAIN data, GEM provides information about our usual procedures and examples from recent presentations or publications, as well as detailed information with specific syntax. This represents an option for analysis that is pragmatic, robust, and cost-effective, but is not intended to be exhaustive or to be the only available option. Evaluators can use GEM to understand their GAIN data, plan analyses, get answers to common questions, see examples of syntax, and access key documents that make up the numerous appendices. Located at ftp://data.chestnut.org in: Evaluator Folder\GAIN EvalManual\

  26. GAIN Scales and Variables File The “GAIN Code Book” Plus This Excel spreadsheet contains a listing of the major scales and indices from the GAIN-I scale name, variable name, time period, GAIN-I items used, number of items, pages, scale type and cut points, purpose, a short description, interpretation, references, syntax, and full text items. There is one for the GAIN-Q too Located at ftp://data.chestnut.org in: Evaluator Folder\Data Summaries and Reports\

  27. Analytic Training Series Memos PDF documents were developed by Lighthouse Institute staff under a variety of grants Used to help researchers overcome specific problems associated with the collection and analysis of data generated during substance abuse treatment research They are available free for download Located at: ftp://data.chestnut.org in: Evaluator Folder\Training\

  28. Syntax and Template Files SPSS syntax and information to help export and prepare local data Create scales and indices (making a fully prepared “analytic” dataset if not a grantee) Create horizontal file for analysis (cases to vars) Other resources Located at ftp://data.chestnut.org in: Evaluator Folder\Syntax Files\

  29. Norms and Psychometrics Norms Tables Excel spreadsheet provides intake N, Mean, sd, alpha, severity group percents, followup alpha and followup interclass correlation coefficient for 3 groups of clients (adult, young adult, adolescent) It includes intake N, Mean, sd, alpha for adolescents by gender, age group, and race GAIN Preliminary Psychometrics Excel spreadsheet provides preliminary tables on the psychometrics of the GAIN including adult norms from the Early Re-Intervention (ERI-2) study for adults and the CSAT Adolescent Treatment 2006 data for adolescents Located at ftp://data.chestnut.org in: Evaluator Folder\Data Summaries and Reports\

  30. Analytic Issues

  31. Controlled Environment Clinical Considerations Etiology and circumstances Understanding use, abuse and dependence Understanding other areas that impact use Understanding mental health issues Program Evaluation Considerations Use S2x (pre-controlled environment use) not S9 (past 90 days use, abuse and dependence). Analysis can describe use very well: S2x, SFS (at baseline, 3, 6, and 12) Also use lifetime Substance Issues Index (SIIL) Substance Abuse Index (SAIL) Substance Dependence Scale (SDSL)

  32. Key Evaluation Questions—Clients Who is being served Baseline characteristics Who is not being served Collect data on those who are screened but do not receive treatment What are major subgroups Demographics, victimization, # problems, etc. Cluster analysis

  33. Key Evaluation Questions—Services Implementation and treatment fidelity Who received which services (program components) Initiation, engagement, continuing care, and completion What services were needed but not received

  34. Key Evaluation Questions—Outcomes Treatment Effect The most predominant change in outcomes appears at the 3-month wave and this is typically followed by a period of relative stability. To determine the treatment effect, the intake is compared to the average of the short-term followup waves. Treatment effect is generally calculated in a horizontal file as either (a) 3-month minus intake on any of the change scores or outcome variables or (b) as the average across followup waves minus the intake value.

  35. Key Evaluation Questions—Outcomes (cont’d.) Treatment Stability To determine treatment stability, compare the average of early followup waves with later waves. Treatment stability requires at least two followup data points on any of the change scores, one in the “early” time periods and one in the “later” time periods. More data points are welcome, but stability cannot be calculated with fewer data points. See Chapter 4 GAIN Evaluator Manual

  36. Treatment Effect and Stability

  37. Collaborating With Other Sites Discuss your idea with the PI from the site(s) you are interested in collaborating with or using data from Sign data-sharing agreements Decide on the scope of the collaboration Informal process: few sites, clear analysis plan, clear roles, etc. Formal process: many sites, sites want analytical or methodological assistance from Chestnut, need formal discussions/ permissions among PIs, etc.

  38. Components of an Abstract A structured abstract (maximum 3 pages) containing the following information: Title Lead author Other (potential) authors Proposed forum(s) (journal or conference) Target dates Data sources (what data set, data and/or time periods) Objectives or questions to be addressed Methods/design/main analyses Variables to be created Relevance to field

  39. Resources for Collaboration With Other Sites Multisite collaboration guidelines An overview of the annual datasets prepared by Chestnut Health Systems including their purpose, inclusion rules, and expectations Located at ftp://data.chestnut.org in: Evaluator Folder\Data Management Folder\Data Manager Required Reading\ Data request topic summary A list of other evaluators and researchers who have requested to use GAIN data, including names, contact information, abstract title, and key topics. Located at ftp://data.chestnut.org in: Evaluator Folder\Requesting and Using GAIN Data\

More Related