DEI Grantee Conference DEI Evaluation March 12, 2013

1. Social Dynamics, LLC Bringing Complex issues into Focus DEI Grantee Conference DEI Evaluation March 12, 2013 In partnership with Altarum Institute Berkeley Policy Associates Mathematica Policy Research

2. Overview of the Session • Overview of the DEI Evaluation Design • Six Main Research Questions of the DEI Evaluation • Partnership & Collaboration Objectives of the DEI Evaluation Team • DEI Evaluation Liaisons • Data Quality • 2013 Data Collection Activities • Preparing for the SSA Match for Round 1

3. The DEI Evaluation is a comprehensive implementation/process, outcome, and impact study. • Implementation: Collection of information on systems change and the strategies implemented by each of the DEI grantees (site visits/telephone interviews). • Outcome: Focus on customer-level outcomes based on administrative data (WIASRD, W-P, DEI Data System). • Impact: Focus on comparing DEI pilot sites with comparison sites (WIASRD, W-P, DEI Data System). The DEI Evaluation Design

4. The DEI Evaluation Design: Random Assignment • Random assignment ensures that the DEI pilot and comparison sites are equivalent in terms of all extraneous variables. In other words, the characteristics of LWIAs in both groups are the same. The comparison group serves to illustrate: What would have happened in an equivalent group of LWIAs in the absence of the DEI?

5. The DEI Evaluation Design (continued) • Site Visits or Telephone Interviews: Years 1 and 3 to discuss start-up and implementation issues. (Component: Implementation) • Mid-Term Telephone Interviews: Year 2 interviews with DEI State Leads to learn how grantees use selected strategies to improve their public Workforce Development Systems and better serve PWDs. (Component: Implementation) • WIASRD, Wagner-Peyser, and the DEI Data System: Customer outcome data will be sent quarterly to the Evaluation Team. (Component: Outcome & Impact)

6. What start-up issues were identified? How were these issues resolved?Sources: Site visits & telephone interviews (Component: Implementation) • What are the impacts of the DEI grants on customer utilization and disability employment outcomes such as employment, earnings, and employment retention? • Source: Administrative data—WIASRD/W-P/DEI Data System (Components: • Customer Outcomes & Impact) • 3. What services and strategies appear to be the most successful in helping customers achieve employment outcomes?Sources: Site visits, telephone interviews, & administrative data • (Component: Implementation) Six Main Research Questions

7. What are the challenges that need to be addressed to improve the employment outcomes of PWDs?Sources: Site visits & telephone interviews (Component: Implementation) • What systems changes need to be made to overcome these challenges?Sources: Site visits & telephone interviews • (Components: Implementation & System Change Outcomes) • What best practices have the pilot sites identified that address these systems changes?Sources: Site visits & telephone interviews (Component: Implementation) Six Main Research Questions (continued)

8. Partnership & Collaboration Objectives of the DEI Evaluation Team • The DEI Evaluation is a partnership between DOL, the DEI grantees, and the Evaluation Team. There are five partnership objectives: • To support the collection of pilot and comparison DEI Evaluation data in LWIAs and American Job Centers through telephone, on-site, and webinar technical assistance. • To provide ongoing support for implementing the DEI Evaluation and using results for program monitoring and improvement.

9. Partnership & Collaboration Objectives (continued) • 3. To involve grantees and stakeholders in the evaluation process and use their input to inform the analysis of findings related to program context and systems change. • 4. To provide annual presentations on DEI Evaluation results (when available) to inform local planning activities. • 5. To maintain open lines of communication throughout the program period.

10. Grantees’ Direct Link to the DEI Evaluation:State Evaluation Liaisons • Each grantee has a State Evaluation Liaison who will: • Respond promptly to DEI Evaluation-related questions from State, LWIB, and American Job Center personnel; • Ensure smooth implementation of the DEI Data System at the American Job Center level; and • Coordinate, design, and schedule technical assistance related to the DEI Evaluation (e.g., Data System site visits, webinars, conference calls, etc.).

11. DEI Evaluation Liaisons Kim Barlow, M.P.H. KBarlow@socialdynamicsllc.com IL, KS, NJ AR, IN, LA, MN Dr. Robert Bartolotta RBartolotta@socialdynamicsllc.com Danielle Herbert, M.A. DHerbert@socialdynamicsllc.com Dr. Becca Jackson Rjackson@socialdynamicsllc.com Dr. Susan Labin Slabin@socialdynamicsllc.com Dr. Robert Bleimann RBleimann@socialdynamicsllc.com HI, WA FL, MA, RI AK, OH, TN, SD Dr. Douglas Klayman dklayman@socialdynamicsllc.com CA, WI IA DE, ME, NY, VA Marilia Mochel, M.A. MMochel@socialdynamicsllc.com 1-855-990-1105 (toll free)

12. Data Quality Why is Data Quality Important? • Better position to accurately measure the impact of the DEI. • Ensures we can produce analyses on important subgroups (e.g., TTW customers). • Ensures that your work is accurately represented in annual and final reports.

13. Data Quality (continued) What Improves Data Quality? • Consistency in how each Data Element is defined. • Is the perceived barriers question asked the same way by each grantee? • Completion of all required Data Elements. • Are all Data Elements being completed by staff at each American Job Center? • Consistency in American Job Center operations. • Do all LWIA American Job Centers ask the “disability question”? • Is data collected from customers at approximately the same time intervals? • Is the eligibility criteria for certain services approximately the same across American Job Centers?

14. Data Quality (continued) How can the DEI Evaluation Team help with Data Quality? • Careful review of all DEI grantee American Job Centers to ensure that they collect the required DEI Data Elements. • Provide Data Quality Reports that pinpoint where errors occur. • Quarterly Data Quality Reports sent to DEI State Leads. • Webinars for State, LWIB, and American Job Center staff on DEI data collection requirements. • Issue Briefs on data quality and data collection requirements. • Toll-Free Telephone Technical Assistance available to DEI State Leads, State IT staff, and American Job Center staff (1-855-990-1105).

15. Percent of Error Ordered by Type of Error ( Pareto Distribution) Cumulative Distribution/ Pareto Curve Cases with Missing, Duplicate, 100 and/or Invalid SSNs 30% of N 90 LWIB1 80 60 60 Percent of Total Cases Missing SSN Percent of Total Error Duplicate SSN LWIB2 40 40 Missing Dis status LWIB3 Invalid SSN LWIB4 20 20 LWIB 5.......k 0 0 Percent of Cases by Type of Error Data Quality (continued) Data Quality and Statistical Control Pareto analysis can be used to detect the most frequently occurring sources (or vital causes) of error and the data reporters that have the highest error rates (vital cases). Pareto analysis does this by arranging the dataset according to priority of importance. Where should we be spending our time?

16. Data Quality (continued) What can you do to Maximize Data Quality? In addition to collaboration between the Evaluation Team, State Leads, and IT staff, several states have taken additional steps to help ensure data quality. For example: • NY had its own webinar to introduce staff to the mechanism for collecting DEI Data Elements. During this call, the importance of complete data was emphasized. • VA permanently adopted all of the DEI Data Elements into their everyday common intake form. • WA assigned data entry duties to Americorpsvolunteers, who were trained to enter DEI data.

17. 2013 Data Collection Activities Round 1: Telephone Interviews with Grantees & LWIAs Focus Groups with Selected LWIAs Submission of WIASRD & W-P Data October–December 2011  (March 15, 2013) January–March 2012  (March 15, 2013) April–June 2012  (March 15, 2013) July–September 2012  (March 15, 2013) October–December 2012  (March 15, 2013) January–March 2013 (April 15, 2013) April–June 2013 (July 15, 2013) July–September 2013 (November 15, 2013)

18. 2013 Data Collection Activities Round 2: Telephone Interviews with State Leads Submission of WIASRD & W-P Data October–December 2012  (March 15, 2013) January–March 2013 (April 15, 2013) April–June 2013 (July 15, 2013) July–September 2013 (November 15, 2013)

19. 2013 Data Collection Activities Round 3: Telephone Interviews with Grantees & Selected LWIAs Technical Calls on Data Submission, DSAs, & Web Portal

20. Preparing for the SSA Match for Round 1 • Work with ETA and ODEP to establish a Sharing Agreement with SSA • Set up necessary protocols for receipt of SSNs from SSA • Upload SSNs file and match with SSA • Clean file and return without identifying information to DEI Evaluation Team • Match is expected to be completed in January–February 2014 • First quantitative analysis of DEI data expected to be completed in Spring 2014

21. Social Dynamics, LLC 481 North Frederick Ave., Suite 410 Gaithersburg, MD 20877 Phone: (301) 990-1105 www.socialdynamicsllc.com To Contact the DEI Evaluation Team: Douglas Klayman, Ph.D. Project Director dklayman@socialdynamicsllc.com301-990-1105 x 105 Toll free: 1-855-990-1105 Visit us online at: www.socialdynamicsllc.com Visit our DEI portal at: http://socialdynamicsllc.com/dei.html