10th International Seminar Itaú Economic Evaluation of Social Projects

1. Life Skills, Employability and Training for Disadvantaged Youth: Evidence from a Randomized Evaluation Design in the Dominican Republic 10th International Seminar Itaú Economic Evaluation of Social Projects Pablo Ibarraran, October 29th 2013

2. Overview • Results of RCT of a youth-training program in the Dominican Republic: “Juventud y Empleo” on: • Labor market outcomes • Behavior, perceptions and expectations • Non-cognitive skills • Motivation for further research in areas related to non-cognitive skills: • Measurement • Training • Labor market impacts

3. Youth Employment in the Dominican Republic • Big challenge, but not necessarily more than in other countries. • Large percentage not working nor studying: 34% of youth aged 18-29. • Youth unemployment rate = 2.3 * Adult unemployment rate. • Large percentage of youth employed in low quality jobs: Youth informality = 2 * Adult informality rate.

4. Juventud y Empleo • Based on the Joven-type of program, first with RCT evaluation component in LAC. • Targets disadvantaged youth (school dropouts, ages 16-19, not studying or working) • Combines classroom training with subsequent internship period of on-the-job work experience. • Role of the private sector: identification of training needs, provision of training sessions, offering of internships.

5. First Impact Evaluation • Carried out by Card et al. with data from a cohort that applied to receive training in early 2004 and was followed up in mid 2005. • Found little impact on employment; though sizable impacts on wages and formality for men. • Randomized design was potentially compromised by failure to include non shows and dropouts in the follow up survey. • Sample size was limited (n=1,345).

6. Revamped Program and Evaluation • Closer work with the training centers and firms to develop tailored courses that train youth for real vacancies. • Strengthening of the non-cognitive skills training (based on private sector claims). • Randomized assignment on a larger sample for each course (20 treatments and 15 controls). • More complete survey instrument with 15 modules.

7. Limited evidence on non-cognitive skills • Trend to include of non-cognitive skills components in training programs. • Evidence on labor market success determinants shows that employers value certain behaviors that are linked to high productivity workers • There is no clear evidence on how to best measure NCS empirically. • Knowledge gap in Latin America: • Measurement of non-cognitive skills. • Relation between non-cognitive skills and labor market outcomes of youth.

8. Randomization Process • The training centers (TC) identify potential participants from a group of eligible applicants • TC submit data on 35 eligible and interested youngsters for each course. • The Ministry receives the information from TC, verifies eligibility and randomly assigns 20 to the treatment group and 15 to the control group. • Training centers replaced up to five slots of non shows and dropouts with control group members.

9. Assignment and Participation Status • Classification based on administrative data. • Number of “never takers” equals number of “always takers” as course size was kept constant.

10. Execution 307 courses 350 Follow-up 300 250 26 Months after the end of the last course 200 150 100 50 0 Timeline

11. Balance at Baseline

12. Follow Up Survey • Carried out 18 to 24 months after graduation. • Based on a random sample of 5,000 out of 10,309 individuals initially registered. • 3,250 in treatment group and 1,750 in control group. • Minimum power of 0.8 to detect an 8% effect in employment. • 70% success rate assumed in completion of follow up surveys.

13. Follow Up Survey • More than 80% of sample located for follow up survey (identical for the treatment (80.8%) and control (80.4%) groups. • Verification of the participation status. • Training centers enjoyed some control over who took the course. • Replacements that declined to participate classified in control group.

14. Realized Treatment and Control Groups • Compliance was low in the control group: • 37% contacted by training center and accepted course. • 23% took course and completed the internship. • The lottery still had a strong impact on the probability of participating and constitutes a strong instrument for the treatment.

15. Intention to Treat Effects • Estimation of ITT, using OLS with std errors clustered at the course level and TC fixed effects. • We center on three impacts (with heterogeneity) • Labor market • Youth behavior and life style, perceptions and expectations • Non-cognitive skills • Correlation with labor market performance

16. Employment History

17. Employment History – Santo Domingo

18. Labor Market Outcomes I

19. Labor Market Outcomes II

20. Life Style and Risky Behaviors: Pregnancy

21. Perceptions and Expectations

22. Instruments to Measure Non-Cognitive Skills • Social and Personal Competencies Scale (CPS): • Six basic competencies: leadership, behavior in situations of conflict, self-esteem, ability to relate with others, order, empathy, and communication skills. • Rosenberg Scale: to measure self-esteem. • Grit Scale: • Measures determination and strength of mind. • Four main categories: persistency of effort, enthusiasm about long term goals, consistency of interests, and ambition.

23. Social and Personal Competencies

24. Rosenberg and Grit Scales

25. Employment and NCS

26. Are Impacts on NCS Relevant? • Statistically significant positive impacts on various measures of non-cognitive development. • 0.11 sd in total CPS score (similar for other categories) • 0.11 sd in the Rosenberg Scale in men. • 0.08 sd in the Grit Scale (similar for other categories). • What does this mean? • Literature on relationship between non-cognitive skills and professional success relatively new. • Still uncertain which specific competencies relate to participation in the labor market. • Limited evidence on the magnitude of the changes required to impact employability.

27. Are Impacts on NCS Relevant? • Reference to literature in cognitive and non-cognitive skills development: • 0.4 standard deviation per year in educational attainment in primary education (Hill et al. 2007). • 0.13 standard deviations in non-cognitive skills assessments with participation in sports clubs (Felfe et al. 2011).

28. Non-Cognitive Skills Unknowns We need evidence on: • The specific skills that are positively related to labor market participation. • Assessment of the magnitude of the change required to impact employability. • Replication of effective training program designs that can be easily scaled up. • Design of relevant instruments to measure (and interpret) non-cognitive skills development.

29. Summing Up • Labor market outcomes: • Job Quality for men: 18% national; 52% Santo Domingo. • Monthly earnings for those employed: 9% overall; 17% Santo Domingo. • No statistically significant impact on employment. • Youth behavior and expectations: • Drop in pregnancy: 28% overall; 43% teenagers. • Positive impact on expectations about the future. • Non-cognitive skills: • Positive impact on the CPS, Rosenberg (for men) and Grit Scales: 0.8 - 0.11 std. dev.

30. Thank youpibarraran@iadb.org