Pittsburgh Promise Research & Evaluation

1. Pittsburgh Promise Research & Evaluation Prepared for Lumina Foundation Convening October 10-11, 2013 Dr. Jennifer Iriti & Dr. William Bickel Evaluation for Learning Group

2. EFL work on Pittsburgh Promise • Since 2008- 5 years, variety of studies • Most work geared toward understanding implementation and providing formative feedback • Data stores and access improving Prepared for Lumina Foundation Convening October 9-10, 2013

3. Research Questions:PK-12 • What strategies are Pittsburgh high schools using to prepare students for post-secondary education? • In what ways, if at all, is the presence of the Promise influencing educators’, parents’, and students’ expectations and behaviors? • What practices are used by high schools that are most effective at getting students eligible for the Promise? • Why do HS graduates who are eligible for and have been awarded the Promise fail to use the scholarship in the subsequent semester? • To what extent do targeted programs influence participants’ “Promise-readiness?” Prepared for Lumina Foundation Convening October 9-10, 2013

4. PK-12 Methodologies • Case studies of 4 high schools guided by framework for college-career readiness • Pittsburgh Promise eligibility/award data paired with HS graduate data to determine high performing school in terms of proportion of graduates eligible for and using the Promise • Interviews with HS graduates who were awarded the Promise, had attained acceptance to a post-secondary institution but did not enroll in the semester after graduation Prepared for Lumina Foundation Convening October 9-10, 2013

5. PK-12 Key Findings • Change in ASPIRATION does not equal change in BEHAVIOR • Schools need research-based college and career readiness framework • Use of coherent set of college and career ready practices linked with greater proportions of graduates eligible for the Promise • Graduates who could have but didn’t reported no strong internal drive to attend post-secondary in the summer after graduation but universally regretted this 9 months later Prepared for Lumina Foundation Convening October 9-10, 2013

6. Research Questions: Post-Secondary • At the outset of their post-secondary studies, how prepared were Promise Scholars to meet the demands of college? • What are the Promise Scholar rates of retention in post-secondary? How do these compare to national rates? • Do particular high schools or post-secondary institutions have higher retention rates when student characteristics are controlled? • What are the most significant challenges to student post-secondary retention? • What strategies do post-secondary institutions use to support student success in higher education? Prepared for Lumina Foundation Convening October 9-10, 2013

7. Post-Secondary Methodologies • Online survey of Promise Scholars in 1st year of post-secondary • Analysis of Promise Scholar post-secondary enrollment, retention, and persistence • Comparison to ACT national rates • Institution Promise Scholar retention rates compared to Institution cohort retention rates • Logistic regressions to identify predictors of Scholar retention • Case studies of 5 higher education institutions (high, moderate, and low performers) Prepared for Lumina Foundation Convening October 9-10, 2013

8. Post-Secondary Key Findings • Scholars felt less well prepared for time management, study skills, and research skills • Scholar retention rates are higher than for ACT national sample for all institution types • For 8 of the 12 higher education institutions Scholars retained at higher rates than their peers at that institution • Family income level, higher high school GPA, and being female were predictive of retention • Once student variables were controlled there was almost no statistically significant differences among high schools • Statistically significant differences were found for individual higher education institutions • Higher education personnel identify challenges to student success ranging from academic preparedness, to “college knowledge,” to financial demands, and personal characteristics • Higher performing post-secondary institutions typically had elaborate programming in place to support Scholars Prepared for Lumina Foundation Convening October 9-10, 2013

9. Prepared for Lumina Foundation Convening October 9-10, 2013

10. Other Data/Evaluation Work • Articulating theory of change to support research design and programming development • Identifying indicators in the PK-12, higher education, and community that are either pre-conditions or targeted outcomes to serve as “dashboard” Prepared for Lumina Foundation Convening October 9-10, 2013

11. Promising Opportunities in Pittsburgh • Linking data systems across sectors to allow more sophisticated and quasi-experimental designs. • Understanding the Promise “ecosystem” ways it changed as a result of the scholarship, and whether these changes resulted in any observed PK-12 and post-secondary attainment improvements. Prepared for Lumina Foundation Convening October 9-10, 2013

12. EFL Research/Evaluation Related to the Pittsburgh Promise • Iriti, J.E., Bickel, W.E., Meredith, J., & Walker, M. (2009). Looking inward to keep The Promise: What do Pittsburgh charter high schools do to prepare students for post-secondary education? Pittsburgh, PA: Learning Research and Development Center, University of Pittsburgh. • Iriti, J.E., & Bickel, W. E. (2010). Looking inward to keep The Promise: How Pittsburgh high schools prepare students for post-secondary education- Year 2. Pittsburgh, PA: Learning Research and Development Center, University of Pittsburgh. • Iriti, J. E., Bickel, W. E., & Nelson, C. A. (2010). Keeping “The Promise:”Practices for building post-secondary readiness and Promise scholarship eligibility in high schools. Paper presented at the Annual Meeting of the American Educational Research Association in Denver, CO. • Iriti, J. E., & Bickel, W. E. (November 11, 2011). “Promises” not yet realized: Those who could but don’t use Promise scholarship funds. Pittsburgh, PA: Heinz Endowments. • Iriti, J. E., & Bickel, W. E., with C. Long (May 31, 2012). Mentoring to The Promise: Evaluation of the Be a 6th Grade Mentor Program. Pittsburgh, PA: United Way of Pittsburgh. • Iriti, J.E., Bickel, W. E., & Kaufman, J. (August 2012). Realizing “The Promise:” Scholar persistence and retention in post-secondary education. Pittsburgh, PA: Learning Research and Development Center, University of Pittsburgh. • Iriti, J., E., Bickel, W. E., Kaufman, J., & Wang, E. (2013). Pittsburgh Promise potential annual report indicators. Pittsburgh, PA: Learning Research and Development Center, University of Pittsburgh. Prepared for Lumina Foundation Convening October 9-10, 2013