Gaining Options for College Collaborative

1. Gaining Options for College Collaborative Testing New Models for Talent Search: Using Data, Empowerment Evaluation and Systems Collaboration Dr. Nicole Norfles Dr. Margaret Cahalan Dr. Stephanie R. Miller Council for Opportunity for Education

2. Topics • Overview of the pilot project and Go-College (i3)project (Stephanie) • Project components • Using data (Stephanie) • Collaboration (Nicole) • Traditional and empowerment evaluation (Maggie) • Discussion and Questions (all)

3. Pilot Project (GE Project) • Project support by the GE Foundation • Launched in 2006 • Student-level intervention (60 students) • Located in four sites • Louisville, KY (1 school, n ~= 300 students) • Erie, PA (3 schools, n ~= 720 students) • Connecticut (1 school, n ~= 240 students) • Harlem, NY • 1 school, n ~= 350 students • 1 school (whole school approach), n ~= 800 students

4. GE Pilot: Project Components • Existing College Access Program (CAPs) serving students in school • Academic and College Coaching Services • Academic advising (quarterly sessions) • Weekly/bi-monthly group sessions • Limited tutoring services • College exploration • Summer programming • One embedded college coach (serving 60 - 80 students per grade) • Limited whole school effort • Base data driven decision-making • Learning communities

5. GE Pilot: Preliminary Findings from 1st Graduating Cohort in Louisville • 69 students served • 32 students enrolling at 2-year school • 30 enrolling at 4-year school • 3 joining the military • 4 not graduating or transferred • Financial Aid • 18 students received some form of scholarship • 6 students received full scholarships • Majors • Animal science • Biology, chemistry • Math education • Nursing • Graphic design • Pre-pharmacy • Business

6. GO College: Investing in Innovation • COE 1 of 49 selected from 1,700 applicants • $20 million grant from Dept. of Education, $4 million match from the GE Foundation • Builds on Talent Search model and GE pilots • Whole school model with intensive learning communities • Implemented in 2 cities: Erie, PA and Louisville, KY • Rigorous external evaluation required (Educational Testing Service - ETS)

7. GO College: Project Components, Add on Model base Talent Search vs. GE Pilot vs. GO College • Selection of students/learning communities (intensive services) • Services • Outcomes • Using data/data process (system) • Collaboration • Evaluation (internal and external)

8. GO College: Collaboration • GO College provides one model where TRIO pre-college programs can meet the rigorous curricula and collaboration requirements of the Higher Education Opportunity Amendments of 2008 • The project could be replicated locally by collaboratives of TRIO programs and high school districts.

9. GO College: Multiple Communities of Learning Stakeholders and Collaborators

11. GO College: CollaborationEngaging the Community • Community events • Churches, community leaders, businesses, parents • Marketing materials • Four press events per year Press Conference and Launch GO College - Erie

13. GO College: The EvaluationA Personal Journey • Contractor Project Director (National Evaluation of Student Support Services, National Evaluation of Talent Search, TRIO performance reporting support contracts) • Department of Education as Technical Monitor • Did review of evaluation studies from last decade—(not often viewed as useful or valid by practitioners----not often find positive impacts --are we asking the right questions?; how can we make studies more useful and still provide input into policy decisions ) • Has evaluation research –overpromised in terms of validity of results and in terms of usefulness---what does lack of effects mean???) • Came to see need for taking a more participatory approach involved and began Designing Next Generation of GEAR UP studies that were developed by grantees using technical assistance from RTI –use traditional models of evaluation

14. GO College: Internal or External Evaluation? • External evaluation required—for I-3 Validation studies using traditional methods meeting What Works Clearinghouse criteria as much as possible—model of validate and then scale up if find positive effects • “The dilemma of whether to use external or internal evaluation is as false as that between qualitative and quantitative methods. The solution is always to use the best of both, not just one or the other” (Michael Scriven) • COE-I3—Go-College Collaborative grant is using both approaches— working collaboratively with ETS and their sub-contractor Brown—ABT is technical assistance provider—(for example, Brown just completed random assignment of rising 9th graders in the 6 schools based on data COE compiled and processed)

15. Traditional vs. Empowerment Evaluation Traditional Evaluation • External • Expert • Dependency • Independent judgment • Developed when data not available to all— elite with resources to collect and skill to analyze Empowerment Evaluation Internal Coach or Critical friend Self-determination & capacity building Collaboration Makes use of Data Revolution—internet, web, real time interactive sharing of knowledge—all publish—face book, blog, twitter

16. Internal and External Evaluation Enhance Each Other • Can provide richer data set that enables more complete external examination • External reality check and quality control—keep on track • External—help question shared bias • Coordinate data needs • Mixed methods • Evaluators—co-equals—not superior or servant

18. GO College: Null-Hypothesis & Methods Null Hypothesis • Implementation of the GE-Pilot and the I-3 enhancement strategies will not be accompanied by measurable change in college readiness and college going rates over the period of study for the whole schools • Rising 9th graders randomly assigned to participate in the more intensive learning communities will not differ in outcomes observed from those not invited to participate • Diverse students selected at random from ability quartiles will not differ in outcomes observed from students who volunteer for the learning communities Associated Methods • Observational study of pre and post –and comparison schools outcome trends in schools over 15 year period–Requesting data from 2000—2015—Case History—–use systems concepts—community asset development study • Experimental study for purpose of understanding best way to select students and to give equal chance for scarce treatment. Departure from usual method of TS of selection based on student and teacher interest • Observational study of differences in outcomes observed in GE pilot in which students volunteered or teachers selected—method used for 10 to 12 graders

19. GO College: Confounders, Challenges and Opportunities • Dynamic Data Driven Focus---implies using the data to improve services on an on-going basis—intervention changes –we are validating a dynamic context driven adaptive reflective process not a static intervention • Community Initiative and Collaborative data use by all stakeholders lends project to involve internal reflection and self evaluation • Fast Changing Context---examples --Severe cutbacks to schools in districts, US presidential challenge to be first in degrees, data use change, no community college in ERIE and recently voted down, 55 000 degrees-Louisville, ACT collaboration • Whole School and Previous Other services received before and during—Talent Search—GE-Pilot—I-3—Others-limits contrasts and meaningful counterfactuals

20. Go-College: Collaborative –Process of Participatory Action & Change

21. Using Empowerment Evaluation As Internal Evaluation Tool • Empowerment evaluation is the use of evaluation concepts, techniques and findings to foster improvement and self determination. It employs both qualitative and quantitative methodologies. Empowerment evaluation: Knowledge and Tools for Self-Assessment and Accountability Fetterman, Kaftaraian, and Wandersman 1996 • Widely-used • Joint Committee for Education Program Standards have included concepts • American Evaluation Association and AERA workshops • Public Health Field • International Development work • Education school systems

22. 10 Key Principals of Empowerment Evaluation(Wandersman et. Al 2005) • Community Ownership— primary responsibility with organization and not outside evaluator • Inclusion— involves representation of key stakeholders • Democratic participation— highly collaborative—opportunity to voices questions—every stakeholders voice is heard • Community knowledge- promotes growth of knowledge in community—stakeholders are considered to be in the best position to understand the issues and generate solutions to problems • Evidence Based strategies- promotes use of strategies with high quality evidence—research evidence of effectiveness—evidence strategies contextualized to fit community

23. 10 Key Principals of Empowerment Evaluation(cont.) • Accountability- provides data that can be used to determine whether a strategy has achieved its goals—negative results are used to inform change in a strategy or the selection of a new strategy for the purpose of producing better outcomes • Improvement— Helps organizations improve strategies so that they are more likely to achieve stated goals—process and outcome evaluation (Rossi 1999) • Organizational learning—fosters a culture of learning—view positive and negative feedback as valuable information and believe that all strategies can be improved • Social justice— Increase capacity to reduce disparities that affect marginalized by persecution, discrimination, prejudice and intolerance • Capacity building— builds capacity of organizations to conduct their own evaluations, understand results and use them to continuously improve organization

24. Preliminary Plan of Topics for Stakeholder Collaborative Report • Ethnographic school histories: Quantitative and qualitative, outcomes assessment • Assessment of the strategies effectiveness and recommendations for improvement (collaboration, data use, whole school, learning communities, diversity/asset based) • Implications for Talent Search and College Access Programming • Modeling Meeting the 2020 College Attainment goals—student contributions –data use (international, national, state, local, individual) • Reflections on use of innovative empowerment evaluation tool—evaluate the evaluation tool

25. Questions and Answers