Challenges in Measuring and Collecting Data for Evaluation - A Literacy@Work Training Case Study

1. “Challenges in Measuring and Collecting Data for Evaluation” A Literacy@Work Training Case Study Community Literacy Summit Washington, D.C. March 19, 2007 Douglas Marriott, Outreach Director

2. Patti Presentation Overview • The Literacy Network of Greater Los Angeles (LNGLA) Guiding Principles and Current Programming • Literacy@Work Project Background • General project challenges of data measurement with MTA collaboration training • The unique and successful case of the MTA training, community literacy in action, measuring and collecting data for evaluation

3. Patti Guiding Principles of LNGLA We envision a Greater Los Angeles where all people are empowered by literacy to achieve their full potential in our community, a community which is socially just, culturally dynamic and economically productive. Our goal is make literacy resources easily accessible to learners, literacy service providers, and employers in our community through partnerships, advocacy, and direct program services.

4. Patti Current Programming • Resource and Referral • That Literacy Phone Book • www.literacynetwork.org • 1-800-707-READ • Special Events • Literacy In Media Awards • Literacy Summit Volunteer Program – Recruits, trains, and places volunteers at partner provider site. Spun off from the Literacy@Work. • Literacy@Work – Develops and implements workplace and workforce literacy trainings for workers within Los Angeles

5. Patti Project Background • Employment Policy Foundation study gave rise to Literacy@Work: The L.A. Workforce Literacy Project • Literacy@Work is a public/private partnership of organizations dedicated to the advancement of workforce literacy • Project has developed and implemented a 25 hour Trainer Certification program and a Quality Standards Handbook • Implementation of planning began in Spring of 2006 and has included workplace trainings at local businesses and workforce trainings at social service agencies

6. Patti The MTA Bus Driver Project We Make the Road by Walking (or riding)… • Between 2002 and 2006, the Los Angeles County Metropolitan Transportation Agency (Metro) began providing more bus service in Los Angeles County. • Over the past two years, Metro has been unable to fill from 200 to 300 bus operator vacancies, costing millions in overtime labor • Metro performed an analysis of the high failure rate in their training program, determining that the bus operator success rate could be significantly improved by offering interested applicants a pre-training program tailored to help them succeed in bus operator training.

7. Patti Collaboration in Action • Recruitment, data management, support services, case management • Lead on project coordination, job training and customer service instruction • Employment! HR orientations, program support • Literacy@Work instructional services workshops), trained instructors, customized curriculum development, volunteer support services, resource and referral information

8. Patti

9. Patti Ongoing Challenges for MTA Project Skills Assessment Implement a standard intake and assessment system and develop customized training for workers and employer. Performance Measurement Create MTA training components and requirements for program completion. Data Management Design and implement a shared, data management tool to support needs of all partnering agencies.

10. Patti Skills Assessment • Need or Expectation: • A uniform tool to place participants in program • Challenge: • Meeting the needs of all partnering agencies and adult learners

11. Patti Skills Assessment • Solution: • A standardized assessment tool (CASAS Workplace Exam) is implemented to determine placement in two week, five week, or sixteen week trainings • An MTA customer service exam is used as a pre and post assessment tool

12. Patti Performance Measurement • Need or Expectation: • Customized training course to prepare participants for success at Metro • Challenge: • Creating a customized curriculum to reach parallel missions of collaborators and individual needs of participants

13. Patti Performance Measurement • Solution: • Participants are provided with job readiness training, customer service training, English as a Second Language (ESL) when needed, basic skills training in reading and writing, study skills and critical thinking skills all contextualized to the needs of successfully completing the Metro training. The training is constantly adapting to lessons learned.

14. Patti Data Management • Need or Expectation: • Each participant needs to be followed to evaluate needs of the program and provide accurate accounting for all partners. • Challenge • Identifying resources and best methods for capturing data

15. Patti Data Management • Solution • WorkSource center dedicates staff member to data management and support services with direct links to MTA HR departments and partner agencies.

16. Patti Results to Date • Between Aug. 2006 and Jan. 2007 there were 7 trainings that reached 233 participants • 219 completed trainings and earned certificates • 198 passed MTA interview and began new employment orientation • MTA statistics show the agency is ½ way to meeting their goal of 300 new operators. The vacancy rate is down 161 from a high of 315 • Part-time attrition decreased from 100% to 33.49% as many are succeeding in MTA training program

17. Patti Conclusions • Continual process improvement impossible without detailed data collection for evaluation • All stakeholders must share a vision for what program success will look like with mutually agreed upon outcomes to be evaluated • Parallel missions, mutual respect for roles and resources, creativity and flexibility are key components to project success.