220 likes | 240 Views
Assessment of PennDOT's LTAP training workshops and technical assistance sessions, including self-assessment, follow-up surveys, workshop statistics, and participant feedback. Evaluations cover learning outcomes, knowledge retention, job performance improvement, and overall satisfaction. Data collected from various programs and workshops to ascertain effectiveness and benefit to participants.
E N D
PENNDOT’S LTAP EVALUATION ACTIVITIES Kim M. Ferroni, PennDOT LTAP Project Manager Barbara T. Harder, B. T. Harder, Inc. LTAP Evaluations
PennDOT LTAP • PennDOT owner and project management • Contracted operations -- Pennsylvania State Association of Township Supervisors; since December 2005 • Sponsored during 2007 • Training workshops, 227 • Technical assistance sessions, 89 on-site (field), off-site 251 TRB Technology Transfer Committee Meeting
PennDOT LTAP Evaluation Activities • Training Workshops • “Happy Sheet” information – evaluation of the experience (facility, time, materials, instructor) • Self assessment of learning • Initiating pre- and post-testing • Follow-up survey TRB Technology Transfer Committee Meeting
PennDOT LTAP Evaluation Activities • Technical Assistance • Self assessment of learning • 6 month follow-up evaluation • General benefits assessment TRB Technology Transfer Committee Meeting
Training Workshops • Report some basic statistics • 3 workshops for illustration • Asphalt Roads – Common Maintenance Problems • Bridge Maintenance • Work Zone Traffic Control • Most frequent workshop offerings during 2 month period of 2007 (representative sample – about 60% of total) TRB Technology Transfer Committee Meeting
What Evaluation Sheets Tell Us • Based on rating of three specific learning goals for each workshop • 100% rating means “doubled” the knowledge TRB Technology Transfer Committee Meeting
Self Evaluation of Learning • Self Evaluation of Learning • Highest assessed learning reported in Work Zone Traffic Control Workshops • Understand requirements or WZTC in accordance with state and national guidelines (83%) • Able to develop traffic control plans using PennDOT publication (100%) • Understand recent changes in work zone rules (115%) 100% rating means “doubled” the knowledge TRB Technology Transfer Committee Meeting
Use of Training Information • 95% of workshop attendees reported that what they learned would improve job performance • 76% will use the information they learned within 6 months of the workshop • 9% use within 1 year, 13% unsure TRB Technology Transfer Committee Meeting
“Happy” Ratings TRB Technology Transfer Committee Meeting
Follow-up Training Workshop Surveys • Initiating use of Survey Monkey to determine follow-up actions (distributed 6 months after training) • Performance • Implementation • Learning • Do not yet have sufficient data to analyze TRB Technology Transfer Committee Meeting
Maintenance and Safety Technical Assistance • Technical assistance • Off-site – telephone, email • On-site • Maintenance and safety topics • Traffic signs • Engineering and traffic studies • Work zone traffic control • Guiderails, posting and bonding • Evaluation requested for every tech assist session – 40% response rate TRB Technology Transfer Committee Meeting
Maintenance & Safety Tech Assists Types of Evaluation Respondent Organizations June 06-Nov 07 County, 0, 0% TRB Technology Transfer Committee Meeting
Maintenance & Safety Tech Assists Frequency of Benefits Evaluations ReceivedJune 06-Nov 07 TRB Technology Transfer Committee Meeting
Maintenance & Safety Tech AssistsUsefulness of Information Evaluations Received June 06-Nov 07 • 80% of respondents anticipate using information provided in Tech Assists within 6 months and 91% within one year AND • 94% of Tech Assist information adequately addressed the issue of concern TRB Technology Transfer Committee Meeting
Use as exception flag for problems Provides opportunity for suggestions and other feedback Maintenance & Safety Tech Assists “Happy Ratings” June 06-Nov 07 * Maximum rating = 4 TRB Technology Transfer Committee Meeting
Maintenance & Safety Tech AssistsSummarized Comments Evaluations Received June 06-Nov 07 • The LTAP staff was (multiple respondents) …quick to respond, answered all my questions, help{ed to} educate me throughout several processes, very helpful, polite, and professional, excellent to work with, easy to talk with, an encyclopedia of traffic engineering and detailed issues… • The information provided was (multiple respondents) … understandable, {enabled us} to use the correct material for our signs, gave us confidence in knowing the correct way, delivered quickly… • We learned a lot. Written info will come in handy in the future also. Thank you for providing this type of assistance. TRB Technology Transfer Committee Meeting
Technical Assistance Follow-up • Follow-up evaluation for on-site visits, 22.6% response rate • “Increase Safety” and “Do Job Right Way” are leader benefits after 6 months TRB Technology Transfer Committee Meeting
Types of Benefits • Quantitative Benefits • 10 of 34 Benefits Evaluations provided some quantitative information (29%) • Candidates for more in-depth assessment • Qualitative Benefits • 32 of 34 Benefits Evaluations provided some type of qualitative information (94%) TRB Technology Transfer Committee Meeting
Example Quantitative Benefits • Engineering & Traffic Studies • Saved $10,200 – did not have to do a traffic study • Traffic Signs & Guiderails • 30 hours saved/month; $11.50 rate/hour • Saved the cost of putting guiderails at the wrong places and the time to put up the correct signs TRB Technology Transfer Committee Meeting
Example Qualitative Benefits • Risk reduced for walking and motoring public. Liability risk reduced for Borough and school district. Safety increased significantly. • The project was beneficial to establishing better working relationship with the township. • Better or cleaner stone for chip roads. Trying to make our tar & chip projects last longer. • Avoid possible law suits; made drivers more aware of upcoming intersection. TRB Technology Transfer Committee Meeting
Questions and Issues for Discussion • Training workshop pre- and post-testing – what works, what doesn’t? • Literacy issues • Voluntary or mandatory • When used • Learning assessment from supervisors • What experiences have others had? • Response rates • Effort required • Benefits measurements – what programs are doing this and how? • Who is providing the information? TRB Technology Transfer Committee Meeting
For More Information • Kim M. Ferroni – PennDOT LTAP Project Manager, kferroni@state.pa.us • Carol Kilko – Contract Manager, PSATS, ckilko@psats.org • Barbara T. Harder – B. T. Harder, Inc. Contract Staff, Evaluations, btharder@verizon.net TRB Technology Transfer Committee Meeting