Biennial FTA State Programs Meeting / State Public Transit Partnerships Conference August 7, 2103

1. NCHRP 20-65 (47) - MTAP Survey Tool Used to Assess FTA Contractor Performance of State DOT Triennial and Other FTA Reviews - An Update Biennial FTA State Programs Meeting / State Public Transit Partnerships Conference August 7, 2103 KFH Group, Inc. Beth Hamby Sue Knapp

2. Background • To assist FTA with improving State Review processes, MTAP collects input from State DOTs on their experience following each FTA review via an online survey tool. • Survey was created in Vovici (used by AASHTO for other surveys). • Value of survey has been somewhat limited by: • open-ended answer formats • challenges in using Vovici

3. Research Objective: • Update MTAP’s online survey to: • improve efficiency / ease of completion • increase the value added nature of the process (improve results for ease of use, benefits) • incorporate technology improvements • provide for more detailed responses to 2-3 specific questions of interest to MTAP, SCOPT, FTA - not just a "comments" field

4. Project Tasks • Task 1 – Assess Survey Tool Improvement Needs • Task 2 – Revise Survey Content & Structure • Task 3 – Assess Available Survey Technology Options • Task 5 – Set Up & Test Revised Survey Using Selected Technology • Task 6 – Prepare Final Report

5. Project Status • Task 1 – Assess Survey Tool Improvement Needs – completed • Reviewed current survey tool - http://vovici.com/wsb.dll/s/54c9g50982 • Reviewed reports generated from current tool • Gathered feedback/suggestions from • States/MTAP Committee • FTA Office of Oversight and Program Guidance • MTAP Coordinator

6. Project Status • Task 2 – Revise Survey Content & Structure – nearing completion • Modified existing questions – standardized responses where possible to make analysis easier • Added new questions • Prepared revised survey for review by MTAP/ Steering Committee; sent late July

7. Draft Revised Survey • Numerous new questions seeking specific feedback • Basic experience of reviewer • Consultant performance on specific compliance topics • For each scaled rating (Excellent/ Good/ Fair/ Poor), asks why (open-ended) • Extent of Regional participation • Effectiveness of entrance/exit meetings • SMR workshop effectiveness • Interest in follow-up discussion with MTAP or FTA • Standardized responses; limited open-ended questions to explanation and “other” • Reworded some questions to align with responses

8. Draft Revised Survey • Contact and Background Info • Added: Basic experience of respondent • Consultant Performance • Added: Fields for explanation • General Experience with Review • Added: Regional office participation • Added: Entrance/Exit meetings • Added: Overall organization and clarity • SMR-Specific Feedback • Added: Respondent’s participation • Workshop/workbook effectiveness • Additional training/tech assistance needs • Additional Feedback

9. Project Status • Task 3 – Assess Available Survey Technology Options – under way • Criteria include: • User-friendliness to create & administer survey • User-friendliness to complete & submit surveys • Functionality/customization options • Reporting capabilities • Cost

10. Survey Approaches General options for getting feedback from states: • Online survey tools – basic to advanced • Focus of our assessment • Pen & paper • Too labor intensive • Telephone interviews • May be appropriate for follow-up

11. Basic Online Survey Tools • Examples include: • SurveyMonkey Basic / Select / Gold • SurveyMoz Free / Premium • SurveyGizmo Basic • QuestionPro Professional

12. Basic Online Survey Tools • Advantages: • Easy to learn and use • Provides some skip logic functionality • Provides some basic cross-tab analysis • Can export results to Excel for additional analysis • Inexpensive ($0 - $25/month)

13. Basic Online Survey Tools • Disadvantages: • Limited collaboration capabilities • Limited survey customization options • Limited reporting / analysis functions – need to export to third party application • Basic tools are becoming increasingly sophisticated (e.g., SurveyMoz) • Some offer higher levels of customization for a higher cost (e.g., SurveyMonkey Platinum - $65/month)

14. Advanced Online Survey Tools • What MTAP is currently using (Vovici) • Known as Enterprise Feedback Management (EFM) Systems • Examples include: • Key Survey • Qualtrics • QuestionPro Corporate • SurveyGizmo Enterprise • Vovici

15. Advanced Online Survey Tools • Advantages: • Allows for greater collaboration in survey administration (behind the scenes) • Multiple user accounts possible • Can share surveys with other survey administrators • Private results through secure dashboard accounts • Greater flexibility in branding and customizing • Can build sophisticated logic into survey • Allows for complex analysis • Custom reports can be developed within the tool

16. Advanced Online Survey Tools • Disadvantages: • Complex interface • Requires training / steeper learning curve • May require frequent use to maintain skills • May require experience in Cascading Style Sheets (CSS) to customize look and feel of survey • Expensive ($100 or more/month – few disclose pricing online) • However, no out-of-pocket cost to MTAP for Vovici; AASHTO already pays for it for other uses

17. Next Steps • Fine-tune survey questions with input from MTAP/ Committee • Finish technology assessment • Set up & test revised survey using selected technology • Prepare final report

18. For more information or feedback Beth Hamby KFH Group bhamby@kfhgroup.com 206-448-6749