1 / 28

Research Performance Measures Survey 2014

Research Performance Measures Survey 2014. Presented at 2014 AASHTO Research Advisory Committee Meeting, Madison WI, July 24, 2014. Bill Stone Research Administrator Missouri Department of Transportation. AASHTO RAC Survey State DOT Survey -- March 2010.

jeb
Download Presentation

Research Performance Measures Survey 2014

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Research Performance Measures Survey 2014 Presented at 2014 AASHTO Research Advisory Committee Meeting, Madison WI, July 24, 2014 Bill Stone Research Administrator Missouri Department of Transportation

  2. AASHTO RAC SurveyState DOT Survey -- March 2010 • Identified very broadly defined performance measures used by state DOTs at the time • Measures taken from public sources such as accountability reports, state DOT websites, and other web-based resources • Out of 40 states, 10 had published measures and 30 had no public information • Survey lists metric, source and URL

  3. DOT State Statshttp://stats.mtkn.org • The DOT State Stats report is a synthesis of facts, figures, statistics and metrics pulled from accountability reports, online performance measurement dashboards and fact books. • 2009: Need existed to easily find and access local statistics • Editions available: 2008-2012

  4. DOT State Stats • Annual figures compiled in easy-to-use synthesis report online. Excel dataset also available online. • Created and launched by previous MoDOT Librarian; assisted by members of the Midwest Transportation Knowledge Network (MTKN) and hosted on their website. • 2012 edition included metrics from 42 states. • Publication is used by the transportation community at large.

  5. DOT State Stats • Synthesis report no index • Dataset searchable but not categorized • No longer current  not updated since 2012 Searching on “research” retrieves only 4 results out of 1,580 measures

  6. RAC Value of Research Task ForceState DOT Survey -- May 2014 • Obtain input from research administrators actual or anticipated use of research performance measures to document the progress and successof their research programs • Measures with performance targets or metrics that track activity • Measures that were program-specific not project-specific • Measures from all research program areas (pure research projects, library and/or technology transfer program)

  7. 2014 Survey Results 49% response rate (25 states)

  8. Does Your State Track Research Program Performance?

  9. For States That Don’t Track Measures

  10. Research Performance Measure Characteristics

  11. How Frequently Are Measures Collected or Reported?All Responses • 20% of states collect or report statistics for more than just one time period

  12. Are the Measures Published?All Responses-13% of States Do Both

  13. How Does Your State Use the Measures? • CURRENT USE • Customer satisfaction (AZ) • Monitor progress toward strategic goals (DC); adjust performance (TX) • Make a difference in organizational business processes (MO) • Determine program effectiveness (IA) • Identify strengths and weaknesses (IN); what’s working or not working (MD) • Tied to individual performance evaluations (LA);

  14. How Does Your State Use the Measures? • CURRENT USE (CONT’D) • Assess researcher performance and the usefulness of completed research (NC) • Select new projects (TX) • Ensure that projects move forward according to set budget and schedule (UT, WY) • Develop and monitor SPR Part 2 program (WI) • FUTURE USE • Will use to direct funding (FL) • Will use to gauge and improve performance (IL)

  15. Measure Categories • Applied 31 unique categories to 103 submitted individual measures • Measures by Status • Current: 69 measures • Planned: 21 measures • Considering: 10 measures • Past: 2 measures • In progress: 1 measures • Some measures have multiple categories applied

  16. Measure Categories • TOP CURRENT: Completion (14), Implementation (9), Library Utilization/Processing (8), Training (7), Satisfaction (6) • TOP IN PROGRESS/PLANNED/POTENTIAL: Implementation (6), Completion (3), Cost Savings (3), Timeliness (3), Within Budget (3)

  17. Completion CategoryExamples of Measures • Related to number • No. of completed research projects • Related to on-time or on-budget • % of research projects meeting original completion date • No. of projects completed in the FY on schedule • % of projects completed according to the initial budget • % of projects completed on time/on budget • % of studies completed within the approved schedule of the Work Program

  18. Completion CategoryExamples of Measures Cont’d • Related to satisfaction • % of completed studies deemed satisfactory by the project sponsor • Customer satisfaction survey for completed Research • Related to implementation • In past 5 years, 75% of completed research projects provide recommendations for implementation of results endorsed by the Project Review Committee

  19. Implementation CategoryExamples of Measures • Related to number • No. of research results and best practices implemented • No. of NCHRP and other external research program results implemented • % of projects implemented • Fully, partially, later, cannot/or should not be implemented, within 2 years of final research report (using 5 years of data) • % of projects actively utilized in the field

  20. Implementation CategoryExamples of Measures Cont’d • Related to number cont’d • % of projects that have resulted in a spec, policy, procedure, manual, requirement or material change • In past 5 years, 75% of completed research projects provide recommendations for implementation of results endorsed by the Project Review Committee • Multi-year tracking of implementation work • Related to funding • Amount of funding for implementation activities

  21. Level of Effort to Compile Measures

  22. Tools Used to Collect MeasuresAll Responses-75% of States Use More Than One

  23. How Have You Used the RPM Website? • CA and MN: Reviewed • DC & LA: Used as basis for developing PMs • IL: Used to develop a structure for functional PMs • IN: Exploring how to move measures to RPM • UT: Use to upload measures • ID, LA, MI, MO, PA and UT: Use for HVR submittals

  24. RPM Website Tools: How Useful?

  25. RPM Web Comments • PROS • “RPM Web was extremely useful in helping us grasp the different types of performance measures and developing a structure for functional performance measures.” • “Need to re-visit the site now that our program is more developed to see how it might help.” • CONS • “Adapting to RPM Web would require a lot of work.” • “Have mixed feelings about the usefulness.” • “Did not find the tools very beneficial, because the benefits need to be customized for individual states …” • “Information may be duplicated elsewhere … Would need to build the measures into projects prior to start … Input not required/hasn’t been a priority.”

  26. Observations/Questions • Who bears the cost of developing measures for each individual project? Researcher? Program director? • What is the benefit/cost difference of applying measures to all projects versus a sample? • What is the right balance between qualitative vs. quantitative data to demonstrate program effectiveness? • Issues/challenges with what administration wants or needs (prediction versus real cost or savings)

  27. Contact Information Bill Stone, PE Construction, Materials and Research 1617 Missouri Blvd. P.O. Box 270 Jefferson City Missouri 65102-0270 573-526-4328 William.Stone@modot.mo.gov Research Ahead

More Related