Efficiency Measurement of Road Maintenance using Data Envelopment Analysis

1. Efficiency Measurement of Road Maintenance using Data Envelopment Analysis PRESENTED BY: KONSTANTINOS TRIANTIS AUTHORS: MEHMET EGEMEN OZBEK JESUS M. DE LA GARZA KONSTANTINOS TRIANTIS DATE: JUNE 28TH, 2007 TENTH EUROPEAN WORKSHOP ON EFFICIENCY AND PRODUCTIVITY ANALYSIS

2. Today… • MOTIVATION FOR THE RESEARCH • PROBLEM STATEMENT • RESEARCH PURPOSE, OBJECTIVES, AND HYPOTHESES • DATA AND MODELING ISSUES • IMPLEMENTATION EXAMPLE • CONTRIBUTION OF THE RESEARCH

3. PERFORMANCE MEASUREMENT Surveys by National Council on Public Works Improvement and American Society of Civil Engineers indicate a substantial deterioration in the road system from 1988 to 2005. • The Federal Highway Administration endorsed “asset management” to be the future approach of road maintenance for all state departments of transportation. • Capital investment to build. • Maintenance to upkeep what is built. For the last two decades, the road maintenance concept has been gaining tremendous attention.

4. Performance-based: The risk of deficient design is imposed on the contractor. The contractor often seeks innovative construction methods to make profit under fixed fee payment. Virginia Department of Transportation (VDOT) has been one of the first state DOTs to implement a pilot performance-based asset management contract. Asset management: • Comprehensive and well-structured approach to the long-term management of assets to provide effective and efficient services. • Calls for the utilization of management, engineering, and economic principles in making decisions as to how resources should be allocated.

5. Earlier research has developed a five-component framework to compare effectiveness of the traditional maintenance approaches against the performance-based asset management approaches and implemented it to evaluate the pilot project Quality of Service (Agency/Users Satisfaction) Timeliness of Level of Service Response Overall Performance Cost Safety (Piñero 2003, p. 47)

6. Current Road Maintenance Performance Measurement Systems • Do not investigate the effect of the environmental factors, e.g., climate and location. • Do not investigate the effect of the operational factors, e.g., traffic, load, design-construction adequacy. • Solely focus on “effectiveness” measures, e.g., level-of-service. • Disregard the “efficiency” concept, • e.g., the amount of resources utilized to achieve such level-of-service. For the cases in which comparativeanalyses are made, disregarding such external and uncontrollable factors and using pure effectiveness results may lead to unfair comparisons. Not knowing how “efficient” state DOTs are in being “effective” can lead to excessive and unrealistic maintenance budget expectations.

7. The purpose of this research is, by using the Data Envelopment Analysis (DEA) approach, to develop and implement a generic, replicable, and comprehensive framework that can: • Measure the overall efficiency of road maintenance operations. • Consider the effects of external and uncontrollable factors on such efficiency. • The specific objectives of this research are, through the use of real data, to identify: • The relative efficiency of different units in performing road maintenance services. • The reasons of the efficiency differences between units. • The effects of the external and uncontrollable factors on the efficiency of units. • The benchmarks (peers) and best practices that pertain to the inefficient units. • The hypotheses of this research are as follows: • State DOTs are more efficient when they utilize the performance-based approach instead of the traditional method-based approach for the maintenance of the roads. • A significant portion of the observed efficiency differences between different units can be attributed to the effects of the environmental factors (e.g., climate, location, etc.) and operational factors (e.g., traffic, load, design-construction adequacy, etc.) faced by such units.

8. This presentation focuses on the data and modeling issues relevant to the first seven tasks of the framework

9. To define the input-output variables and uncontrollable factors, the process that is under investigation should be studied.

10. To define the input-output variables and uncontrollable factors, the process that is under investigation should be studied. Output Level-of-Service (LOS) Preventive and Restorative Deterioration Maintenance Works in the favor Works against Consumes inputs Consumes outputs Affected by uncontrollable factors Inputs: Labor Material Equipment

11. This research deals with all of the asset items present within the right-of-way fences. However, this presentation will focus only on the bridges.

12. Defining Input-Output Variables for the Maintenance of Bridges METRICS Input Metrics to Input Metric to Establish Input Establish Cost Process Output Metric Technical Efficiency Efficiency Labor Hours Bridges’ National Bridge Inventory Cost Quantity of Material Maintenance Rating Equipment Hours Outputs*: Benefits made as a result of the process undertaken by the DMUs Inputs: Resources used by DMUs Output Inputs Process Labor Bridges’ Bridges Maintained Material Maintenance to meet LOS Requirements Equipment *: Undesirable outputs are not considered in this research

13. Once everything is considered, the list of input-output variables and uncontrollable factors becomes exhaustive.

14. The DMU is chosen to be the county of Virginia and the time unit of analysis is chosen to be a fiscal year- Both decisions are dictated by the cost data. 1 2 3 4 5 6 7 8

15. The resulting number of DMUs dictated the approach to be used to deal with uncontrollable factors. • One Stage Approaches • Uncontrollable Factors treated as Controllable Variables in the DEA Model • Uncontrollable Factors treated as Uncontrollable Variables in the DEA Model • Uncontrollable Factors used to Develop Categories of DMUs to be included in the DEA Models • Continuous Uncontrollable Factors used to Restrict the Peer Reference Set • Multi Stage Approaches • Uncontrollable Factors used to Perform Regression Analysis over the Obtained Efficiency Scores • Parameters Obtained by the Regression Analysis used to Build an Overall Environmental Harshness Index • Uncontrollable Factors used to Perform Bootstrapped Regression Analysis over the Obtained Efficiency Scores

16. There is a number of approaches to refine the comprehensive lists of controllable variables and uncontrollable factors.

17. To refine the comprehensive list of variables, aggregation method was used. Outputs • Undesirable outputs are out of the scope of this research • Critical output variables are Deck, Superstructure, Substructure, and Slope/Channel Protection • No cost data for Slope/Channel Protection is provided by VDOT • Deck, Superstructure, and Substructure are combined into one variable by using the weighting structure developed and implemented by Virginia Tech as agreed by VDOT Inputs • Using the research by Dadson (2001), 16 input variables (Variable 2- Variable 4 and Variable 6-Variable 18) can be combined into a single input variable that represents all of those to a great extent

18. Dadson identified the effects of the VA Environmental Regions on the deterioration of the bridges. TW: Tidewater EP: Eastern Piedmont WP: Western Piedmont N: Northern CM: Central Mountain SM: Southwestern Mountain

19. As a result of the refinements, total number of variables (controllable and uncontrollable) reduced from 26 to 4.

20. Before showing the implementation example of the developed framework for bridges, it is essential to discuss the data. Level-of-Service Data • Provided by Federal Highway Administration. • Federally-mandated program. • Used to establish investment requirements, to develop data summaries at the national level for reports to Congress, and to respond to inquiries from entities. Cost Data • Provided by VDOT as obtained through Financial Management System II. • Per fiscal year, per county. • Includes only preventive maintenance and restorative maintenance expenditures. No rehabilitative maintenance expenditure is included. • No district or central office overhead is included.

21. Data mining and data cleaning were performed.

22. The data was needed to be converted into the format suitable to represent the variables. Moreover some rearrangements had to be made in the data to make it meet the structuring requirements of the DEA model.

23. A number of final refinements accompanied the DEA model selection and run. • BCC Model is selected to be utilized as it is believed that the bridge maintenance process is more likely to experience variable returns to scale. Change in Overall Bridge Condition Total Area Served Regional Effect Variable Cost Modified Change in Overall Bridge Condition DMU TAS DMU REGIONAL EFFECT Cost adjusted for overhead and inflation

24. Frontier Analyst was used to run the chosen DEA model and to obtain the results. • Conclusion 1: All counties are efficient relative to each other. • Conclusion 2: The model, as structured and run above, is not able to discriminate between the DMUs and thus overlooks the relative efficiency differences (inherent within the DMUs) that exist in reality.

25. To test the validity of the alternative conclusions, further analyses were performed. • Reducing the Number of Variables using DEA Based Analyses • Increasing the Number of DMUs

28. Increasing the number of DMUs changed the efficiency scores to a great extent.

30. Given the fact that the value of the cost variable for the DMUs belonging to Roanoke County is extremely low, further analysis (this time excluding the DMUs belonging to the Roanoke County) is needed to verify that Spotsylvania County is indeed extremely inefficient relative to other counties in the data set.

31. Removing the Roanoke County from the analysis affected onlythe efficiency score of the Spotsylvania County.

32. The results of the models need to be validated and verified by the decision makers at VDOT before actually taking any of the actions suggested by the model. • DEA does not directly pinpoint the underlying causes of inefficiencies of DMUs. • The findings of a DEA study are intended to be used as guides for managerial actions and policymaking as calculated targets for inputs and outputs indicate potential performance and efficiency increases for inefficient DMUs.

33. Contribution to the Highway Maintenance literature: A framework that is able to differentiate effective and efficient maintenance strategies from effective and inefficient ones. A framework that is able to take into account: multiple views (e.g., the driver’s versus the maintenance provider’s) important societal goals (e.g., safety) uncontrollable considerations (e.g., climate, location, etc.) Contribution to the DEA / Measurement Science literature: Application of DEA within the engineering discipline Illustration of issues associated with definingandrefining the listofinput-output variables and uncontrollable factors data availability issues modeling issues associated with definingthe appropriate decision making unit the potential impact that efficiency analysis can have on road maintenance improvement Proposed research’s contribution to BOK is twofold: