FHWA 2008 TRANSPORTATION PLANNING COOPERATIVE RESEARCH (DTFH61-08-R-00011)

1. Practical Implications of Finding Consistent Route FlowsHillel Bar-GeraPurdue University, and Ben-Gurion University of the Negev, IsraelYu Nie and David BoyceNorthwestern UniversityMay 20, 2009 FHWA 2008 TRANSPORTATION PLANNING COOPERATIVE RESEARCH (DTFH61-08-R-00011)

2. Acknowledgements Sponsor and study participants: FHWA Office of Environment and Planning and six collaborating transportation planning organizations ________________________________________________________________________________________________________________ Other research team members: Yang Liu, Yucong Hu ________________________________________________________________________________________________________________ Volunteers:Jeffrey Casello, Birat Pandey, Robert Tung ________________________________________________________________________________________________________________ Software vendors:Caliper, Citilabs, INRO, PTV ________________________________________________________________________________________________________________ The authors alone are responsible for the content and views expressed in this presentation.

3. Travel forecasting challenges • Influence decisions, be useful! • Data, data and data, particularly travel times. • Model choice and model assumptions. • Calibration and validation. • Computational quality challenges: • Sufficient precision (convergence) for scenario comparisons; • Reasonable and consistent route flows.

4. Comparing total link flows and link costs between FW (10 iterations) and TAPAS for the Chicago regional network

5. Distribution of deviations in link flowsfor the Chicago regional network

6. Distribution of deviations in link flowsfor the Chicago regional network

7. Solutions evaluated in this study Six solutions for Chicago are evaluated in this study, produced by six tools: CUBE, EMME-LA, TransCAD-FW, TransCAD-OUE, VISUM (RB), and TAPAS. Commercial tools that are not evaluated are: EMME-PG (RB); ESTRAUS (FW, OB); SATURN (FW, OB, RB); VISUM-Lohse (FW), VISUM-LUCE (OB).

8. Precision of evaluated solutions • All evaluated solutions are converged to 1E-4. • A very precise solution (1e-12) produced by TAPAS is used as a reference, when needed. • Definitions of “convergence” are not identical, but are within the same order of magnitude. • Convergence of 1e-4 is chosen because: • Scenario comparisons require at least this level of convergence. • All commercial tools can achieve it. • This is the current “best practice.”

9. Precision of total link flows Comparison of total link flows to precise link flows

10. Distribution of residuals of total link flow

11. Comparison of convergence of research tools for the Chicago regional network

12. Precision of solutions used in this study • It is important to separate the discussion of precision of specific solutions from the discussion of precision of methods. • All solutions have similar level of precision, with residuals less than 10 vph on the majority of links. • Proper precision comparison between methods is beyond the scope of the current project. • Precision performance of FW-type tools (CUBE, EMME-LA, TransCAD-FW) is very different from quick-precision tools (TransCAD-OUE, VISUM).

13. Multiple UE route flow solutions A 2 100 40 40 160 4 D 1 120 60 120 B 3

14. Who needs route flows? • Multi-class models • “Select Link” Analysis: determine the distribution of link flows by their origins and destinations • Estimation of OD flows from link flows • Derivation of OD flows for a subarea of a region, e.g. for micro-simulation • License plates surveys: • Validate model results against survey data; • Design a survey to capture travelers at least twice, or as much travel as possible.

15. OD flows through North Ave. Bridge WB Each point represents vehicle flow per hour for one OD pair. X - reference solution (TAPAS-3057 ODs) ; Y - evaluated solution (RG = 1e-4)

16. OD flows through North Ave. Bridge EB Each point represents vehicle flow per hour for one OD pair. X - reference solution (TAPAS-3,376 ODs) ; Y - evaluated solution (RG = 1e-4)

17. OD flows through Harlem Ave. SB Each point represents vehicle flow per hour for one OD pair. X - reference solution (TAPAS-4,752 ODs) ; Y - evaluated solution (RG = 1e-4)

18. OD flows through Harlem Ave. NB Each point represents vehicle flow per hour for one OD pair. X - reference solution (TAPAS-5,034 ODs) ; Y - evaluated solution (RG = 1e-4)

19. How to choose a single route flow solution? The condition of Proportionality Same proportions apply to all travelers facing a choice between a pair of alternative segments. Consider the pair of segments [1,2,4] and [1,3,4]. First segment proportion is 40/(40+120)=1/4. A 2 100 40 40 160 4 D 1 120 120 60 B 3

20. How to choose a single route flow solution? The condition of Proportionality Same proportions apply to all travelers facing a choice between a pair of alternative segments. Consider the pair of segments [1,2,4] and [1,3,4]. First segment proportion is 40/(40+120)=1/4. 25 A 2 100 40 40 160 4 D 1 75 120 120 60 B 3 For travelers from A to D the proportion is 25/(25+75)=1/4.

21. How to choose a single route flow solution? The condition of Proportionality Same proportions apply to all travelers facing a choice between a pair of alternative segments. Consider the pair of segments [1,2,4] and [1,3,4]. First segment proportion is 40/(40+120)=1/4. 15 A 2 100 40 40 160 4 D 1 120 120 45 60 B 3 For travelers from B to D the proportion is 15/(15+45)=1/4.

22. The condition of proportionality Same proportions for the two segments. Origin and destination do not matter. Previous or subsequent decisions do not matter. By proportionality, flow on designated route is: 200 * (150/200) * (160/200) * (180/200) = 108

23. The condition of proportionality • Reasons: • Simple, reasonable, consistent, stable, and therefore useful. • Proportionality is testable. • Are there any other practical suggestions? Implications: The set of routes should be consistent, meaning that any route that can be used while keeping the same total link flows, should be used. “No route is left behind (without a reason).”

24. North Lake Shore Drive Paired Alternative Segments 5075 Segment 1 Segment 2 North Lake Shore Drive 5435

25. Paired Alternative Segments near Lake Shore Drive Each point represents vehicle flow per hour for one OD pair. X - Segment 1; Y - Segment 2; all solutions converged to RG = 1e-4.

26. Paired Alternative Segments near Lake Shore Drive Each point represents vehicle flow per hour for one OD pair. X - seg. 1 flow + seg. 2 flow; Y – Log (seg. 1 flow / seg. 2 flow)

27. North Avenue Bridge Paired Alternative Segments 8032 Segment 2 10344 Segment 1 North Avenue Bridge

28. Paired Alternative Segments near North Ave. Bridge Each point represents vehicle flow per hour for one OD pair. X - Segment 1; Y - Segment 2; all solutions converged to RG = 1e-4.

29. Paired Alternative Segments near North Ave. Bridge Each point represents vehicle flow per hour for one OD pair. X - seg. 1 flow + seg. 2 flow; Y – Log (seg. 1 flow / seg. 2 flow)

30. Effect of convergence on consistency in TAPAS Each point represents vehicle flow per hour for one OD pair. X - seg. 1 flow + seg. 2 flow; Y – Log (seg. 1 flow / seg. 2 flow)

31. Conclusions: FW-type methods • Solutions tend to satisfy the condition of proportionality, although deviations do occur. • Route set consistency is problematic due to small flows on non-optimal routes. • To achieve the precision needed for scenario comparisons (1E-4 or better), hundreds of iterations may be necessary, implying relatively long computation times. 

32. Commercial quick precision methods • Available in VISUM and TransCAD; soon in EMME and CUBE; • Important for scenario comparisons of total link flows; • Do not satisfy proportionality and route consistency, which could be problematic in select link and similar analyses. 

33. Research progress A new method, TAPAS, has been developed: • Quick-precision assignment • Reasonably consistent set of routes, mainly at higher levels of convergence • Satisfaction of the condition of proportionality • Functionality is limited to research purposes • Additional experiments are on-going

34. Summary • Route flows are used often in practical applications, at various levels of aggregation. • Results of select link and similar analyses are quite different from software to software. • The set of routes should be consistent: “No route is left behind.” • The assumption of proportionality ensures unique, consistent and stable route flows.

35. Software vendor reactions • Quick precision is more important than proportionality. Select flows are more meaningful at aggregate levels; e.g., flow through a selected link on other nearby links, rather than by OD. (PTV) • Lack of proportionality for a tiny amount of traffic is insignificant. (Citilabs) • Proportionality is a potentially useful mechanism for rendering path flows unique; applications for multi-class assignments as well as behavioral or empirical validation would lend it credibility. (INRO)

36. Discussion: What makes a model useful? • Proper sensitivity to policy decisions • Reasonably accurate (i.e. realistic) predictions • Ability to obtain needed data for inputs, as well as for calibration and validation • Stability, repeatability, and consistency • Computational efficiency • Insights, understanding and accessibility • Convincing

37. Slide Notes

38. Slide 1: This research is about route flows in the static deterministic UE model. The research is funded by FHWA. It began in September 2008, and is scheduled for one year. The research is conducted by Marco Nie, David Boyce, and Hillel Bar-Gera. Slide 2: Our purpose in this research is to support decisions about future improvements to travel forecasting practices. I am glad to say that the software vendors, who are important leaders of progress in this field, took a similar point of view. They offered us help in various ways, including many useful and productive comments, which we highly appreciate. Of course, this does not mean that they necessarily agree with the content of this presentation. We also want to acknowledge the contribution of several people that worked very hard with us to prepare the results presented here: Yang Liu and YucongHu, Northwestern University Jeffrey Casello, Assistant Professor of Planning and Civil Engineering, University of Waterloo, Ontario BiratPandey, Senior Engineer, PBSJ, Austin, Texas Robert S. Tung, RST International, Inc.

39. Slide 3: The main focus of this research is finding route flows, which is a computational challenge. We realize that you, as travel forecasting practitioners, devote most of your time and efforts to address other important challenges. Among the many difficult decisions you need to make, you need to choose which assignment method to use and for how long to let it run. This is why practitioners should be aware of assignment computational challenges. The main computational challenges in the static, deterministic, user equilibrium (UE) model are precision and route flows. In some ways these two issues are quite intertwined, while in other ways they are completely orthogonal. In particular, as far as we know, not much has been done in practice regarding route flows. On the other hand, there has been a remarkable change, almost a revolution, regarding precision over the last five years or so. From our point of view, when we asked practitioners five years ago how many iterations they use, the answers were 10, or 20, or sometimes 5. If we suggested that more iterations might be helpful, the response was that this would be a complete waste of computer time. This presentation starts with evaluation of the precision of a solution obtained by the FW algorithm in 10 iterations. During the last year we showed this evaluation to several practitioners, and they immediately jumped and said that 10 FW iterations do not provide sufficient precision for any analysis. We think that in order to put in context the issue of route flows, it should be discussed together with precision. This is why nearly half of this presentation will be devoted to precision, and only then we will discuss route flows.

40. Slide 4: This is a comparison of a solution obtained by the FW algorithm in 10 iterations with a very precise solution obtained by TAPAS, which is converged to a Relative Gap of 1e-12. In theory, total link flows and link costs are uniquely determined by a UE assignment. Indeed we see a good match between the link flow results, but it is not perfect; the link cost results are more problematic. Slide 5: We can examine the comparison “under the microscope” by considering the distribution of the differences between the two solutions. Notice that the difference in flow on the horizontal axis is in log scale. We see that a difference of 10vph or more, which is not trivial, occurs for 40% of the links. In many applications this precision is not enough, so more iterations are needed. Slide 6: As the number of iterations increases, the precision increases, and the differences become smaller, as expected. The needed level of precision depends on the application. One way to choose is to pick a threshold and choose a solution with sufficiently small tail beyond that threshold. If the threshold is 10 vph, then 10-iterations solution is clearly not good enough, but 1000-iterations solution probably is. Slide 7: FW – The Frank-Wolfe or Linear Approximation (LA) method RB – route-based method OB – origin-based method Slide 9: As you can see, the match in total link flows, with the reference TAPAS solution (1e-12), is quite good in all six evaluated solutions.

41. Slide 8: The level of convergence is defined in term of the Relative Gap (RG). Slide 9: As you can see, the match in total link flows, with the reference TAPAS solution (RG = 1E-12), is quite good in all six evaluated solutions. Slide 10: Examination of the differences in total link flows from the reference solution shows that all six solutions are fairly precise, with only a small “tail” of differences above 10 vph. This evaluation gives us the confidence that the conversion of inputs to all the software was done properly, which is not a trivial thing, and that the subsequent comparison of select link analyses results are valid. According to this figure the precision of all six solutions is in the same order of magnitude. This does not mean that the methods have similar precision performance, because in order to compare methods we need to consider CPU time. Performing such a comparison between commercial software in a proper manner is far beyond the scope of this project. To give you an idea about the possible differences between methods we show here a comparison of convergence vs. CPU time for several research tools. Slide 11: We can see here that modest levels of precision can be obtained fairly quickly by several different methods, including FW. When higher precision is needed, the computation time for FW increases dramatically, while other methods can achieve high precision fairly quickly. We refer to such methods as “quick-precision” methods. Slide 12: To summarize our discussion about precision, here are the main conclusions.

42. Slide 12: To summarize our discussion about precision, here are the main conclusions. Slide 13: It is quite well known that under the UE assumption route flows are not unique. Here is a simple example to explain why. Suppose that the total link flows indicated here represent perfect UE solution, for which the two segments from 1 to 4 have exactly the same cost. If we switch one vehicle from A that uses the segment through 2 with a vehicle from B that uses the segment through 3 the total link flows remain the same, so link costs remain the same and the perfect equilibrium situation also remains. In this table we can see 3 different route flow solutions; and all of them correspond exactly to the same total link flows shown above. Slide 14: In many practical applications, for example in most cost-benefit analyses, we are interested only in the full aggregation of route flows to total link flows. It is quite rare to find practical application where fully disaggregate route flows are needed. But there are quite a few applications where various different intermediate levels of aggregation are needed. A few of them are listed here. The important point is that different route flow solutions may lead to different answers in each of these partially aggregated analyses. Slide 15: One of the most typical partially aggregated analyses is select link analysis. In this analysis we want to know the breakdown of a flow on a single link by OD pair. We can see here a comparison for one link in the Chicago network between the six evaluated solutions and the reference solution. OD flows on both axes are in log scale. Points along the axes represent values below 1E-4, including zeros. On the left you see the three FW-type methods, and on the right you see the quick-precision methods.

43. Slide 15 (continued): If we compare the number of OD’s identified by the various solutions, these numbers are quite different from each other. (In the reference solution there are 3,057 OD’s that use this link.) So clearly the sets of OD’s using this link are quite different in all the solutions. If we focus on the comparison of OD flows through this link, and particularly the larger flows, evaluated solutions from the three FW-type methods as well as from TransCAD OUE and TAPAS are quite similar to the reference solution, while the Visum solution is slightly different. Slide 16: Considering the same link in the other direction we see that a match with the FW type methods and a mismatch with the quick-precision methods. (In the reference solution there are 3376 OD’s that use this link.) Slide 17: For a completely different link on Harlem Ave. we get fairly similar patterns. (In the reference solution there are 4752 OD’s that use this link.) Slide 18: Considering the same link on Harlem Ave. in the opposite direction we find a mismatch with all the methods. (In the reference solution there are 5034 OD’s that use this link.) This small sample of 4 links out of 40,000 was chosen fairly arbitrarily, and is not necessarily “statistically representative.” Even so, it is enough to conclude that differences between solutions at the select link analysis level do occur.

44. Slides 19-22: It would be useful to find a way to choose one specific solution out of all the many options. One way to do that is by the condition of proportionality, which is explained in these four slides. Slide 23: The main reasons to adopt proportionality are: 1) a reasonable condition that is easy to understand, implying consistent treatment which may be important when equity issues are present, and 2) provision of stable solutions with respect to model inputs. All of these properties make the resulting model quite useful. As we will see soon, it is possible to test whether any particular method satisfies proportionality or not. And the only other existing alternative is to make a completely arbitrary choice. An important implication of proportionality is that any route that can be used under the UE condition, should be used. For example, in the previous slide there are 8 routes; under proportionality all of them are used. So a precondition to satisfying proportionality is to make sure that “no route is left behind,” unless of course it is not a minimum cost route. We refer to this property of the set of routes as “consistency.” Slide 24: The assumption of proportionality is based on pairs of alternative segments. In the Chicago model there are 5000 basic pairs of alternative segments, which can be used to construct all other pairs of alternative segments. Here is one of them.

45. Slide 25: In each evaluated solution we found the breakdown of the flow on the two segments by OD. So each point here represents a single OD, and the horizontal and vertical axes represent the flows on segments 1 and 2 respectively. If the same proportions apply to all OD pairs, all the points should fall on a straight line (with a slope of 45 degrees). Both axes are in log scale. Points along the axes represent values below 1E-4, including zeros. A fairly straight line is observed for all FW-type solutions, especially for higher flows, as well as the evaluated TAPAS solution (RG = 1E-4). For TransCAD OUE we see three main lines, each line corresponds to a different origin. This means that within each origin proportionality is maintained, but between origins proportions are not the same. The Visum solution in this case is quite extreme, where only one OD pair uses both segments. Slide 26: Another way to look at the same data is shown here, where the horizontal axis shows the sum of flow on both segments in log scale, and the vertical axis shows the log of the flow ratio. Under proportionality the flow ratio should be constant, so the log of the flow ratio should be constant, so all OD pairs should be on a horizontal line. This is pretty much the case for TAPAS. It is more or less the case in the FW-type solutions for the higher flow values. For the VISUM solution most OD pairs have all their flow either on segment 1, with log ratio of infinity, or on segment 2, with log ratio of minus infinity. So clearly proportionality does not hold. In the TransCAD solution we see a group of OD’s that use only segment 1, and three other groups corresponding to three origins, each having its own ratio.

46. Slide 27: Here is another pair of alternative segments we examined. Slide 28: Again we see linear lines for all FW-type methods, but not for the three quick-precision methods, including the evaluated TAPAS solution (1E-4). Slide 29: Using the log ratio plots further enhance the same conclusions regarding FW type solutions. All three quick-precision methods suffer from substantial inconsistency, as many OD’s use only one segment out of the two. When OD’s use both segment, in the TAPAS solution the proportions are the same, while in the commercial quick-precision methods each OD has its own proportion. The two pairs of alternative segments do not necessarily represent all 5000 other pairs in this model. They do offer an idea for what might be expected in other cases. Slide 30: Consistency in TAPAS solutions improves considerably with convergence. Nearly perfect consistency is shown here at relative gap around 1E-9, and a noticeable progress is shown already at relative gap around 1E-7. Notice that reaching these higher levels of precision does not require too much computation time. Preliminary experiments with commercial quick-precision tools did not demonstrate improvement in consistency or proportionality at higher levels of precision, but additional exploration is needed to verify these observations.

47. Slide 33: At present, TAPAS exists only as a research code. As such its functionality is limited to research needs. It did not go through the extensive testing expected from commercial products. The results from TAPAS demonstrate the potential of incorporating proportionality into assignment methods. The results are not perfect, so there are possibilities for future improvements. Slide 35: • We agree that quick precision is more important than proportionality, but proportionality is also important. • We plan to study other levels of aggregation in the future. • We agree that small flow values are less important, the problem in practice is how to tell whether the flows are “small” and should be attributed to solution imprecision, or whether they are not so small and represent something else. • We certainly appreciate the positive reaction from INRO. Slide 36: We think that the main consideration when choosing a model is its usefulness. The main criterion for usefulness is the model ability to support decision processes. All other criteria are derived from this one. Reasonable realism is obviously important. All else being equal, a more realistic model should be preferred. However, there are many other considerations, so in most cases not all else is equal. As a result, in some cases a more realistic model is not more useful. More important to our discussion, if several methods produce solutions that are equally realistic, other criteria should be considered to choose the most useful method. We believe that all other considerations, and particularly the ability to understand why the method chose a specific solution, and the ability to explain that to others, indicate that solutions that follow the proportionality condition are more useful. Thank you.