A Practical Investment Optimization Tool by Means of Certainty-Uncertainty Searches

1. A Practical InvestmentOptimization Tool by Means of Certainty-Uncertainty Searches A Presentation for the Transportation Data Palooza: A Showcase ofInnovative Technology Solutions at the United States Department of Transportation Headquarters, Washington DCBrian G ChowMay 9, 2013

2. DOT Has Long Used Benefit-Cost Methodology for its Investment Analysis and Optimization • For project selection and optimization, DOT performs standardized investment/performance analysis, e.g. • Highway Economic Requirement System (HERS) • National Bridge Investment Analysis System (NBIAS) • Transit Economics Requirements Model (TERM) • The foundation of these analytical tools is a benefit-cost methodology

3. Benefit-Cost Methodology Can be Enhanced by Casting in a Portfolio Framework • Methods currently used by DOT can be improved by adding features to better deal with the post-9/11 environment and a stressful budget • Need to express results in confidence level in order to address uncertainties in input data, future outcomes, and risk • Current DOT methods based on expected values and supplemented by sensitivity analysis can be inadequate • Need to allow the selected projects to meet multiple requirements individually • Current DOT methods tend to score and sum a potential project’s contributions to various requirements in “oranges,” “applies,” etc., risking that selected projects are all great in providing “oranges” but together still providing too few “apples”

4. The Essential Difficulty of Portfolio Optimization Under Uncertainty • Difficult to find the optimal portfolio among so many possible portfolios • Number of possible portfolios grows exponentially as the number of projects grows merely linearly • Computers in the foreseeable future cannot search for the optimal portfolio by brute force

5. Approximations Used in Traditional Methods Are Problematic for Real-World Investment Optimization • While traditional methods work well in a certainty world, they are ill-equipped to perform investment selection and optimization in a real-world, which is rife with uncertainties • A popular traditional method: Expected Value with Sensitivity Analysis • Pre-select about 10 possible portfolios; go through uncertain scenarios to find the most robust portfolio; and finally check it with sensitivity analysis and what-if analysis • Since there are typically billions or trillions of possible portfolios, the chance that the optimal portfolio is among the chosen ten is practically nil • Other traditional methods place unrealistic limitation on the numbers of decision variables, uncertain parameters and uncertain scenarios

6. RAND’s PortMan Has a New Approach to Perform Investment Optimization under Uncertainties • Simulate 10,000 future states of the world (FSW) by making random combination of the uncertain parameters • Each generated FSW is a certainty state • Use a mixed-integer linear programming model to automatically identify the local optimal solution or portfolio among trillions of possible solution for a given certainty state • Use multiple straightforward search rules, one at a time, to iteratively going through Steps 1 and 2 to arrive at the global optimal portfolio

7. One of the Search Rules Used • Select the project that is most frequently chosen among the 10,000 local optimal solutions as the first project (P1) in the global optimal solution • This is like forming a team for a job that is full of uncertainties as to what actual tasks will have to be performed • The idea behind forming an optimal team is to start by picking the most valuable player first • Generate another 10,000 FSWs but insisting that P1 is in the global optimal solution and select the most frequently chosen project among these 10,000 FSWs (each with P1) as the second project (P2) for the global optimal solution • The idea is to select the second team member to best complement the first team member • Repeating Step 2 for additional projects until the budget for developing and/or implementing selected projects is fully committed

8. Optimal Investment Portfolio with PortMan Running in Uncertainty Mode(based on 10,000 future-state-of-the-world runs for each data point) • Total remaining lifecycle budget ($35B) is the sum of the total remaining S&T budget ($2B) and the total implementation budget ($33B) • If a given portfolio meets all requirements and constraints within budgets in 9100 out of 10,000 FSWs, the feasible percentage is 91% Note: RLCC is the remaining lifecycle cost; S&T is science and technology; Feasible percentage is a proxy for confidence level Source: Chow et al., Toward Affordable Systems II, MG-979-A, 2011

9. Comparison of PortMan with Two Traditional Methods-I • Benefit/Cost Ratio • For each project, divide the total contributions to all requirements by the remaining S&T budget • Optimal portfolio is composed of projects with the highest ratios until the total remaining S&T budget is fully committed • This optimal portfolio is run to determine its feasible percentage under uncertainty (i.e. among 10,000 FSWs)

10. Comparison of PortMan with Two Traditional Methods-II • Certainty Model • A mixed-integer programming model is used with expected parameters as inputs, subject to a given total remaining R&D budget and a total remaining lifecycle budget (R&D plus implementation) • The optimal portfolio for this certainty case is obtained with an objective function for maximizing the total project values (i.e. contributions) • This optimal portfolio is run to determine its feasible percentage under uncertainty (i.e. among 10,000 future-state-of-the-world runs)

11. The Best ATOs to Select Are Not Necessarily the Ones with High Benefit-Cost Ratios Note: ATO=Army Technology Objective, Army’s highest priority S&T projects The selection cannot be derived from an analysis of benefit-cost ratios—a PortMan-type model is essential

12. At Equal Cost, PortMan Does No Worse and Often Does Better Than Traditional Methods under Uncertainty At equal confidence level, PortMan does not cost more and often costs less than traditional methods to meet all requirements under uncertainty

13. Advantages of This New Approach • This new approach allows DOT to incorporate real-world situations (uncertainties) into its investment analysis, selection and optimization • It tells under what situations, • current benefit-cost based methods can continue to be used as good approximations • current methods needed to be revised and how • Relative to highly mathematical search algorithms, the straightforward search rules used in this approach • Are much easier to implement • Are much easier to understand why such a rule can find the (global) optimal portfolio • Are much easier to see the flaw of a rule so as enabling the design of complementary rule(s) to mitigate the flaw PortMan has been used on past RAND projects and currently is being used for three ongoing projects sponsored by the Army, the National Institute of Justice, and the Centers for Disease Control and Prevention. Perhaps, it would be of interest to DOT as well

14. Free Reports: Downloaded at Links below or Requested from chow@rand.org • Chow, Brian, Portfolio Optimization by Means of Multiple Tandem Certainty-Uncertainty Searches: A Technical Description, RR270-A/OSD,2013(http://www.rand.org/pubs/research_reports/RR270.html) • Chow, Brian, Richard Silberglitt, Caroline Reilly, Scott Hiromoto, and Christina Panis, Choosing Defense Project Portfolios: A New Tool for Making Optimal Choices in a World of Constraint and Uncertainty, RB-9678-A, 2012 (http://www.rand.org/pubs/research_briefs/RB9678.html) • Chow, Brian G., Richard Silberglitt, Caroline Reilly, Scott Hiromoto, and Christina Panis, Toward Affordable Systems III: Portfolio Management for Army Engineering and Manufacturing Development Programs, MG-1187-A, 2012 (http://www.rand.org/pubs/monographs/MG1187.html) • Chow, Brian, Richard Silberglitt, Scott Hiromoto, Caroline Reilly, and Christina Panis, Toward Affordable Systems II: Portfolio Management for Army Science and Technology Programs Under Uncertainties, MG-979-A, 2011(http://www.rand.org/pubs/monographs/MG979.html) • Chow, Brian, Richard Silberglitt, and Scott Hiromoto, Toward Affordable Systems: Portfolio Analysis and Management for Army Science and Technology Programs, MG-761-A, 2009 (http://www.rand.org/pubs/monographs/MG761.html)