SpaceLiner 100 Propulsion Task Force Candidate Technology Evaluation and Assessment &

1. SpaceLiner 100 Propulsion Task Force Candidate Technology Evaluation and Assessment & Prioritization Workshop Huntsville, Al. April 10-11, 2001 Russel Rhodes NASA KSC

2. Purpose of Review • Provide a review of the “Technology Evaluation Process” developed by the SPST in support of the SL-100 Functional Requirements and Key Objectives. • Focus is on Criteria Development and weighting used in Workshop • Supportive of Civil, DOD, and Commercial needs for Third Generation Space Transportation Systems • Challenge the affected technical community to consider the knowledge and information from this process in developing and justifying the technologies capable of satisfying SL-100 key objectives. • “Example: Two orders of magnitude decrease in costs & 10,000 times safer than today's Space Transportation Sys.”

3. Define Customer Customer Establish Relative Merit of Each “How” What Does the Customer Want? Develop “Whats” Weight the “Whats” Result List of Prioritized and Weighted “Hows” Which Produce the “Whats” That the Customer Wanted Determine the “Hows” for Each “What” Synergy Group Synergy Group Establish Correlation of Each “How” to Every “What” Synergy Group Prioritized List of Propulsion Technologies Likely to Achieve Goals Evaluate Each Candidate on Each “How” Tech. Candidates The Evaluation Tool Development Process for Technical Benefit

4. The Attributes of a Space Transportation System Operating How do we improve in all these phases? Programmatics

5. The Attributes Prioritized 3 fold scoring used - importance, plus need to improve and where are we now. But these are still qualitative “whats” … we required a more useful expression of “how”...

6. Measurable Criteria - “How” Used an Iterative Structured Process Process forces the consideration of improvement in customer wants

7. SpaceLiner 100 Propulsion Task Force SPST Task Charter • SPST Approach Complies with National Space Policy: • Attributes are Anchored to the National Space Policy: • SPST Flow Diagram Responds to National Space Policy: • SPST Will Provide Recommended Functional Requirements Necessary to Develop Space Transportation Architectures, Concepts, and Systems that Surface the Technology Needs Required to Realize the Objectives of SpaceLiner 100 “3rd Generation RLV Feasible Concepts” • SPST will Assist NASA in the Development of a Prioritized Portfolio of Technologies Capable of Reaching the SpaceLiner 100 3rd Generation RLV Objectives SPST SPACELINER 100 PROPULSION TASK FORCE ”FUNCTIONAL REQUIREMENTS SUB-TEAM" OBJECTIVE: • Develop SL-100 functional requirements capable of achieving the 3rd generation RLV objectives • Develop the Technology Evaluation Criteria and Weights for both Technical Benefit and Programmatics for use in the AHP model to be used at a Technology Evaluation Workshop while anchoring on the existing SPST data base where possible. This sub-team accepted Garry Lyles challenge to further develop and take accountability for his algorithm for “Systems Approach to Safety, Reliability, and Cost” and anchor the Evaluation Workshop Criteria to the algorithm

8. SpaceLiner 100 Propulsion Task Force In addressing this objective, SpaceLiner 100 Propulsion Technology 3rd Generation RLV Evaluation Criteria--The First Step: ---Select the correct team make-up ---Must be balanced with Designers, Operators, Managers, and Technologists ---Required to bring the needed experience and knowledge together for consensus building to provide quality and creditable process

9. Russel Rhodes, NASA-KSC - Lead Uwe Hueter, NASA-MSFC Walt Dankhoff, SAIC Bryan DeHoff, Aero.Tech.Serv. Glen Law, Aerospace Corp. Mark Coleman, CPIA Robert Bruce, NASA-SSC Ray Byrd, Boeing-KSC Clyde Denison, NGC Bill Pannell, NASA-MSFC Dan Levack, Boeing/Rocketdyne Bill Escher, SAIC Pat Odom, SAIC David Christensen, LMCO Jim Bray, LM-MAF Tony Harrison, NASA-MSFC Keith Dayton/John Robinson, Boeing Co Andy Prince, MSFC Carey McCleskey, NASA-KSC Jay Penn, Aerospace Corp. John Hutt, NASA-MSFC SpaceLiner 100 Propulsion Task ForceFunctional Requirements Sub-Team Membership • CUSTOMER PROVIDING EVALUATION INPUT: • Uwe Hueter, NASA-MSFC

10. 71% Influence (29% Other Factors) Safety Operable Responsive Recurring Cost SpaceLiner-100 Propulsion Task ForceSpaceLiner 100 Key AttributeInfluence Relationships Dependable (Inherent Reliability) 41% (59% Other) Spaceliner Goals 52% (48% Other) Lyles Algorithm Influence Diagramming Approach 33% (67% Other) Influence Diagramming 44% 56% Stresses importance of the Dependability Attribute Objectives Cost on achieving SpaceLiner Goals (Life Cycle Cost) Non-Recurring Cost Management Visibility of Influence Achieved when using Weighted Design Criteria for Technology Prioritization ( The How’s to achieve the What’s Desired )

11. SHORT SCHEDULE LOW RISK R&D LOW COST R&D R&D ATTRTIBUTES DUAL USE POTENTIAL BENEFIT FOCUSED NON-RECURRING INVESTMENT LOW NON-RECURRING COST TECHNOLOGY OPTIONS INVESTORS INCENTIVE LOW RISK DDT&E DDT&E ATTRIBUTES FLEET PURCHASE LOW DDT&E ACQUISITION COST SHORT SCHEDULE LOW LIFE CYCLE COST L I F E C Y C L E C O S T DEPENDABLE INHERENT RELIABILITY ATTRIBUTES KEY R&D DDT&E OPERS OPERS OPERS COST FOCUS OPERABLE LOW RECURRING COST OPERATIONS ATTRIBUTES RESPONSIVE 100X CHEAPER COST, $/LB SAFE GEN3 GOALS 10,000 X SAFER R&D Investment Influence on Achievement of SpaceLiner-100 Key Objectives Operations and DDT&E Integrated Key Attributes Influence Relationships

12. SpaceLiner-100 Propulsion Task Force SpaceLiner-100 Assessment/Prioritization Process & Criteria Sub-Team Products for SL-100 3rd Gen. RLV • Developed Functional Requirements for SL-100 3rd Generation RLV • Developed the Influence Diagram Algorithm with SPST 3rd Gen. RLV data base focusing on SL-100 Goals • Established and weighted desired attributes focusing on the importance and need to improve to achieve the SL-100 performance requirements • Established and weighted measurable design criteria and provided paretos • Identified and selected the top 26 good discriminating design criteria with weights for use at evaluating top level technologies

13. SpaceLiner-100 Propulsion Task Force • SpaceLiner-100Assessment/Prioritization Process & Criteria • Sub-Team Products for SL-100 3rd Gen. RLV "Continued" • Established programmatic factors and weighted measurable quality characteristics for both the Acquisition and R & D Phases and provided paretos • Provided functional requirements, design criteria, and programmatic criteria to Technology Candidate Developers • Provided Technical Benefit (Design Criteria) and Programmatics Criteria (Acquisition and R & D Phases) for use in the AHP evaluation tool • Developed a definitions reference document to capture terminology used in this work to provide understanding and communications required for process usefulness

14. BENEFIT (TECHNICAL) ATTRIBUTES WEIGHTING Note: Weighting Factors are 1 to 5 with 5 being the most important

15. BENEFIT (TECHNICAL) ATTRIBUTES WEIGHTING

16. BENEFIT (TECHNICAL) ATTRIBUTES WEIGHTINGPRIORITIZED

17. SpaceLiner-100 Propulsion Task ForceSL-100 Propulsion Assessment/Prioritization Process & Criteria Sample Piece of Actual Matrix

18. Pareto of all Design Criteria down to Top 26 good discriminators (*) used in Workshop Process # of active systems required to maintain a safe vehicle (-) 603 2.72% # of different propulsion systems (-) 582 * 2.62% 5.34% # of systems with BIT BITE (+) 542 2.45% 7.79% # of components with demonstrated high reliability (+) 541 2.44% 10.% # of hands on activities req'd (-) 534 2.41% 12.63% # of active components required to function including flight operations (-) 527 * 2.38% 15.01% # of potential leakage / connection sources (-) 527 2.37% 17.39% # of systems requiring monitoring due to hazards (-) 523 2.36% 19.74% System margin (+) 508 * 2.29% 22.03% # of toxic fluids (-) 495 * 2.23% 24.27% % of propulsion system automated (+) 488 * 2.20% 26.47% # of unique stages (flight and ground) (-) 483 * 2.18% 28.64% % of propulsion subsystems monitored to change from hazard to safe (+) 470 2.12% 30.76% # of in-space support sys. req'd for propulsion sys. ( - ) 465 2.10% 32.86% Design Variability (-) 464 * 2.09% 34.95% # of active on-board space sys. req'd for propulsion ( - ) 454 * 2.05% 37.00% On-board Propellant Storage & Management Difficulty in Space (-) 453 * 2.04% 39.04% # of purges required (flight and ground) (-) 428 1.93% 40.97% # of confined spaces on vehicles (-) 427 1.92% 42.89% Technology readiness levels (+) 425 * 1.92% 44.81% # of active ground systems required for servicing (-) 420 1.89% 46.71% # of different fluids in system (-) 404 * 1.82% 48.53% # of checkouts required (-) 403 1.82% 50.34% # of propulsion sub-systems with fault tolerance (+) 398 * 1.79% 52.14% # of inspection points (-) 390 1.76% 53.90% Mass Fraction required (-) 387 * 1.75% 55.64% Hours for turnaround (between launches or commit to new mission) (-) 374 1.69% 57.33% ISP Propellant transfer operation difficulty (resupply) (-) 371 1.68% 59.01% # of pollutive or toxic materials (-) 350 1.58% 60.58% # of expendables (fluid, parts, software) (-) 348 1.57% 62.15% Minimum Impulse bit (-) 332 1.50% 63.65% # of criticality 1 failure modes (-) 329 1.48% 65.13% # of element to element interfaces requiring engineering control (-) 320 1.44% 66.57% Ave. Isp on refer. trajectory (+) 310 * 1.40% 67.97% # of parts (different, backup, complex) (-) 296 1.33% 69.31% #of umbs. req'd to Launch Vehicle ( - ) 276 * 1.25% 70.55% # of engines (-) 274 * 1.24% 71.79% Resistance to Space Environment (+) 268 * 1.21% 73.00% # of physically difficult to access areas (-) 265 1.19% 74.19% # of active engine systems required to function (-) 247 * 1.11% 75.30% Integral structure with propulsion sys. (+) 239 * 1.08% 76.38% Hours to refurbish propulsion system (-) 237 1.07% 77.45% # of manhours (c/o, handle, assemble etc) on system between on and off cycles (Low Cycle Fatigue) or use (High Cycle Fatigue) (-) 229 1.03% 78.48% # of modes or cycles (-) 227 * 1.02% 79.50% # of ground power systems (-) 226 * 1.02% 80.52% Mean time between major overhaul (+) 221 1.00% 81.52% Amount of energy release from unplanned reaction of propellant (-) 219 * 0.99% 82.51% Margin, mass fraction (+) 215 * 0.97% 83.48% Margin, thrust level / engine chamber press(+) 211 * 0.95% 84.43% Transportation trip time (-) 211 * 0.95% 85.38% # of engine restarts required (-) 201 * 0.91% 86.29% SpaceLiner-100 Propulsion Task Force SL-100 Propulsion Assessment/Prioritization Process & Criteria

19. (Quality Characteristic) Chart for Weighting SL-100 Advanced Space TRD-# full scale ground or flight demonstrations TRD-total annual funding by item at peak dollar PA-total system DDT&E concept development TRD-estimated time to reach TRL 6 from start Programmatic Criteria TRD-#related technology databases available (+) TRD-# technology breakthroughs required to TRD-# multiuse applications including space TRD-# of new facilities required costing over Transportation PA-time required to establish infrastructure TRD-# operational effectiveness attributes TRD-# operational effectiveness attributes (Pre-Operational Phases, PA-infrastructure cost: initial system implementation (capital investment) (-) R&D and Aquisition) addressed for improvement (+) TRD-cost to reach TRL -6 (-) develop and demonstrate (-) (schedule of R&D phase) (-) previously demonstrated (+) and implementation cost (-) TRD-Current TRL (+) transportation (+) Requirements (-) required (-) of R&D (-) $2M (-) Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 (Demanded Quality) Attributes WEIGHT Technology R&D Phase Cost 30 9 3 9 9 9 9 9 9 9 3 1 Benefit Focused 30 9 3 1 9 3 3 9 9 3 3 1 Schedule 15 3 1 3 9 9 3 3 9 9 3 1 Risk 15 1 1 3 3 9 9 9 9 9 9 1 Dual use Potential 10 1 3 1 3 1 1 3 9 Program Acquisition Phase Cost 25 9 9 3 Schedule 15 3 3 9 Risk 25 1 1 3 Technology Options 10 3 3 3 Investor Incentive 25 9 9 9 RAW SCORE 600 210 400 750 630 550 750 820 640 390 180 550 550 540 SUM OF RAW SCORES TRD 5920 SUM OF RAW SCORES PA 4550 NORMALIZED SCORE 10 4 7 13 11 9 13 14 11 7 3 12 12 12 SPST / SL-100 Space Propulsion Attributes versus Programmatic Criteria - Matrix / Pareto PA-technology readiness at program acquisition PA-# major new technology development items PA-# items requiring major ground test articles PA-technology capability margin (performance and demonstration (example: new engines) (-) PA-# of other options available (+) (engines, airframes, TPS, etc) (-) milestone: TRL 6 + margin (+) as fraction of ultimate) (+) 15 16 17 18 19 9 3 3 9 3 9 3 3 9 3 9 9 9 9 3 9 9 3 9 3 3 3 3 9 3 750 510 450 900 300 16 11 10 20 7

20. Program Acquisition Phase PA-# major new technology development items (engines, airframes, TPS, etc) (-) 20% 16% PA-technology readiness at program acquisition milestone: TRL 6 + margin (+) PA-time required to establish infrastructure (schedule of R&D phase) (-) 12% PA-total system DDT&E concept development and implementation cost (-) 12% PA-infrastructure cost: initial system implementation (capital investment) (-) 12% 11% PA-technology capability margin (performance as fraction of ultimate) (+) PA-# of other options available (+) 10% PA-# items requiring major ground test articles and demonstration (example: new engines) (-) 7% Technology R & D Phase TRD-# technology breakthroughs required to develop and demonstrate (-) 14% TRD-estimated time to reach TRL 6 from start of R&D (-) 13% TRD-# operational effectiveness attributes addressed for improvement (+) 13% TRD-Current TRL (+) 11% TRD-# full scale ground or flight demonstrations required (-) 11% TRD-cost to reach TRL -6 (-) 10% TRD-# operational effectiveness attributes previously demonstrated (+) 9% TRD-#related technology databases available (+) 7% TRD-# of new facilities required costing over $2M (-) 7% TRD-total annual funding by item at peak dollar requirements (-) 4% TRD-# multiuse applications including space transportation (+) 3% SPST / SL-100 Space Propulsion Attributes versus Programmatic Criteria - Matrix / Pareto Note: TRD - Technology Research and Development Stage PA - Program Acquisition Phase

21. Attributes versus Programmatic Criteria - Matrix / Pareto SPST / SL-100 Space Propulsion TRD-# technology breakthroughs required to develop and demonstrate (-) TRD-# operational effectiveness attributes addressed for improvement (+) TRD-estimated time to reach TRL 6 from start of R&D (-) TRD-# full scale ground or flight demonstrations Required (-) TRD-Current TRL (+) TRD-cost to reach TRL -6 (-) TRD-# operational effectiveness attributes previously demonstrated (+) TRD-# of new facilities required costing over $2M (-) TRD-#related technology databases available (+) TRD-total annual funding by item at peak dollar Requirements (-) TRD-# multiuse applications including space Transportation (+) PA-# major new technology development items (engines, airframes, TPS, etc) (-) PA-technology readiness at program acquisition milestone: TRL 6 + margin (+) PA-infrastructure cost: initial system implementation (capital investment) (-) PA-total system DDT&E concept development and implementation cost (-) PA-time required to establish infrastructure (schedule of R&D phase) (-) PA-technology capability margin (performance as fraction of ultimate) (+) PA-# of other options available (+) SCORE PA-# items requiring major ground test articles and demonstration (example: new engines) (-) 0 5 10 15 20

22. SpaceLiner 100 Propulsion Task Force SpaceLiner 100 Propulsion Assessment/Prioritization Process & Criteria Technology Evaluation Benefits (Technical) Attributes and Associated Design Criteria Benefits (Technical With Sense of Goodness and Normalized Weighting) Affordable / Low Life Cycle Cost Min. Cost Impact on Launch Sys. Low Recurring Cost Low Cost Sens. to Flt. Growth* Operation and Support Initial Acquisition Vehicle/System Replacement Raw %Score Weight No. 49 # of unique stages (flight and ground) (-)4835.3 % No. 75 # of active on-board space sys. req'd for propulsion ( - )4544.9 % No. 78 On-board Propellant Storage & Management Difficulty in Space (-)4534.9 % No. 38 Technology readiness levels (+) 4254.6 % No. 59 Mass Fraction required (-)387 4.2 % No. 54 Ave. Isp on refer. trajectory (+)3103.4 % No. 70 # of umbs. req'd to Launch Vehicle ( - )276 3.0 % No. 58 # of engines (-) 2743.0 % No. 79 Resistance to Space Environment (+)268 2.9 % No. 82 Integral structure with propulsion sys. (+)239 2.6 % No. 85 Transportation trip time (-) 211 2.3 % Dependable Highly Reliable Intact Vehicle Recovery Mission Success Operate on Command Robustness Design Certainty Raw %Score Weight No. 10 # of active components required to function including flight operations (-)527 5.7 % No. 87 Design Variability (-)464 5.0 % No. 14 # of different fluids in system (-) 404 4.4 % No. 60 # of active engine systems required to function (-) 247 2.7 % No. 48 # of modes or cycles (-) 227 2.5 % No. 16 Margin, mass fraction (+) 215 2.3 % No. 18 Margin, thrust level/engine chamber press (+)211 2.3 % No. 64 # of engine restarts required (-) 201 2.2 %

23. SpaceLiner 100 Propulsion Task Force • SpaceLiner 100 Propulsion Assessment/Prioritization Process & Criteria • Technology Evaluation Benefits (Technical) Attributes and Associated Design Criteria • Con’t • Responsive • Flexible • Capacity • Operable • Process Verification • Auto. Sys. Health Verification • Auto. Sys. Corrective Action • Ease of Vehicle/System Integration • Maintainable • Simple • Launch on Demand • Easily Supportable • Resiliency Raw %Score Weight No. 37 # of different propulsion systems (-) 582 6.3 % • No. 66 System Margin (+)508 5.5 % • No. 33 % of propulsion system automated (+) 488 5.3 % • No. 53 # of ground power systems (-)226 2.5 % • Environmental Compatibility • Minimum Impact on Space Environ. • Minimum Effect on Atmosphere • Minimum Environ. Impact all Sites • Safety • Vehicle Safety • Personnel Safety • Public Safety • Equipment and Facility Safety Raw % • Score Weight • No. 5 # of toxic fluids (-) 495 5.4 % • No. 6 # of propulsion sub-systems with fault tolerance (+)398 4.3 % • No. 4 Amount of energy release from unplanned reaction of propellant (-)219 2.4 % • Public Support • Benefit GNP • Social Perception

24. SpaceLiner 100 Propulsion Task ForceIn-Space Propulsion Assessment/Prioritization Process & Criteria Candidate Technologies PROGRAMMATIC Assessment Criteria Program Acquisition Phase PA-# major new technology development items (engines, airframes, TPS, etc) (-) 20 % PA-technology readiness at program acquisition milestone: TRL 6 + margin (+) 16 % PA-time required to establish infrastructure (schedule of R&D phase) (-) 12 % PA-total system DDT&E concept development and implementation cost (-) 12 % PA-infrastructure cost: initial system implementation (capital investment) (-) 12 % PA-technology capability margin (performance as fraction of ultimate) (+) 11 % PA-# of other options available (+) 10 % PA-# items requiring major ground test articles & demonstration (ex: new engines) (-) 7 % Technology R & D Phase TRD-# technology breakthroughs required to develop and demonstrate (-) 14 % TRD-estimated time to reach TRL 6 from start of R&D (-) 13 % TRD-# operational effectiveness attributes addressed for improvement (+) 13 % TRD-Current TRL (+) 11 % TRD-# full scale ground or flight demonstrations required (-) 11 % TRD-cost to reach TRL –6 (-) 10 % TRD-# operational effectiveness attributes previously demonstrated (+) 9 % TRD-# related technology databases available (+) 7 % TRD-# of new facilities required costing over $2M (-) 7 % TRD-total annual funding by item at peak dollar requirements (-) 4 % TRD-# multi-use applications including space transportation (+) 3 %