WG 1 Product S&T

1. Bill Mark Gregor Kiczales Mike Evangelist Helen Gigley Linda Northrop Kevin Sullivan LiGuo Huang Dan Port Don Reifer Bill Scherlis WG 1 Product S&T DoD Software Summit9 Aug 01

2. Report outline • Recommendations • Defense software • Role • Need • Case study: FCS • Product attributes: challenges and potential • Interoperability • Dependability • Evolvability • Contributing technologies • Selected examples • Technology evaluation issues • Phased strategy (tbd)

3. Recommendations • Initiate a multi-year S&T program to advance the software science and technology base, particularly for system of systems • Address DoD-critical needs, and exploit dual-use opportunities • Focus on tech base, and involve user/integrator base • Problem-driven and strategic. Mid- to long-horizon. • Structure as a program suite at aggregate funding $50M/yr. • Create virtual laboratories for research and experimentation at scale, to accelerate transition by providing early validation • Multi-stakeholder participation • Enable work across lifecycle • Realism and reality transfer: scale, complexity, (sanitized) artifacts • Evaluate technologies, artifacts, designs, operational concepts, methods, theories, tools • Document value and impact. • Employ more aggressive models for co-evolving operational concept and engineering design in order to develop the most aggressive possible solutions • Conceptualization/demonstration/evaluation • Rapid coupled iteration. Cf. JWID, CPOF double-helix.

4. Software wins wars • The 21st Century Defense environment • Asymmetry, coalition, rapid response • Organizational structure more complex, changing more rapidly • Need for rapid robust adaptation: short time scale, high consequences • E.g., Patriot, EW, ECM • Need to validate quickly: “Sorry, Dave, I can’t do this upgrade” • Increasing requirements uncertainty, evolution, and volatility • Defense computing is at the core • Huge diversity of system elements • Operate in adversarial environment • Software is the key engineering medium • Embodies: • Decision capability, algorithmic content, flexibility/malleability, … • Focus of capability: A principal competitive differentiator • For the soldier: Software as force multiplier • Not realizing full potential

5. Defense software challenges • Will commercialindustry do this for DoD? • Technology providers are not fully addressing DoD critical needs • DoD does have critical needs • Will defense integrators solve these problems for DoD? • Software interventions occur at all stages of the acquisition process • Integrators depend on vendors and other technology providers • Will the software S&T community deliver the technology? • The DoD software S&T community is dispersing. • Military challenges must once again become a dominant driver • Other programs not effectively addressing these challenges • DoD is too small a player to drive key elements of the IT food chain purely through demand • Bottom line: • DoD needs to build on the rapidly evolving commercial base • DoD needs to stimulate technology providers to innovate in its mission-critical areas, leveraging the market where possible • DoD needs to reestablish its software S&T community

6. FCS Concept Is this right? • Operations • Accelerate SA/Plan/Execute/Replan cycle • Accelerate power projection • Empower the commander • Performance • Reliable, robust, lightweight • Enter-once principle • Acquisition • Rapidly and safely incorporate new capability • Improve affordability and flexibility • Provide common basis for interoperability

7. Key technology drivers Interoperability Dependability Evolvability Attributes Definition Inhibitors Strategy Impact System of systems

8. Interoperability – definition • Elements of Interoperation • Composability • Safe and predictable interconnection • Frameworks • Safe, predictable, and scalable interconnection • COM, CORBA, Beans, EJB, HLA, etc. • Content sharing • Information linking • Common UI elements • Cut&Paste

9. Interoperability – inhibitors • Increasing: • Scale, distribution, etc. • Versions, configurations • Framework constituents • Shared domain semantics • End-to-end requirements • Real-time • Use of OTS elements • Rate of change • Assurance, framework compliance

10. Interoperability – strategy • Develop improved taxonomy of interoperability criteria • Select areas for developmental investigation • E.g., field maneuvers across services • Conduct aggressive experiments to drive problem identification • Develop basic metrics to understand effect on operations. • Adopt subjective assessment where measures lack • Apply current knowledge to achieve several levels of interoperability. • Analyze what is required to go from current state to required • Develop theory and technologies to address the criteria

11. Interoperation – impact • Value to the mission • Common information base • Enter once • Diff, reconciliation, fusion • Increased situation awareness • Right decision faster • Remove process costs • Assurance about the aggregate • Value to the PM • Cross-cutting issues addressed • Interdependence, jointness • Limit PM risk exposure

12. External dependability Availability Five 9s and beyond Security Prevention Detection Tolerance/Healing Safety Capability Human usage Internal dependability Framework robustness Component compliance for component aggregation Robust service interfaces Pre/post conditions Security, etc. Code safety Performance Deadlines Model compliance AOSD, UML Code safety and concurrency, … Domain Dependability – definition

13. Scale Environment Distribution Number of components Subsystem complexity Heterogeneity Integration into SWE process/tools E.g., ongoing measures, robustness testing… Assurance integration Aspect consistency mgmt Technology Appropriate analysis and formal methods Models, aspects, and support tools Design principles Measures and ROI Empirical validation: measures at scale Analytical validation at scale Researcher access At-scale artifacts Dependability failure data Dependability – inhibitors

14. 1. Enablers/levers Measures Dependability-assurance Co-evolution of code, models, argumentation Assurance technology Analysis / Assertion /Typing / Certification Component certification Robustness testing Human error models Component SWE technologies Composability Compositionality Framework technology 2. Principles Incrementality Adoptability Integration into lifecycle mgmt Development process, practices, tools, evaluation, etc. 3. Actions Technology Self-stability Robustness test Lightweight formal methods Etc. Testbed experimentation Dependability – strategy

15. Dependability – impact • Information Armor • Reliable, predictable warfighting capability • Confident adaptation and change • Improved reliability, security • Less “ility-constraint” on capability • More predictable systems costs

16. Includes Enhancement Functional Performance Structural Integration, interconnection Repair Dependability improvement Tolerance Robustness Etc. Encompassing Design for evolution Ride commercial growth curves Rapid adaptation Rapid exploration Creating tomorrow’s legacy Avoid software rust belt Legacy management Program understanding Information capture Evolvability – definition

17. Evolvability – inhibitors • Inadequate knowledge early in life-cycle of innovative systems to make appropriate architectural choices • And inadequate architectures, in any case • Imperfect understanding of the current/future environment • Lack of centralized control over system design • Intractability of legacy software systems • Lack of complete knowledge of design, dependencies & consequent difficulty of assessing impacts of change • Lack of present rewards for investments in future benefits • Failure to consider importance of corresponding “wetware” • Affordability • Costliness of revalidation

18. Technical approaches Theories, tools, practices of design & evolution without total central control (including COTS) Models and tools for discovering and managing dependencies across “administrative domains” Generative methods: raising abstraction level Self-protecting systems: resist bad upgrades Legacy analysis and evolution support systems Qualitative and quantitative evolution models Principles of operation Aim to make fundamental advances, but do it in a way that can be evaluated on a year-to-year basis through the production of advances in short, mid- and long-term Enable “valuation” and “monetization” of evolvability Evolvability – strategy

19. Evolvability – impact • Improved ability to meet evolving threats • Substantially reduced lifecycle costs • Enable emergence of systems of systems (including legacy/evolving components) • Improved ability to meet user needs

20. A sample of technical ideas that contribute to these goals Aspect oriented software design Value-based design approaches Architecture-based performance prediction Chains of evidence for incremental assurance Self-stabilizing systems Description attributes Definition Approaches to demonstration and evaluation Maturity of ideas and community supporting them Impact and value to the mission Points of intervention in the software lifecycle Technical ideas

21. Technical ideas • Idea: • Aspect-oriented software development (AOSD) • Modularize crosscutting concerns (next modularity after OO) • Manage complexity through localization of aspects • Evaluation • Can large systems be developed/maintained more effectively, with higher-quality, etc. • Drive thru: design, coding, validation, other practices • Maturity • Xerox, UBC, IBM, NEU, MIT, CMU, VUB (Brussels)… • 10 workshops in last 4 years, conference in 2002 • Impact/outcomes • Composability at next levels of scale/dynamicity… • Intervention • Arch, design/implementation, evolution, assurance

22. Technical ideas • Idea: • Value-based design approaches for software • Design for value added: products, processes, portfolios • Evaluation: • Sensible theory, tools and practices, empirical evaluation • Maturity • U Va, USC, CMU, U Wash, SEI, NRC Canada, USCD • Workshops (EDSER) • Impact • Significantly increase economic efficiency • Value-added construction of much more complex systems • Intervention • Requirements, architecture, evolution, lifecycle

23. Technical ideas • Idea: • Architecture-based analytical performance modeling • Build on queuing theory, etc. • Evaluation • Compare with predicted performance • Maturity • SEI, U Va, UIUC, MIT, Ohio St, U Texas • Impact • No performance surprises on system integration • No performance surprises when system overloaded • Intervention • Design, test, upgrade (prediction)

24. Technical ideas • Idea: • Chains of evidence for incremental assurance • Code safety: exceptions, arrays, encapsulation, concurrency, etc. • Maintain assurance argument and code simultaneously • Evaluation • Case studies at scale • Maturity • Compaq (ESC/Java), MIT/UVa (LCLint), CMU, U Brussels • Impact • Increments of effort yield increments of value in constructing assurance rationale • Intervention • Design/development, evolution

25. Technical ideas • Idea: • Self-stabilizing systems • Evaluation • Empirical/analytic comparison with standard ad hoc approaches. • Maturity • U Texas, Ohio St, IBM, Technion • Workshops • Impact • Improvement over standard fault tolerance approaches. • Coherent extension of SSS to load balancing, security, etc. • Intervention • Design

26. Additional technical ideas • Appropriate formal methods • Proof-carrying code • Frameworks and patterns • Theory and practice • Evaluation of components for compliance • Evaluation of frameworks • Product architecture, human organization, evolution and dynamics • Complex adaptive systems • Understanding/predicting emergent behaviors • Model-based systems architecting and software engineering • Avoiding model-clashes through explicit model integration • Engineering through model-views

27. The role of scientific approaches to technology evaluation • Benefits • Early validation • Do we understand the requirements? • Does the op concept match the technology? • Prototype early • “10k users before publishing” • IETF model • Rapid iteration in technology development • Caveats • Diversity of stakeholders and value metrics • Light under the lamppost • Not all needs have measures • Indeed, few needs have useful measures • Dependability? Evolvability? Interoperability? • Capability to measure/evaluate lags capability to produce • Internal measures often lacking • Incorporate strategic/subjective value in program mgmt

28. Technology insertion • Points of technology intervention occur throughout the lifecycle • Major product -ilities must be addressed through combinations of process, tools, information mgmt, models, etc. • Interoperability, evolvability, dependability • Technology intervention is being facilitated through incremental approaches to evaluation and adoption • These are now emerging for some SWE capabilities • Reduce costs/risks of adoption • Improving evaluation is important • Relatively few evaluation criteria have useful measures • But keep caveats in mind – they are significant • Major changes in the tech base and in the marketplace are enabling new approaches to long-standing challenges • Early prototyping and rapid iteration • Collaboration and information sharing