Briefing for INCOSE

1. Briefing for INCOSE • Learn. • Perform. • Succeed. Rev. Oct 07

2. DAU Mission We train the AT&L Workforce through certification and assignment-specific courses We promote career-long learning through our Continuous Learning Center We offer performance support to the AT&L Workforce through consulting, Rapid Deployment Training, and targeted training We facilitate knowledge sharing through online resources and communities of practice Provide Human Capital Strategic Planningfor the AT&L Workforce Provide practitioner training, career management, and services to enable the AT&L community to make smart business decisions and deliver timely and affordable capabilities to the warfighter.

3. DAU Regional Orientation We are part of the community… not just a place to go to take classes. Mid-Atlantic (Pax River) AT&L WF ~22,476 Capital & Northeast and DSMC (Fort Belvoir) AT&L WF ~33,357 West (San Diego) AT&L WF ~25,408 Midwest (WP AFB) AT&L WF ~18,945 South (Huntsville) AT&L WF ~28,056 Located with our Customers

4. AT&L Performance Learning Model • CL Modules - Online modules to help • you earn continuous learning points • Conferences • PEO / SYSCOM • Business Manager • DAU Acquisition Community Conference • Consulting - We come to your workplace to assist you • Targeted Training - Tailored learning for your organization • RDT - On-site and online training on the latest AT&L policies 24/7 Learning Assets for the Classroom and the Workplace Training Courses - DAWIA Certification, assignment-specific, and executive & leadership courses – in the classroom and online • AKSS - Online gateway to AT&L information & tools • ACC - Online collaboration communities tailored to your needs • Virtual Library - Keeping you connected to research tools when you are not on campus

5. Core Plus is Here! “Plus” or job competency point-of-need training (frequently CLMs) (tailored to your specific job or assignment) Core Plus is designed to guide workforce members to additionaltraining beyondwhat’s required for certification. Career Field foundation knowledge and skills (all career field members) Common acquisition foundation knowledge & skills (all DoD AT&L workforce members) Core Plus is an enhancement to the existing AT&L certification framework. It is not a wholesale replacement!

6. FY08 T&E Level II Core Plus Framework Three Level Certification Framework Acquisition & Functional Core DAWIA Certification Standards – applies to all career field members Tailored Cross-Functional “Plus” Training, Education & Experience Opportunities – based on the job or assignment EXAMPLE

7. P P FY08 Test & Evaluation Field Training P = Prerequisite Level III Certification Level I Certification Level II Certification TST 203 Intermediate Test & Evaluation P P TST 302 Advanced Test and Evaluation ACQ 101 Fundamentals of Systems Acquisition Management P TST 102 Fundamentals of Test and Evaluation ACQ 201 Intermediate Systems Acquisition 5 days classroom 5 days classroom P 35 hrs, online 25 hrs, online SYS 202 Intermediate Systems Planning, Research, Development and Engineering, Part I 35 hours, online 1 week classroom SYS 101 Fundamentals of Systems Planning, Research, Development and Engineering CLM 029 Net-Ready Key Performance Parameter CLE 023 Modeling and Simulation for Test and Evaluation 30 hrs, online • Knowledge based • GS 5-9 & E7-O3 • Case/scenario based • GS 13-14 & E7-O5 35 hrs, online Level I “Core Plus” Courses & CL Modules (See DAU catalog for details) Level III “Core Plus” Courses & CL Modules (See DAU catalog for details) Level II “Core Plus” Courses & CL Modules (See DAU catalog for details) Vers 8.1

8. FY08 SPRDE: Science & Technology Manager P = Prerequisite Level III Certification Level I Certification Level II Certification ACQ 101 Fundamentals of Systems Acquisition Management P STM 201 Intermediate Science and Technology Management P STM 302 Advanced Science and Technology Management NONE 3 days classroom 25 hrs, online P CLE 021 Technology Readiness Assessment 5 days classroom • Knowledge and Application/case based • GS 9-12 & E7-O4 • Case/scenario based • GS 13-14 & E7-O5 Level III “Core Plus” Courses & CL Modules (See DAU catalog for details) Level II “Core Plus” Courses & CL Modules (See DAU catalog for details) Vers 8.1

9. P FY08 SPRDE: Systems Engineering Training P = Prerequisite Level III Certification Level I Certification Level II Certification SYS 202 Intermediate Systems Planning, Research, Development and Engineering, Part I CLL 008 Designing for Supportability in DoD Systems P P SYS 101 Fundamentals of Systems Planning, Research, Development and Engineering ACQ 201 Intermediate Systems Acquisition ACQ 101 Fundamentals of Systems Acquisition Management 30 hrs, online P 35 hours, online 1 week classroom SYS 203 Intermediate Systems Planning, Research, Development and Engineering, Part II 25 hrs, online SYS 302 Technical Leadership in Systems Engineering P P 35 hrs, online CLE 003 Technical Reviews 10 days classroom • Knowledge based • GS 5-9 & E7-O3 • Application/case based • GS 9-12 & E7-O4 5 days classroom • Case/scenario based • GS 13-14 & E7-O5 Level I “Core Plus” Courses & CL Modules (See DAU catalog for details) Level III “Core Plus” Courses & CL Modules (See DAU catalog for details) Level II “Core Plus” Courses & CL Modules (See DAU catalog for details) Vers 8.1

10. P FY08 SPRDE: Program Systems Engineer Training (new career field) P = Prerequisite Level I Certification Level II Certification Level III Certification SYS 202 Intermediate Systems Planning, Research, Development and Engineering, Part I P CLL 008 Designing for Supportability in DoD Systems P SYS 101 Fundamentals of Systems Planning, Research, Development and Engineering ACQ 201 Intermediate Systems Acquisition ACQ 101 Fundamentals of Systems Acquisition Management 30 hrs, online P SYS 302 Technical Leadership in Systems Engineering 35 hours, online 1 week classroom SYS 203 Intermediate Systems Planning, Research, Development and Engineering, Part II P 25 hrs, online P 35 hrs, online CLE 003 Technical Reviews Two 100 level DAU courses from among PM, T&E, PQM, LCL, BCFM, IT or CON career fields 10 days classroom Two 100 level DAU courses from among PM, T&E, PQM, LCL, BCFM, IT or CON career fields 5 days classroom One 100 or 200 level DAU course from among PM, T&E, PQM, LCL, BCFM, IT or CON career fields Two 200 or 300 level DAU courses from among PM, T&E, PQM, LCL, BCFM, IT or CON career fields LOG 204 Configuration Management 15 hrs, online • Knowledge based • GS 5-9 & E7-O3 • Application/case based • GS 9-12 & E7-O4 • Case/scenario based • GS 13-14 & E7-O5 Level I “Core Plus” Courses & CL Modules (See DAU catalog for details) Level II “Core Plus” Courses & CL Modules (See DAU catalog for details) Level III “Core Plus” Courses & CL Modules (See DAU catalog for details) Vers 8.1

11. DAU Strategic Partnerships Credit for DAU courses toward degrees & certificates at colleges and universities Shorter time to get your degree Saves tuition assistance and out-of-pocket expenses “Excelerate” Your Master’s Degree… Excelerate is an exciting new program where you can earn your master’s degree in less time. Through this program, several universities are offering the AT&L workforce credit toward masters degrees for DAWIA Level II and III certification.* *DAWIA course and masters programs vary More information on DAU’s partnership program can be found at: http://www.dau.mil/about-dau/partnerships.aspx

12. DAU Learning Assets at: www.dau.mil DAU Training Courses Performance Support Continuous Learning Center http://www.dau.mil/registrar/enroll.asp http://www.dau.mil/performance_support/ https://learn.dau.mil/html/clc/Clc.jsp AT&L Knowledge Sharing System (AKSS) Webcast/Podcast Center DAU Acker Library http://www.dau.mil/library/ http://akss.dau.mil http://www.dau.mil/webcast_podcast/webcast_podcast.asp

13. DAU’s Commitment to the DoD AT&L Workforce • Training- 24 / 7 learning assets: in the classroom and in the workplace • Continuous Learning - Helping you learn in the workplace: what you need to know , when you need to know it • Performance Support - Supporting your organization • Knowledge Sharing - Connecting you – the Engaged Learner - with the experts, resources and materials you need to do your job Shaping a culture of engagement and career-long learning

14. Current Events ELO # 3.1 and 3.2

15. Honorable John J. Young, Jr. Dr John Foulkes LtGen Henry Obering III Vacant

16. Systems & Software EngineeringOrganizational Core Competencies Deputy Director Enterprise Development Deputy Director Developmental Test & Evaluation Deputy Director Software Engineering & System Assurance Deputy Director Assessments & Support Nic Torelli SES Chris DiPetto SES Bruce Amato (Acting) SES Glyn James (Acting) SES CORE COMPETENCIES CORE COMPETENCIES CORE COMPETENCIES CORE COMPETENCIES • SE Policy • SE Guidance • SE in Defense Acquisition Guidebook • Technical Planning • Risk Management • Reliability & Maintainability • Contracting for SE • SoS SE Guide • SE Education and Training • DAU SE Curriculum • SPRDE Certification Reqt • Special Initiatives • Corrosion • RTOC • VE • DT&E Policy • DT&E Guidance • T&E in Defense Acquisition Guidebook • TEMP Development Process • DT&E Education and Training • DAU DT&E Curriculum • DT&E Certification Reqt • Joint Testing, Capabilities & Infrastructure • Targets Oversight • Modeling & Simulation • Acquisition System Safety • SWE and SA Policy • Guidance • SoS, SA Guides • Education and Training • DAU SW Acq Curriculum • Continuous Learning Modules for SW, SoS, SA • Software Engineering • Acquisition Support • Software Engineering Institute (SEI) • Process Improvement • CMMI • DoD/National Software Investment Strategy • Industrial Base capability • Support of ACAT I and other special interest programs (MDAP, MAIS) • Assessment Methodology (Defense Acquisition Program Support – DAPS) • T&E Oversight and Assessment of Operational Test Readiness (AOTR) • SE/T&E Review of Defense Acquisition Executive Summary Assessments (DAES) • Lean/6-Sigma Training/Cert Acquisition program excellence through sound systems and software engineering Director, Systems & Software Engineering Kristen Baldwin (Acting) SES

17. DoD AT&L Workforce

18. Government Oversight Pendulum 2008 Auth. Bill SE Emphasis • Less Guidance and Oversight: • Performance Specs • SOO • Prime Contractor is the Integrator (LSI) • More Guidance and Oversight: • MILSPECS • SOW • Government is the Integrator

19. Bill Would Bar Contractors from Running Defense Programs • Main Provisions from the 2008 Defense Authorization Bill Passed by the HASC • No more Lead System Integrators (LSIs) after 2011 • DoD directed to study its acquisition workforce and identify and fill gaps in skills needed to effectively manage programs • Current LSI managed programs not effected unless opened up to competitive bid

20. Current Programs Managed byLead System Integrators • Ground Based Midcourse Defense • Boeing • Future Combat System • Boeing/SAIC • Littoral Combat Ship • Lockheed Martin • Coast Guard Deepwater Modernization • Lockheed Martin/Northrop Grumman

21. JCIDS PublicationsMay 1, 2007 • CJCSI 3170.01F (JCIDS) • CJCSM 3170.01C (Documents) • Key Performance Parameters • Force Protection & Survivability (Req) • Sustainment (Materiel Availability)(Req) • NR-KPP (Req) • System Training (Selectively Applied) • Energy Efficiency (Selectively Applied)

22. Key System Attribute(KSA) • Those system attributes considered most critical or essential for an effective military capability but not selected as a KPP • KSAs provide decision makers with an additional level of capability prioritization below the KPP • But with senior sponsor leadership control • Generally 4-star, Defense Agency Commander or Principal Staff Assistant

23. A View From the Top • We tend to chase technology • 90% of the work is in the first 10% of System Development • The Dir, DOT&E wants DOT&E more involved with DT where he feels we have a lack of government oversight.

24. Defense Science Board Task Force on DT&E • AT&L memo, April 30, 2007, request • Examine… • OSD organization, roles & repsonsibilities for T&E oversight (compare DT&E to OT&E; changes?) • Changes required to establish statutory authority for OSD DT&E oversight • Recommend improvements in DT&E process to discover suitability problems early • Mr Charles Adolph – Chair the Task Force • Mr Chris DiPetto (ODUSD (A&T) and Dr Ernest Seglie (ODOT&E), Executive Secretaries

25. Reliability Availability and Maintainability (RAM) Findings • Acquisition Reform implementation had a detrimental impact on RAM practices • With some exceptions, the practice of reliability growth methodologies was discontinued during System Design and Development (SDD) and deferred until production • Relevant military specifications and standards were cancelled and not, in all cases, replaced with industry standards • The technical/managerial workforce was reduced in most government program offices and test organizations • RAM shortfalls are frequently identified during DT, but program constraints (schedule and funding) often preclude incorporating fixes and delaying IOT&E • In some instances, programs had such serious RAM concerns that they were precluded from proceeding to production until the problems could be corrected. Improved RAM decreases life cycle costs and reduces demand on the logistics system

26. Impact of Low Reliability on Life Cycle Cost (LCC) • Recent Logistics Management Institute (LMI) and Army studies investigated relationship between reliability and Life Cycle Cost (LCC). Both studies concluded that reliability investment during design and development reduces support cost • LMI study concludes that: • 30-60% reduction in support cost is achievable • Reacting to lower reliability after IOC (vice during development) results in significant support cost increases • OSD CAIG study shows that support costs are two-thirds or more of total system cost • DoD should address acquisition and LCC strategies concurrently • Support cost bill payers should participate in requirements definition and trade studies during development Upfront Investment in Robust RAM Effort Emphasizing Reliability Growth Reduces Life Cycle Costs

27. RAM Recommendations The single most important step necessary to correct high suitability failure rates is to ensure programs are formulated to execute a viable systems engineering strategy from the beginning, including a robust RAM program, as an integral part of design and development. No amount of testing will compensate for deficiencies in RAM program formulation. To this end, the following RAM-related actions are required as a minimum: • Identify and define RAM requirements during the Joint Capabilities Integration Development System (JCIDS) process and incorporate them into the Request for Proposal (RFP) and as a mandatory contractual requirement • During source selection, evaluate the bidders’ approaches to satisfying RAM requirements • Ensure flow-down of RAM requirements to subcontractors • Make RAM, to include a robust reliability growth program, a mandatory contractual requirement and document progress as a part of every major program review • Develop a military standard for consistent RAM development and testing that can be readily referenced in future DoD contracts • Ensure an adequate cadre of experienced RAM personnel are part of the Service acquisition and engineering office staffs

28. Roles and Responsibilities of Government T&E OrganizationsFindings The aggregate lack of process guidance due to the elimination of specifications and standards, massive workforce reductions in acquisition and test personnel, and the high retirement rate of the most experienced technical and managerial personnel in government and industry has a major negative impact on DoD's ability to successfully execute increasingly complex acquisition programs. • Major personnel reductions have strained the pool of experienced government test personnel • A significant amount of developmental testing is currently performed without the needed degree of government involvement or oversight and in some cases, with limited government access to contractor data • The current trend away from governmental involvement in DT makes operationally oriented testing in some areas during development even more difficult to accomplish

29. Roles and Responsibilities of Government T&E Organizations Recommendations As a minimum, government test organizations should retain a cadre of T&E personnel to perform the following functions: • Participate in the translation of operational requirements into contract specifications, and in the source selection process, including RFP preparation • Participate in developmental test and evaluation planning including TEMP preparation and approval • Participate in the technical review process including SRR, PDR,CDR, TRR and FRR • Participate in test conduct, data analysis, and evaluation and reporting, with emphasis on analysis and reporting • Capture and re-apply best practices and lessons learned

30. Integrated Test and EvaluationFindings • Service acquisition programs are incorporating integrated testing to a limited degree through varying approaches  • Army has integrated DT and OT organizations into one command • Navy utilizes a full-spectrum RDT&E approach to conducting Test & Evaluation • Air Force employs Combined Test Force concept which consolidates test execution • Additional emphasis on integrated testing can result in greater T&E process efficiency and program cost reductions

31. Integrated Test and EvaluationRecommendations • Implement OSD policy change mandating integrated DT&E and OT&E planning and execution throughout the program • Require sharing and access to all appropriate system-level and selected component-level test and model data by government DT and OT organizations as well as the prime contractor, where appropriate • Incorporate data access requirements in contract • Integrate test events, where practical, to satisfy OT and DT requirements • Define which testing will be accomplished by the prime contractor, government DT lead, and OT as the lead agency prior to award of contract • Require an operational evaluation framework as a part of the Milestone B TEMP • Make available a cadre of operational personnel to support DT for Acquisition Category (ACAT I) and special interest programs, as a minimum • Better integrate Operational Test Agencies (OTAs) into the DR process to include participation on Joint Reliability Maintainability Evaluation Team (JRMET) or Corrective Action Review Board throughout DT

32. Operational Test Readiness Review (OTRR)Findings • DoDI 5000.2 requires that “the Service Acquisition Executive (SAE) shall evaluate and determine materiel system readiness for IOT&E" • For all Services, the OTRR process is an in-depth review of readiness • Decision authority is frequently delegated to the Program Executive Officer (PEO) • Shortcomings in system performance, suitability, and RAM are usually identified during the OTRR • In most cases, the operational test readiness certifying authority is aware of the risk of not meeting OT criteria when major shortcomings exist • Because of the funding constraints, the low priority given to sustainment, as well as the urgency in recent years to get new capabilities to the Warfighter, major suitability shortcomings have rarely delayed the commencement of dedicated IOT&E

33. OTRR Recommendations • Conduct periodic operational assessments to evaluate progress potential for achieving pre-determined entrance criteria for operational test events • Conduct an independent AOTR prior to the OTRR (included in latest draft DODI 5000.2) for oversight programs • Results of the AOTR should be briefed at the OTRR • Include a detailed RAM template in preparation for the OTRR • CAE submit a report to OSD that provides the rationale for the readiness decision • Include an evaluation of weapon system’s capabilities against Critical Operational Issues (COIs) • Certify that the DT evaluation is complete and indicates acceptable risk of passing OT • Include an explanation for recommending go-ahead when there is a reasonable expectation that the system is not effective and/or suitable DT&E Task Force Strongly Endorses Recent AT&L/DOT&E Initiative to Strengthen OSD OTRR Process

34. OSD DT&E OrganizationRecommendations • The most important change required for improved OSD DT oversight is to is to reestablish a recognizable, focused, integrated, and robust DT organization within AT&L • Consolidate DT-related functions in AT&L * • Program oversight and policy, and Foreign Comparative Test • Have Director, DT&E directly report to DUSD(A&T) • Restore TEMP approval authority to Director, DT&E  • Integrate Test Resource Management Center (TRMC) activities early into DT program planning *Three Task Force members out of fourteen voted against consolidation

35. Reliability Improvement Working Group • DUSD(A&T) and DOT&E memo signed February 15, 2008 requests CAE support to establish a working group to implement recommendations to improve RAM on DoD weapon systems during system development • Specific Tasks: • Ensure execution of a viable SE strategy , including RAM growth program, as an integral part of design and development • Ensure government orgs reconstitute cadre of experienced T&E and RAM personnel • Implement mandated Integrated DT and OT, ensure access to all appropriate data (contractor and government) and conduct T&E in an operationally representative environment as early as possible

36. The entire DSB report is available at: http://www.acq.osd.mil/ dsb/reports/2008-05-DTE.pdf

37. Continuous Process Improvement (CPI) • Establishment of AT&L Continuous Process Improvement (CPI) Program • To help optimize our organizational performance • To establish a basic understanding of various CPI principles and roles, all AT&L personnel had to complete CLE 015 (CPI Continuous Learning Module) before March 30, 2007.

38. Prototyping & Competition • Many programs initiated with inadequate technology maturity • Program decisions based on paper proposals • There is a need for & benefits of quality prototyping • Issues need to be discovered before SD&D phase • Acquisition strategies requiring AT&L approval must be formulated to include competitive, technically mature prototyping through Milestone B

39. Resources • OSD SSE: http://www.acq.osd.mil/sse/ • OSD SE Pubs: http://www.acq.osd.mil/sse/guidance.html • ASSIST: http://assist.daps.dla.mil/online/start/ • NAVAIR Tech Reviews: http://www.navair.navy.mil/kms/41g/ • Army LSS: http://www.amc.army.mil/pa/LeanSixSigma.asp • USAF SE: http://www.afit.edu/cse/

40. Resources (cont) • DAU: www.dau.mil • DAU SE: https://acc.dau.mil/se • DAG Chapter 4: https://akss.dau.mil/dag/Guidebook/IG_c4.0.asp • DAU Cont Learning: http://clc.dau.mil/ • DAU Training: http://training.dau.mil/ • AMRDEC: http://www.redstone.army.mil/amrdec/

41. Questions? Wallace J. “Wally” Tubell Professor of Engineering DAU South 256-722-1081 Wallace.tubell@dau.mil