NATO IV&V: Work in Progress

1. NATO IV&V:Work in Progress Brad Bigelow, SHAPE Arend Smit, M&I/Partners NASA 2013 IV&V Annual Workshop, 11 September 2013

2. What is NATO? • NATO is a political and military alliance whose primary goals are: • The collective defence of its members • The maintenance of a democratic peace in the North Atlantic area • Established in 1949 • 28 Member Nations • Albania, Belgium, Bulgaria, Canada, Croatia, the Czech Republic, Denmark, Estonia, France, Germany, Greece, Hungary, Iceland, Italy, Latvia, Lithuania, Luxembourg, the Netherlands, Norway, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Turkey, the United Kingdom, and the United States. • Headquarters in Brussels, Belgium NASA 2013 IV&V Annual Workshop, 11 September 2013

3. NATO: A Pioneer in Software Engineering Software Engineering Conference sponsored by the NATO Science Committee, Garmisch, Germany, 7-11 October 1968 NASA 2013 IV&V Annual Workshop, 11 September 2013

4. NATO Communications & Information Agency Established 1 July 2012 Primary agency for developing and operating NATO’s civil and military information services and networks Over 3,500 military and civilian staff Over 30 locations in Europe, USA and Afghanistan NASA 2013 IV&V Annual Workshop, 11 September 2013

5. NATO C&I Agency Customers NATO Military Commands NATO Military Operations NATO Nations NASA 2013 IV&V Annual Workshop, 11 September 2013

6. Initial Use of IV&V • Began in 2001 • Driven by concern of some NATO member nations over the capacity and independence of in-house QA functions • Initially focused on QA during prime contractor execution • Review of key deliverables • Oversight of prime contractor testing & independent testing • Support to project team during milestone reviews

7. Initial Results • Use of contracted IV&V delivered real benefits: • Faster turn-around on draft deliverables • More consistent/higher quality technical support • Greater attention to test-driven design and development

8. Initial Lessons Learned • Two opportunities for improvement: • Broadening scope from QA of prime contractor activities to QA of overall project scope—including NATO internal factors • Bringing IV&V support on board during preparation of the prime contractor solicitation • Better quality control over complex solicitation materials • Statements of Work, System Requirements Specifications • Opportunity to review winning bidder’s material prior to contractor award • Enabled mitigation of some risks during pre-award discussions

9. Programmatic Approach to IV&V • In 2007, NATO launched an initiative to establish a program management and integration capability for the Bi-SC AIS • Bi-SC AIS • Bi-Strategic Command Automated Information Systems • One of NATO’s largest investment programs—over 500M Euro • Covers all Information Services and Systems used in NATO’s Military Commands • Supporting 15,000 users in 28 nations • 10+ year rolling program • Over 100 active projects

10. 3 Key Aspects • IV&V support staff reported to program manager, not project managers • Issues elevated, resources made more quickly • IV&V resources managed as a pool at the program level • Enabled program to prioritize and shift IV&V support to different projects • Same contractor for both program and IV&V support • Ensured higher degree of integration among project

11. Initial Results of Programmatic Approach • Pooled approach to IV&V resources has several advantages: • Better workload balance—fewer peaks and valleys • Compensates for differences in project timelines • Ability to concentrate less common/higher cost experts • Management at the program level enables better adaptation to changing project needs • Some projects have continuing QA challenges, but others experience “QA crises” that can be resolved through prompt intervention

12. New NATO Headquarters 120,000 M2 complex Construction started in October 2010 Planned to be completed early 2016 A secure, collaborative , energy-efficient building NATO C&I Agency is implementing the network/IT infrastructure

13. ANWI: Active Network Infrastructure • For four security domains, ANWI provides • Processing/storage • Wired and wireless LAN switching • Communications (telephone, teleconferencing, site security communications, IPTV, GSM) • End-user equipment • Core information services • Cross domain support for external networks interconnection and information services interoperability

14. ANWI (continued) • Integration with systems/services external ANWI scope • Migration of services to new building • Support for migration of business applications • Integration Test Services (ITS) • Baselinedreference suite of functions • Host Nations test their capabilities prior to deployment in new building

15. ANWI IV&V • Support in pre-contract award phase • Focus on major risk areas during execution • Security accreditation • Migration • Service based specification, implementation, and acceptance • Introduction of new capability • Unified Communications and Collaboration • Service Management • Cross Domain services • Monitor prime contractor’s Integration Test Services

16. Questions? Brad Bigelow Principal Technical Advisor, J6 Supreme Headquarters Allied Powers Europe (SHAPE) (former Bi-SC AIS Program Manager, NATO C&I Agency) brad.bigelow@shape.nato.int ArendSmit, PhD IT and QA consultant M&I/Partners arend.smit@mxi.nl NASA 2013 IV&V Annual Workshop, 11 September 2013