Streamlining and Assuring the IT Acquisition Process Leveraging the Architecture Assurance Method

1. Interoperability Clearing House IT Acquisition Advisory Council Streamlining and Assuring the IT Acquisition Process Leveraging the Architecture Assurance Method Kevin Carroll, Director, ICH Fellow Interoperability Clearinghouse Vice Chair, IT-AAC A non-profit research institute Kevin.Carroll@ICHnet.org 703 768 0400 www.ICHnet.org

2. ICH Charter to advance standardized assessment methods that assure successful acquisition of information technology solutionsaligned with business drivers in a timely, inclusive, and conflict free environment “It ought to be remembered that there is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things. Because the innovator has for enemies all those who have done well under the old conditions, and lukewarm defenders in those who may do well under the new” Nicolai Machiavelli

3. ICHnet.org Architecture Resource CenterProviding standards of practice for effective IT Program Mgt • Architecture Assessment Methodology assessing goodness of requirements, capabilities, services and technologies • proven to mitigate risk, OCI, and complexities of fast paced IT Market. • Comply with Clinger Cohen Act directive to avoid duplication of effort • Aligns and integrates disparate architecture and acquisition decision points • Reusable Solution Architecture Frameworks that maximize COTS/Open Source solutions • Standardized solution architectures for E-Gov, IT Infrastructure, Cyber-Security, E-Health • Requirements & Capability Assessment that prevent over specification • Acquisition performance metrics and service levels agreement (SLAs) • Technology Feasibility/Risk Assessment. Verify technology readiness levels and maturity. • IT Acquisition Advisory Council that taps the combined expertise and experiences • Organic structure for tapping into standards of practice & standards bodies • Leverages expertise from academia, standards bodies, innovators and COIs. • “Grey Beard Peer Review” that guides agencies around mine fields and pot holes. AF Scientific Advisory Board Report; “Most (PMs) are struggling with the complexity and a few have failed miserably. The complexities are numerous and less than obvious. …. Requirements must flow into an architecture that can truly exploit the advantages of COTS. Contractors must shift from a “design and build” unique product to “buy and integrate” standards products.”…”(Everyone) freely admits that have made every mistake imaginable along the way. Unfortunately, others can’t image the mistake that are about to make”.

4. Fast paced IT market undermines traditional government research and assessment efforts (AF SAB 00) • DoDAF lacks theoretical foundation for C4ISR Systems (ICCRTS 07) • No formal mechanism of deriving COTS selection criteria from Business Reference Model (IAC EA SIG 03) • No body of knowledge from which PM can judge competing IT offerings (ECCWG Report 99) • No mechanisms for assessing risks, composability or interoperability of commercial solution (AF SAB Report 00) • Current acquisition mechanisms result in 80% failure rates of major IT programs (CMS IC CIO 03) The IT Acquisition Process ChallengeCurrent DoD processes are not optimized for buying IT What Past Studies Tell us; PMs Feel... Overwhelmed by offerings Cycle Times to Assess Tech Hype vs Reality Interoperability? Common Vocabulary ?

5. Federal Program Assessment ChallengesMilSpec assessment methods do not expose real risks Major IT programs in government fail between 72-80%, costing over $20B/year; Revamp Governance and Solution Assessment processes to accommodate new paradigms in both War Fighting and Business Operations. (DSB 09) Budget pressures drive Reusable & Standardized Solution Architectures, Performance Metrics and Assessment Results. Reuse can significantly reduce cost, risk and cycle times. (OMB report on Solution Architecture Integration Lab) DoD Leadership demanding Independent Architectures (DSB 08) & Technology Clearinghouse (NDAA 08) that transparently captures innovations and stake holder needs. Evidence from early adopters of the Acquisition Assurance Method show 70% reduction in cycle times, greater transparency and prevention of over specification. (AAM implementation results at SAF/XC, Navy PEO C4I, USMC CIO, PTO, GPO, GSA) ECCWG report to the DEPSECDEF “Current architectural initiatives (C4ISR, JTA) don’t quite fit the E-Business problem space”. “Rate of Technology Change Exacerbates the Problem” “the lack of an easily understood process modeling technique is a root cause for the lack of participation of the general user in the definition of business processes.” 4

6. Architecture Assurance Methodology (AAM)A repeatable, measurable, sustainable method for assessing IT Programs Developed as a consensus based process standard for architecture assessments (OMB A119 mandate) that incorporates by reference; ITIL, SOA, Value Stream Analysis, Evidenced Based Research, BPR Derived from Solution Architecture Commercial Best Practices and Lessons Learned (GM, GE, Citigroup, Financial Services Technology Consortia, Delta, OMG, OASIS, AF, FedEx, Discovery Channel) Tuned for the fast paced COTS/Open Source market (Clinger Cohen Act, NTTAA) Provides bottom up view of the COTS/Open Source “realm of the possible” view needed to prevent requirements over specification (IT-AAC, AF SAB, DSB, IAC) Proven to mitigate risk and assure implementation success in many large govt/industry programs (AF, Navy, PTO, GPO, USMC, Discovery, DHS, GSA, OSD HA) Operated much like SEI’s CMM; training, mentor, equip both Acquisition and CIO staff. BTA is primary sponsor. “We have put to practice the AF Solution Assessment Process (ASAP) at the Air Force Communications Agency (AFCA) with some well documented success. It was developed with Interoperability Clearinghouse (ICH) and provides a structured and measurable IT assessment process with the agility to provide decision-quality assessments ranging from quick-looks to more in-depth capability-focused technology assessments and lightweight business case analysis.” General Mike Peterson, AF CIO

7. AAM: Integrating and Assuring the Acquisition LifecycleRepeatable, Measurable, Sustainable • ICH/AAM Tools: • How To Manuals • Stake Holder Training • Solution Architecture Working Groups • Mentoring and Outreach • Domain Expertise Analytical Decision Support Consumer Report Like Results – CAPABILITIES, AOAs, TCO

8. Honorable Michael Wynn, former Secretary of USAF Honorable Dave Patterson, former OSD Comptroller, University Of Tennessee Honorable Dave Oliver, former OSD ATL, EADS Honorable Robert Cresanti, former Secretary of Technology General (ret) Charlie Croom, former DISA Director, LMCO Dr. Robert Childs, President, NDU General (ret) Paul Nielson, CEO, SEI CMU Lt General (ret) Jack Woodward USAF, former AF Deputy CIO General (ret) Charlie Croom, LMCO Gen (ret) George Campbell, former Army G6, BAE Systems Dr. Marv Langston, former OSD C3I DCIO, former Navy CIO Kevin Carroll, former Army PEO EIS, ICH Fellow Charles Tompkins Esq., Chair Systems Management Dept, NDU IRMC RADM (ret) Lenn Vincent, Industry Chair, DAU Frank Weber, former AF ESC 554 Wing Commander Robert Burton, former OMB Acquisition Director, Venerable Dr. Jerry Mechling, Director of E-Gov Education, Harvard KSG, Steve Cooper, Partner, First DHS CIO, Strativest Jim Young, former CIA, Google Larry Allen, Executive Director, Coalition for Government Procurement Ed Black, President, Computers & Communications Industry Consortia Barry Robella, Professor of Systems Engineering, Defense Acquisition University Stephen Buckley, Kerberos Consortia, MIT Sloan Al Mink, Vice President SRA Edward Hammersla, EVP Trusted Computing Solutions William Lucyshyn, Director of Research, School of Public Policy, U of MD Dan Johnson, Sr. Council, Computers & Communications Industry Association Lewis Shepherd, former DIA CTO, Microsoft Alan Balutis, CISCO Systems Kirk Phillips, the Kirk Group Dr. Kevin Kreitman, Principle Consultant, Aerospace Corp Frank Cooper, VP/CTO, Concurrent Technology Corp John Weiler, Executive Director, Interoperability Clearinghouse IT Acquisition Advisory Council (IT-AAC)Providing our National IT Leadership with an Actionable Roadmap

9. Critical Success Factors for Acquisition ExcellenceA public/private partnership compliments vs competes with supply chain partners

10. ICH’s Architecture Assurance Method;Streamlining and Assuring the IT Acquisition Decision Support and Assessment Processes Mitre Assessment of ICH Method“... the concept of the Interoperability Clearinghouse is sound and vital. Its developing role as an honest broker of all interoperability technologies, no matter what the source, is especially needed. Such efforts should be supported by any organization that wants to stop putting all of its money into maintaining archaic software and obtuse data formats, and instead start focusing on bottom-line issues of productivity and cost-effective use of information technology.”

11. AAM Models: A SOA enabled Decision FrameworkValidates and aligns business drivers with proven interoperable IT Services Knowledge Exchange Stake Holders Business Requirements, Policy, & Guidance • Phase 1: • Business Process: • Determination • Prioritization • Validation SDOs/Labs/ Universities Value Coalition Analysis Integrators Vendors/ISVs Govt CxOs Prioritized Business Requirements Normalized Business Models Solution Exist? • Phase 2: • Service Component • Feasibility • Architecture • SLAs Solution Patterns Y Align Proven Capabilities w/ business needs N Knowledge Exchange Service Oriented Specs and SLAs Model New Solution Normalized Service Components Solution Set Evidenced-Based Assessment • Phase 3: • Technology Assessment: • Selection • Certification • Interop • Openness Validated Past Performance Validated Blue Prints & Build of Materials Vetted Solution Architecture Analysis of Alternatives Solution Architecture Validation and Demonstrations COTS Comparative Analysis, Evidence Y N

12. ICH’s Architecture Assurance MethodA services oriented approach for architecture alignment Reference Models Associated Metrics BRM Business Process & Data Models Requirements & Capability Gaps Performance Metrics Core Business Mission Objectives Business Processes & Infrastructure Security Profiles Services Component Specifications Service Component Models & SLAs BRM Efficiency Appl Service Components Layer 1 Business Driven Top Down Infrastructure Service Components Layer N Common Evaluation Criteria SAIL Solution Frameworks Aligns with business needs Technology & Standards Profiles BRM Technical Reference Model & Metrics Interoperability, Fit, Finish Application Layer 2 Common Infrastructure Layer M Secure Solutions

13. AAM Process – Compliant with Reg’s & StatutesA step by step approach for assuring implementation success

14. Predictable Outcomes from ICH EngagementsConflict free Mechanisms for SOA Common Services & Solution Assessment A Standardized Architecture Method for: Aligning DODAF and FEA-PMO structure Capturing Business Process and Information Sharing Driving SOA Enabled IT Acquisition Lifecycle Assuring Services Integration and Contractor Mgt Capability Prioritization w/ SLAs and metrics Assessment Framework for: Vetted Business Processes and Requirements Assessing Vendor Service Components (COTS, GOTS, Open Source) Discerning Technology Feasibility/Risk Assessment. Verify market ability to perform Service Level Agreements Access to Critical Expertise and Knowledge Sources: Architecture and Acquisition Processes that work Healthcare Informatics Domain Expertise Emerging technologies, innovations and open source markets Industry best practices in IT Infrastructure and SOA Stake Holder facilitation and outreach Biz Value to PMs Mitigate deployment risk. AAM’s structure provide decision quality data earlier Concentrates on Capabilities and their Importance to Mission Formalizes the Prioritization Process Greater Access to Health Care Solutions Innovative approach for assuring successful IT Acquisitions

15. Optimizing Stake Holder Contributions in the IT Value Chain Office of the Secretary of Defense, DCIO (2001)”Since the value of the ICH to our programs increases rapidly through results sharing, we encourage the defense community and IT industry to participate directly in the public service initiative in terms of sponsorship and lessons learned"

16. Policy Value Chain: Agency Leadership, Congress, GAO & OMBCompliance Map • Provides view into agency IT planning process • Establishes metrics for mission fulfillment • Aligns IT Capital investment with Agency Mission • Leverages industry best practices/lessons learned • Gain access to industry expertise outside of lobbying/marketing arena. • Catalyst for advancing President’s Management Agenda Policies Compliance Congress, Agency Leadership: COO’s, CFO’s Auditors

17. Requirements Value Chain: Enterprise Users (CIO, CTO, Architects, PMs)Validating Requirements into Capabilities IT Management: CIO/CTO/CAO: Real World Requirements • Validate and Prioritize Requirements w/ business driven metrics • Identifies viable COTS/Open Source solutions, including small/innovative offerings • Establish Open Architectures and market readiness • Model/leverage relevant industry best practices, lessons learned • Assist in “what if” Modeling and Analysis of Alternatives • Provides In-Context, Just-In-Time Research Data via many communities of practice • Acquisition Strategies that have proven to deliver value and hold suppliers accountable Best Practices, Proven Solutions Congress, Agency Leadership: COO’s, CFO’s Auditors

18. SI Value Chain: Solution Integrator & ConsultantsVerifying past performance and ability to deliver Open Solutions IT Management: CIO/CTO/CAO: Integrators Re-usable BluePrints Integration Best Practices • Provides Solid Evidence for making COTS decisions • Shares Integration Success Record with Agency Planning process w/out OCI concerns • Finds Potential COTS Component Matches Based on Prior Similar Context Successes. ID small/innovative solution providers • Provides “What If” Modeling and Analysis for component composition • Provides Design Differentiation to Enhance Value Proposition to Customers • Outsource market research to conflict free entity Congress, Agency Leadership: COO’s, CFO’s Auditors

19. Value Chain Benefit: Standards Bodies, Academia and Industry GroupsAligning the theory of standards w/ market realities IT Management: CIO/CTO/CAO: SDOs, Academia & Industry Groups Integrators Solution Metrics • Captures Use-Cases that Support and Justify Standards • Use of standards/Engineering Spec in vetting of COTS solutions • Links Pertinent Standards to Products Features/functions • Provides Dynamic, but Solid Compliance Record (A119) • Enable collaboration between disparate industry groups • Provide IP and resources for framing complex issues. Research Projects Congress, Agency Leadership: COO’s, CFO’s Auditors

20. Innovators Value Chain: VARs, COTS vendors, Component Builders, Innovative BusinessesDefining the realm of the possible IT Management: CIO/CTO/CAO: SDOs, Academia & Industry Groups Integrators • Provide Buyers with the Contextualized Product Information in terms of business fit, interoperability and security. • Certifies Vendor Past Performance Based on 3rd-Party Validation/History in an architecture context • Increased Buyer’s Chances for Successful Use of Vendor Product --> Successful implementation! • Generates normalized COTS solution blue prints Normalized Spec & Customer References Open Market Place Integration Partners Reduced cost of sale Congress, Agency Leadership: COO’s, CFO’s Auditors VARs, COTS Vendors, Component Builders Small Businesses

21. AAM Use Cases; Assuring Transparency and Implementation success “The unique work you’ve done to frame issues about Interoperability effects on IA of systems is particularly encouraging. I believe it can contribute to ongoing work in the field of IA metrics and composition ongoing at DARPA and other agencies.” “I was pleasantly surprised at the effective and efficient use of funding I was able to invest in this effort as well as your ability to leverage other funding sources. You guys came through!” DARPA PM regarding ICH Information Assurance Modeling Project

22. Case Study: Streamlining the DoD Business IT Acquisition Process Challenge: Establish OSD BTA’s Capability Assessment Method (CAM) • Applied ICH new Acquisition Assurance Method • Developed IT Business Systems lifecycle entry/exit criteria for great transparency • Established enhanced Clinger Cohen Act process guide for OSD BTA CIO • Developed Value Chain Capability Assessment Methodology (CAM) • Established IT Acquisition Advisory Council to overcome cultural impediments. • Outcomes; IT Acquisition Reform we can believe in • Complemented Business Capability Lifecycle (BCL), providing analytical tools for framing decisions • Enabled actionable Clinger Cohen Act compliance that goes beyond check list • Enabled Component Acquisition Executive with means of judging business value of IT investments • Provided OSD BTA with alternative approach to DoD weapons systems style processes Office of the Secretary of Defense, DCIO (2001)”Since the value of the ICH to our programs increases rapidly through results sharing, we encourage the defense community and IT industry to participate directly in the public service initiative in terms of sponsorship and lessons learned" Past Performance

23. Case Study: Streamlining the IT Acquisition processSAF XC/AQ; AF Solution Assessment Process (ASAP) Challenge: Establish a common and repeatable AF Wide COTS assessment/acquisition process • Integrated ICH Architecture Assurance Method into all major AF IT components • Developed root cause of analysis of current weaknesses and deficiencies • Identified and integrated both AF and industry best practices into a common framework • Developed series of templates and input/exist criteria for each stage of the SDLC process • Outcomes • Provided a common, enterprise wide process designed for leveraging COTS • Augmented architecture process to address legacy and COTS capabilities • Reduce market research and analysis in a fraction of the cost and time by leveraging existing expertise and lessons learned of the market • Provided mechanisms for forcing adoption of 80% solution. “We have put to practice the AF Solution Assessment Process (ASAP) at the Air Force Communications Agency (AFCA) with some well documented success. It was developed with Interoperability Clearinghouse (ICH) and provides a structured and measurable IT assessment process with the agility to provide decision-quality assessments ranging from quick-looks to more in-depth capability-focused technology assessments and lightweight business case analysis.” General Mike Peterson, AF CIO Past Performance

24. Case Study: Validating SOA and Cross Domain SolutionsNavy PMW 160 CANES Program Establish an actionable solution architecture that leverages SOA & COTS implementation best practices Provide a standardized Solution Assessment Methodology to leverage best practices and mitigate deployment risk (compliment NESI). Establishes a Solution Architecture standard and public/private research partnership that maximizes use of commercial trends (COTS/Open Source solutions) via an actionable Open Architecture (OA) Enable Capability Based Acquisitions. Reveal Gaps in both requirement and industry offerings (define realm of the possible). Establish SOA performance metrics and SLAs that reflect real world limitations and hold suppliers accountable. Outcomes of ICH engagement; Proved out as a standardized IT Assessment & Solution Architecture process that will mitigate deployment risk. AAM assessment products used: Capability Determination and Metrics Service Component Prioritization and Alignment and Feasibility/Risk Assessment Demonstrated the feasibility and viability of using GOTS/COTS/Open Source products within the CANES Architecture Demonstrated a method and a plan to: Assess SOA Service Components for CANES Assess migration to Netcentric “need-to-share” systems Produced a large body of artifacts that are important for the architecture phase Challenge: Establish a enterprise ship board infrastructure for CANES Past Performance

25. Case Study: Digitizing the Worlds Largest Printer Government Printing Office Future Digital System Project Challenge: Structure requirements process to acquire a COTS based digital content mgt system • Applied ICH Architecture Assurance Method to GPO Future Digital System • Normalized and aligned requirements with market best practices • Provided strategy for cutting integration cost and managing risk • Leveraged industry best practices to avoid common failure patterns • Outcomes • Validated requirements against marketplace offerings • Developed contextual evaluation metrics derived from best practices • Improved confidence in technology decisions via business driven assessment framework • Significantly reduced time/cost/risk of acquisition process • Guided evolution of emerging Electronic Content Management Standards that drive interoperability (iECM Past Performance

26. Case Study: Most innovative Media Company Discovery Channel’s Enterprise Multi-media Web Services Project Challenge: Architect an enterprise web infrastructure to enable multi-media information sharing • Applied ICH Solutions Validation Program • Performed architecture baseline assessment • Provided guidance and selection support for Web-app server, VPN, portal, last-mile wireless connectivity • Outcomes • Validated requirements against marketplace offerings • Developed contextual evaluation metrics derived from best practices • Improved confidence in technology decisions via business driven assessment framework • Significantly reduced time/cost/risk of global implementation • Assured successful implementation of world wide, web based, multi-media solution by vetting interoperability issues prior to acquisition. Past Performance

27. Case Study: World Largest Healthcare Agency OSD HA’sGovernment Wide e-Healthcare program Challenge: Define information sharing architecture for patient record integration • Applied ICH Architecture Immersion Program • Developed architecture validation criteria to GCPR Program Office • Developed product selection guidelines for Prime Contractor • Applied ICH Architecture Assurance Method • Outcomes • Enabled award based on unambiguous design specs • Augmented architecture process to address legacy and COTS capabilities • Was able to cycle through market research and analysis in a fraction of the cost and time of traditional efforts. • Ensured viability of Solution Architecture in terms of; meeting HIPPA, security, and interoperability requirements “The ICH repository data and analysis methodologies was very helpful in supporting a quick turn around for [Information Assurance] section of COTS security products. Highly detailed ICH technology domain and product evaluation data comprised over 60% of this urgently needed [architecture] report”.Northrop Grumman on ICH’s GCPR support Past Performance

28. Case Study: World’s Largest Research Agency DARPA’s Information Assurance Architecture Project Challenge: model and simulate information assurance and interoperability attributes • Develop a method for represent EA heuristics, and use AI tools to model solution • Developed common criteria to represent each major IT class (DB, OS, Middleware) • Create profiles and templates that can represent complexity of market • Apply Java Expert System Shells (JESS) to build rules based AI tool • Take existing system specifications from DII COE to make decisions • Outcomes • Developed approach for modeling complex solutions based on rules based engine • Developed means of representing security attributes into a solution architecture • Developed a tool that could do “what if analysis” on the fly based on partial and incomplete data. • Demonstrated how component-based architectures could be applied to improve the richness of our planning and EA processes as a means of managing risk “The unique work you’ve done to frame issues about Interoperability effects on IA of systems is particularly encouraging. I believe it can contribute to ongoing work in the field of IA metrics and composition ongoing at DARPA and other agencies.” “I was pleasantly surprised at the effective and efficient use of funding I was able to invest in this effort as well as your ability to leverage other funding sources. You guys came through!” DARPA PM regarding ICH Information Assurance Modeling Project Past Performance

29. Case Study: Managing Successful ERP GSA FMS Project Challenge: making EA actionable, eliminate redundant Financial Mgt. Systems • Applied Value Chain Analysis • Developed metrics for FMS implementation success • Evaluated current EA products • Developed Value Chain assessment model • Moved EA effort into CFO office • Outcomes • Enhanced and normalized existing EA products • Identified key business processes required for implementation • Enabled senior management to interact with EA process for the first time. • Helped GSA go from “red” to “green” based on Value Chain effort • Identified over $100 M in potential savings via ICH Value Chain Approach OMB Report on ICH Architecture Assurance Method: The Road Map contains information on a variety of interrelated topics including, but not limited to, software and vendor performance, best practices, and lessons learned. The Road Map provides a web-based library to directly support those implementing financial management systems. The modernized program must continue to provide information and communication using current technology trends and tools. Past Performance