Risks of using COTS in Information Technology Systems

1. Risks of using COTS in Information Technology Systems Symposium on Risk May 9, 2001 Ronald Kohl Titan Systems Co., AverStar Group rjkohl@prodigy.net

2. Agenda • Benefits of COTS • Challenges and Issues Using COTS • Risks of Using COTS in IT Systems • Requirements Engineering Differences • New Roles When Using COTS • Dormant Code, Overlap Code • Conclusions

3. Benefits of COTS • Reduced development costs:product development costs are shared over many users • Large ‘debugging base’: to find bugs, limitations and Dormant Code in the product (and the producer may actually fix these bugs) • Faster new technology adoption: products contain it without having to learn all about it yourself • Reduced development time/risk: when the COTS product provides all the features you need

4. Drawbacks of COTS Software • Integration of COTS with other stuff • Feature availability and timing (vaporware) • Bug fixes often only available in later releases • Difficult features dropped in later releases • Unwanted features can swell resource requirements • Unknown features (Dormant Code) can impact system • Long-term support of COTS software • Vendor survival (escrowing source code) • Evolution of features

5. Requirements Engineering Differences

6. Which Comes First, the Chicken (Requirements) or the Egg (COTS)? • Conventional system development sequence: • Needs identification • Needs analysis • System requirements definition • Allocation of functions to hardware, software and personnel • Derivation of software requirements • Write software, test, integrate, deploy • Problems with this software sequence • COTS do what they do, no more/less • COTS capabilities could/should drive detailed requirements • Lack of early COTS knowledge leads to gaps, glue code, etc • COTS volatility requires early and frequent (re)-assessments!

7. Process Changes • Early COTS capability assessment: • determines optional products/vendors • influences requirements (add/ delete/change) • early insight into acquisition/maintenance costs • early establishment of evaluation and acceptance criteria • Inputs into the ‘make/buy’ decision • Frequent COTS capability re-assessments • revalidate functionality, performance, interfaces • assure vendor viability • revalidate acquisition/maintenance costs • keep COTS product candidates consistent with evolving System requirements

8. What Risks arise if you don’t change? • Early COTS capability assessment: • uncertainty of commercial solution candidates, early • possible incomplete, incorrect requirements • uncertainty of acquisition and maintenance costs • lack of or incomplete evaluation/acceptance testing criteria • lack of insights into product/vendor options • Frequent COTS capability re-assessments • uncertainty of commercial solution candidates, often • lack of confidence in vendor viability • unknown changes in acquisition costs • lack of confidence that candidate solutions still match evolving requirements

9. New Roles for COTS based Systems

10. Background • COTS-Based Systems (CBS) lifecycle differs from custom built systems! • In CBS, • some requirements may come from candidate COTS solutions! • Mapping between COTS features and requirements is mandatory • COTS products need to play well with each other • COTS products may do undesirable things • COTS products, vendors and marketplaces change!! • COTS product configuration control is more complex

11. What new roles are needed? • Requirements Archeologist • Requirements Dating Game host • Plug n Player • Puzzle Box assessors • Tracker of Commercialization of Technology (techno-market watcher) • Herder of Cats (er, uh, COTS)

12. Requirements Archaeologist • Recognizes ‘interesting’ from ‘useful’ • Unearther of COTS features • System expertise, Domain expertise • Requirements ‘Dating Game’ host • Match COTS capabilities to Rqmts, Needs • Identify Overlap, Dormant, Gaps • Domain expertise, Marketplace savvy • Solution awareness is critical

13. Plug n Players • Determine if COTS play well together • Ability to build complex systems from small blocks • Fits square peg into round hole without much help • Puzzle Box Assessors • Determines precisely what COTS does • Inquisitive, curious, nefarious, likes challenges • Likes to think off-nominally, outside the box • Can make good things perform badly

14. Techno-market Tracker • Monitors the emergence of technologies • Monitors the commercialization of technology • Recognizes useful stuff • Produces technology assessment reports!! • Herder of Cats (er, uh, COTS) • A tougher CM job! • Monitors the new/obsolescing stuff • Now has to do CM on vendors/marketplaces • Requires proactive, investigative attitude

15. COTS Dormant Code and OverLap Code

16. Background • Ideally, there is always a 1-1 mapping between requirements and operational software • With increased use of COTS, this is not always the case • some requirements are not met w/COTS (custom/glue code needed) • some COTS features are not required (Dormant Code) • some products provide the same features (Overlap Code)

17. COTS 1 COTS 2 SubSys 2 SubSys 3 COTS 3 SubSys 1 SubSys 4 The ‘way we wish it were’ COTS picture

18. COTS Overlap Code COTS 1 COTS 2 SubSys 2 SubSys 3 Boundary Crossing Code COTS 4 SubSys 1 SubSys 4 COTS Dormant Code COTS 3 The realistic COTS picture

19. What are the Risks? • DC/OC identification and analysis • Impacts of activation & contingency plans • Disabling/prevention • Contractual impacts • Cost impacts

20. Overall Conclusions • COTS-based systems pose new challenges • Requirements requires early/frequent insight into products/vendors • New Roles can make/break a project or enterprise • Dormant/Overlap Code increases likelihood of undesirable behaviour • Awareness of and impact/risk assessment of these risks is critical to success of IT systems