Experiences With the Evaluation of Complex Software Products Under the Common Criteria

1. How To Eat A Mammoth Experiences With the Evaluation of Complex Software Products Under the Common Criteria Gerald Krummeck (atsec), Bill Penny (IBM)

2. Agenda • Our Experience • Challenges from complex systems • Evaluations under the Common Criteria • The influence of complexity • Strategies in mastering complexity • Summary

3. atsec‘s Experience • Evaluation Labs in Germany, USA, Sweden • More than half of all OS evaluations performed world-wide • z/OS (IBM Mainframes) • z/VM (IBM Mainframes) • Linux (SuSE, Red Hat, Oracle) • AIX • Cray • PR/SM, AIX LPAR • Databases • IBM DB2 • Oracle DB • Tivoli System Management Products

4. IBM‘s experience • ISO 9001 Certified since 1993 • WW development organization • US, Canada, Germany, Australia, US • Mexico, Russia, China • Historically Independent • Long History of IT Management • Project Management • System Management • Process Control • Large Complex Systems • HW, SW • New Function and Service Models • Support Largest WW Business Requirements • High availability, security, integrity

5. Challenges from complex systems Dimensions of complexity in evaluations • Size of the product • Size of the TOE (what part will be evaluated) • Amount of security functions • Protection Profiles • Depth of evaluation (EAL) • Global distribution of development • Multi-national • Large number of organisational units

6. Evaluation under Common Criteria Security Target Design SecurityPolicyModel FunctionalSpecification High-Level Design Low-Level Design Implemen- tation Product Correspondence Guidance documentation Tests Vulnerability Analysis Development Process (Life Cycle) Processes Delivery and Operation Configuration Management

7. Influence of Complexity • Simple Systems • „Isolated“ evaluation possible • Without knowledge of its origin and heritage • Emphasis on design, test, guidance, vulnerability analysis • Complex Systems • Cannot be fully investigated • Need to find additional ways to establish assurance/trustworthiness • Establish trust in the development process

8. Example: IBM z/OS Version 1Release 8 • Size • Several Millions LOC (Assembler, PL/X, C, Java) • Over 30 years development history • Over 300 Manuals (120.000 pages) • Over 630 Claims on security functions in the ST • 10 development sites distributed globally • 10 CM systems • Common Corporate Standards and Processes • Toute la Gaule est occupée… Toute?

9. Interim Result • You cannot look at everything • But you don‘t need to • Security functions can be located quite accurately and can be tested thoroughly • Requires sufficient experience and product know-how of the evaluators • Development processes become very important • Build trust in the developer to comply with his duties for every piece that has not been scrutinized by the evaluators • Again: Evaluators need experience and product know-how: • It is an illusion to assume that everybody can perform a good evaluation just by applying the CC methodology (not everybody can eat the mammoth without choking on it) • Customers need to identify the right laboratory for them with evaluators skilled in their type of product

10. Strategies to master complexity • Not everything at once • How to eat the mammoth • Assistance • Site Certification

11. Not everything at once • Start modest • Focus on core functionality • Start with lower assurance level (EAL2 oder EAL3) • Pro: Get your first certificate in due time • Con: lower assurance level than competition • Example Linux: • Start with EAL2, restrictive configuration • Now EAL4, CAPP/LSPP, almost all packages included • In between: write low-level design, add audit functions

12. Example z/OS • MVS: Orange Book B1 (in the mist of times…) • V1R6 – 2005 • EAL3, CAPP+LSPP (multilevel security) • Core functions: RACF, BCP, JES2, CS390, … • V1R7 – 2006 • EAL4 • Additional security functions • V1R8 – 2007 • Major expansion of security functionality • V1R9 • …

13. How to eat a Mammoth? • Bite by bite, of course! • Don‘t become intimidated by the size • Don‘t try to swallow it in one piece, either • Important factors: • Experience • Confidence • Perseverance

14. Assistance • 2 Teams from evaluation lab • Evaluators • Working on-site with developers is beneficial • Additional testers with product know-how • Consultants • Help developer to gather evidence,prepare required documents • Do not influence product itself or developer‘s decisions • Experienced certifiers help, too

15. Developer committment • Multi-year committment • Strong project management to coordinate all participating organizations • Strong technical leadership • „Divide and Conquer“ • Strong leaders at distributed locations • Educate, track, report • Focus by area (ST, CM,HLD, Test) • Communicate with Evaluation Team • Open, early and frequent discussions

16. Site Certification • Reduce complexity of the evaluation by reference to certification of sites • Idea • Certify development process for one site • Re-use certificate in all applicable evaluations • BSI tasked with development of site certification methodology • Since 2005 development and test of certification process • 2006 first pilot certification • Acceptance in CC community • Still more experience needed.

17. Conclusion • Evaluation of complex products fits well in CC scheme • Medium to long term strategy (and committment!) • Start modest • Increase assurance level and functionality • Processes must fit • Find the right partner with experience and product know-how • ITSEF and certification body

18. Questions, Comments