1 / 51

Introduction (1)

guthrie-hynes
Download Presentation

Introduction (1)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The 8th International Common Criteria Conference, 25~27, Sep. 2007, Rome, Italy An empirical study on effort-ratio among EALs and product types for estimation of evaluation duration and costGang Soo LeeProfessor, Dept. of Computer EngineeringHan Nam University, Dae-jeon, 306-791, KOREAgslee@eve.hannam.ac.kr ICCC-2007, Rome, Gang Soo Lee

  2. Introduction (1) • Quality Assurance • Dependability : a critical quality attribute of IT products and systems : a function of security, safety, reliability and availability • Time & cost consuming activity for developers (or venders) and evaluators • Security Engineering • A set of methodologiesfor cost-effective development and evaluation of security systems • An instance of software engineering • Project management : a main activity of software engineering • Project : limited project resources (time, cost, man and tools) • CC evaluation project • Minimize time (duration) & cost of evaluation • Consider both the minimum EAL and security functionality ICCC-2007, Rome, Gang Soo Lee

  3. Introduction (2) • Developers (or vendors) and evaluators • Interests in evaluation cost and duration • This presentation • Results of research on estimation of evaluation cost/duration (2003 ~) • Survey on information about evaluation cost/duration at CC schemes and research results • Estimation of effort-ratio among Product-types • Estimation of effort-ratio among EALs of CC 2.3 and CC 3.1 • Combination of evaluation effort-ratio among EALs and Product-types • Analysis and evaluation ICCC-2007, Rome, Gang Soo Lee

  4. Background (1) • Poorly answered FAQs • How much does evaluation cost? • How long does evaluation take? • Reasons • Creative and mental activity : confirm, determine, execute/conduct/perform, test, analysis (= software development, architecture building, lawyer) • Depends on : evaluator’s environment (controllable), sponsor’s environment (hard to control, un-predictable) • Evaluator’s environment: evaluation tools in evaluation test laboratory, ability of evaluator, support form sponsors • Sponsor’s (developer) environment: complexity of TOE (PP, EAL, ST), sponsor’s support to evaluator, quality of deliverables, communication, evaluator's environment. ICCC-2007, Rome, Gang Soo Lee

  5. Background (2) • Treat as confidential information • Conventional guide on evaluation cost/duration [“corsec” company's FAQ and “CC user Guide” (1999)] • Composition of evaluation cost (view 1) • Testing lab fee: charge on an hourly basis • Modification of product to meet CC • Preparation of deliverables: Costs of document preparation vary depending on the quality and content in the product documentation, as well as the author's familiarity with CC, and the evaluation of the vendors’ class of products. • Validation fee: pay to the Certification Body ICCC-2007, Rome, Gang Soo Lee

  6. Background (3) • Composition of evaluation cost (view 2) • Pre-evaluation cost: Pre-evaluation consultancy may be taken either from an evaluation facility or from an independent consultant. • Evaluation cost: National schemes will differ in the approach that is taken to charges for formal evaluation. A typical approach would be to offer a fixed price for the evaluation, with a further sum requested for any re-work that has to be carried out following the identification of problems. • Certification cost • Internal cost: additional testing, additional analysis, addressing issues raised by the evaluators ICCC-2007, Rome, Gang Soo Lee

  7. Background (4) • Cost influence factors • EAL: The more assurance that is required, the more work that the evaluators are required to do. • Scope of security evaluated functionality • Design of TOE: Application of software engineering methodologies (e.g., structured-oriented SE, object-oriented SE, component-based SE, aspect-oriented SE) • Problems encountered: Where the evaluators encounter problems, either with the evaluation deliverables or with the product itself, there will be a need for remedial action, and some of the evaluation activities will need to be revisited. ICCC-2007, Rome, Gang Soo Lee

  8. Background (5) • Duration influence factors • EAL: assurance package claimed • The extent of security functionality • Product development timescales: timing of product releases • Quality of evaluation deliverables: If the documentation is clear, and conforms to requirements, then the evaluation will progress smoothly. • Availability of developer and evaluator resources • Quality of communication between developer and evaluator ICCC-2007, Rome, Gang Soo Lee

  9. Background (6) • Information on CC evaluation cost and duration • KISA’s evaluation cost • Researched on estimation of evaluation cost/duration for CC since 2003. • Evaluation effort-ratio among EALs and product-types of CC 2.1 • Calculate real evaluation cost for EAL1, EAL2, EAL3, EAL4 at KISA by using real cost data (in fiscal year 2000 ~ 2002) in 2005. • Even though the Republic of Korea has joined CCRA member as "Certificate Authorizing" in  May 9. 2006, tens of firewall products, introduction detection products, VPN and so on, have been evaluated and certified by using "K-criteria“ and CC since 1998. The K-criteria is previous IT security product evaluation criteria which is a modification of ITSEC. Evaluation level of K-criteria are K1 ~ K6. K1, K2, K3, K4 corresponds EAL1, EAL2, EAL3, EAL4, respectively.) ICCC-2007, Rome, Gang Soo Lee

  10. Background (7) • Evaluation cost data (informal) • Obtain evaluation cost/duration information (informal and rough) from various media such as documents, communication and homepage of evaluation schemes and facilities. • The information is informal and rough. • Decide base-line value from the obtained information. • Fig. 1 presents  F. Forge's data [7'th ICCC]. • Fig. 2 presents GAO's data [Information Assurance - National Partnership Offers Benefits, but faces considerable challenges, GAO-06-392, GAO, March 2006]. ICCC-2007, Rome, Gang Soo Lee

  11. Background (8) • Fig.1 • Fig.2 ICCC-2007, Rome, Gang Soo Lee

  12. Background (9) • SW development and test cost estimation • “Korea’s SW development cost estimation criteria” • based on B. Boehm’s COCOMO model and Function Point model • includes • recommended estimation method of SW development • SW maintenance and re-engineering • construction of system operational environment • construction of Database • information strategy planning (ISP) • COCOMO’s Project cost driver (1) Product types • required system reliability • complexity of system module ICCC-2007, Rome, Gang Soo Lee

  13. Background (10) • extent of documentation required • size of Database used • required % of reusable component (2) computer • execution time constraint • volatility of development platform • memory constraint (3) personal • capability of project analysis • personal continuity • programmer capability • programmer experience in project domain • analyst experience in project domain • language and tool experience • (4) project • use of software tool • development schedule compression • extent of multisite working and quality of inter-site communication ICCC-2007, Rome, Gang Soo Lee

  14. Background (11) • Activity of CC evaluation • confirm, determine, test, vulnerability analysis, etc. • SW development cost estimation model is not suitable for evaluation cost/duration of CC and CMVP evaluation.  • Hardware and material testing domain • Microsoft announced test cost • ‘platform’ test ($400), ‘designed for’ test ($600 ~ $11,000), ‘certified for’ test ($1,000$ ~ $30,000$), ‘retired’ test ($25,000 ~ $20,000) [https://partner.microsoft.com/global/] • Cisco security test cost • $5,400 ~ $7,750 [http://www.keylabs.com/cisco/csec/fees.html] ICCC-2007, Rome, Gang Soo Lee

  15. Background (12) • Problems • Results of KISA evaluation cost: specific evaluation environment such as Korea’s K-criteria, and Firewall and IDS products • F. Forge’s data and GAO’s data : too rough to estimate evaluation cost for each EALs • Germany  BSI scheme fee : not evaluation cost, but certification fee • There is no research on evaluation cost for each EALs of CC 2.3 and CC 3.1.  We need empirical study on effort-ratio among EALs and product types for estimation of evaluation duration and cost. ICCC-2007, Rome, Gang Soo Lee

  16. Effort-ratio among product-types (1) • Obtain and classification of PP and ST • Product-types • DB, Firewall, VPN, Network, OS, Smart Cards, Access Control, Key Recovery, IDS, MISC • Surveyed 33 PPs and 67s ST in August 2003 ICCC-2007, Rome, Gang Soo Lee

  17. Effort-ratio among product-types (2) • Analysis of PP/ST • When ST (or TOE) is developed, • 28% of ST has “PP conformance claim”. • 83.6% of ST use functional requirements in CC • Estimation of effort-ratio among product-types • effort-weight for all functional component by • ‘Function Point method’ • ‘Hierarchical relation among components’ • effort-amount for all functional components • effort-ratio among product-types from effort-amounts for all functional components ICCC-2007, Rome, Gang Soo Lee

  18. Effort-ratio among product-types (3) [evaluation effort-amount estimation algorithm] • FClass = {FAU, FCO, FCS, FDP, FIA, FMT, FPR, FPT, FRU, FTA, FTP}: a set of functional classes • fsi: a functional class (∈FClass). wfsi:aneffort-weight of fsi. WFSi:aneffort-amount of fsi • ffi: a functional family.  wffi:aneffort-weight of ffi. WFFi : aneffort-amount of ffi • fci : a functional component i.  wfci: an effort-weight of fci.FFCi:aneffort-amount of  fci • PP : a set of real PP • TYPE = {DB, Firewall, VPN, Network, OS, Smart card, Access Control, Key recovery, IDS, etc.}: a set of product-types ICCC-2007, Rome, Gang Soo Lee

  19. Effort-ratio among product-types (4) • PPTi: set of PP of Product-type typei.   ppti: real PPi in PPTi • FRi : functional requirements set used in ppti (i.e., FRi ⊆ FReq, FReq is total functional requirement in CC)            • fri: a functional component in FRi For all fsi in FClass, do   Get wfsi by means of Function Point method;end_do; For all ffi in fsjin FClass, do   Get wffi by means of Function Point method;end_do; For all fci in ffjin fsk in FClass, do   Get wfci by means of Hierarchical relation among components; // wfci = 1 or 2 or 3//; end_do; ICCC-2007, Rome, Gang Soo Lee

  20. Effort-ratio among product-types (5) For PPTi in TYPE, do   // analysis of 33 PPs For all ppti in PPTj , do     Get FRi ;    Compute Ui,j(usage-percentage of fri in FReq)               // Ui,j = (numbers of PPs using fri ) ÷ (numbers of PPs in TTPi ) ;  end_do; // Ui,jis usage ratio (0 ~ 1) of functional component j of typei // end_do For all i, j, k,do   WFCi,j,k = wfci , × wffj × wfsk ; end_do // WFCi,j,k is functional component in family ffj in class fsi For all typepin TYPE, do   For all k,do     EFF p = EFF p + WFC*,*,k× Uk,p; end_do; end_do   // EFF p is evaluation effort-amount of Product-type typep ICCC-2007, Rome, Gang Soo Lee

  21. Effort-ratio among product-types (6) • Product-type of VPN has more security functions than other product-type. • DB and OS, those seem to have larger amount of evaluation effort, has small effort-ratio, because only of security functions are evaluated. Effort-ratio among product-types ICCC-2007, Rome, Gang Soo Lee

  22. Effort-ratio among product-types (7) • Characteristics of algorithm • Usage-percentage • “usage-percentage” of specific function for each product-type • Function point • Relative effort-weight (i.e., complexity of evaluation of specific security function) among functional class and family in a specific product-type (e.g., OS) is estimated • An useful complexity metrics for SW development cost estimation • Empirically estimated functional point values of each functional classes and families from 4 graduate students who has 1~3 years of SW development • If effort-weight of FAU is 69.71, then that of class FCO is 58.49. • If effort-weight of FAU_ARP is 51.58, then that of class FAU_SAA is 72.28. ICCC-2007, Rome, Gang Soo Lee

  23. Effort-ratio among product-types (8) • Hierarchical relation among components • Effort-weight of a component in a family is defined by level of tree which present ‘hierarchical relations among functional components’ • For example, FAU_SAA’s tree (i.e., hierarchical relations) includes 4 components. • effort-weight of FAU_SAA.1 = 1 • effort-weight of FAU_SAA.2 and FAU_SAA.3 = 2 • effort-weight of FAU_SAA.4 = 3 ICCC-2007, Rome, Gang Soo Lee

  24. Effort-ratio among EALs (1) • Estimation method • Overview • Assurance component • ‘developer action elements’, ‘content and presentation of evidence (CPE) elements’, ‘evaluator action (EA) elements’ • Common evaluator action (CEA) • “The evaluator shall confirm that the information provided meets all requirements for content and presentation of evidence.” • Evaluation effort : proportion to numbers, amount (i.e., whole or part) and rigorousness (i.e., informal, semi-formal or formal) of the ‘CPE elements’ ICCC-2007, Rome, Gang Soo Lee

  25. Effort-ratio among EALs (2) 1. Identify and adjustment of numbers of ‘CPE elements’ for each assurance component • In some case, numbers of elements in ‘CPE elements’ of upper level of component is the same as that of lower component, even semantic (i.e., amount and rigorousness) of statement is different from each other. • Thus, we adjustment the numbers of ‘CPE elements’ for each assurance component. • Let AF.i.Ck (k = 1 , ... , #AF.i.C) be element statement of ‘CPE elements’ of assurance component i (denotes AF.i) in an Assurance Family AF,  #AF.i.C be number of elements,  #AF.i.C* be adjusted number of elements. Then following assertions should be satisfied: #AF.i.C ≤ #AF.i+1.C   (i=1, ..., n)         #AF.i.C* ≤ #AF.i+1.C*   (i=1, ..., n)         #AF.i.C ≤ #AF.i.C*   (i=1, ..., n) ICCC-2007, Rome, Gang Soo Lee

  26. Effort-ratio among EALs (3) • To Satisfy the assertion, following rules should be applied: • Rule 1: #AF.1.C* = #AF.1.C • Rule 2: #AF.i.C* = #AF.i.C + added # of statements × 0.5  ( i = 2, ...., n) (example, ‘part’ is changed to  ‘full’, ‘semi-formal’ changed to ‘formal’.) • Rule 3: if  #AF.i.C  > #AF.i+1.Cthen #AF.1+1.C* =  #AF.i.C* + 1    (for ADV_FSP of CC 3.1) • For example, results of adjustment of the six assurance components in ADV_FSP of CC 3.1 are present in next Table. • [Note: # of added statements of ADV_FSP.4 is “ - 1”. This is an error of CC 3.1. #ADV_FSP.4.C* = 9+1 = 10 according to Rule 3. ICCC-2007, Rome, Gang Soo Lee

  27. Effort-ratio among EALs (4) ICCC-2007, Rome, Gang Soo Lee

  28. Effort-ratio among EALs (5) 2. Evaluation effort-weight of EA elements • EA elements for each assurance components in CC are consisted of • one common EA. • zero or more component specific EA elements. • Next Tablepresents  result of effort-weight for EA element. • Empirically obtained by means of  • a question from real evaluators working in the Korea Information Security Agency (KISA), an evaluation facility of the Korea evaluation and certification scheme (KECS) in 2003. ICCC-2007, Rome, Gang Soo Lee

  29. Effort-ratio among EALs (6) ICCC-2007, Rome, Gang Soo Lee

  30. Effort-ratio among EALs (7) ICCC-2007, Rome, Gang Soo Lee

  31. Effort-ratio among EALs (8) • Rigorousness among EALs • confirm: used to indicate that something needs to be reviewed in detail, and that an independent determination of sufficiency needs to be made. The level of rigorous required depends on the nature of the subject matter. • demonstrate: refers to an analysis leading to a conclusion, which is less rigorous than a “proof”. • determine: requires an independent analysis to be made, with the objective of reaching a particular conclusion. It implies a truly independent analysis, usually in the absence of any previous analysis having been performed. • verify: similar in context to “confirm”, but has more rigorous connotations. ICCC-2007, Rome, Gang Soo Lee

  32. Effort-ratio among EALs (9) • effort-weight • relative amount or difficulty of evaluation of EA element in comparing to the common EA. • may be changed according to evaluation environment such as, evaluator's ability, evaluation tools, and so on. 3. Evaluation effort-weight of assurance components and families • Next table presents estimation of effort-weight of AVA_VAN of CC 3.1.  • Assumed effort-weight : ‘independent vulnerability analysis’ = 6.11, ‘implementation presentation’ = 7.11,  no level penetration test = 5.45, basic level penetration test = 4.45, enhanced-basic level penetration test = 5.45, moderate level penetration test = 6.45, high level penetration test = 7.45. ICCC-2007, Rome, Gang Soo Lee

  33. Effort-ratio among EALs (10) ICCC-2007, Rome, Gang Soo Lee

  34. Effort-ratio among EALs (11) • evaluation effort-weight of assurance component = adjusted number of elements + total effort-weight of assurance component - 1 (where, “- 1” because common EA is counted twice.) 4. Evaluation effort-weight of EALs • From the evaluation effort-weight of assurance component and assurance component definition table of EALs (EAL1 ~ EAL7), we evaluate effort-weights for each EALs of CC 2.3 and CC 3.1. • Next table presents evaluation effort-weight of EALs. ICCC-2007, Rome, Gang Soo Lee

  35. Effort-ratio among EALs (12) ICCC-2007, Rome, Gang Soo Lee

  36. Effort-ratio among EALs (13) 5. Evaluation effort-weight of composed assurance packages in CC 3.1 • Evaluation effort-weights for each CAP-A, CAP-B, CAP-C of composed TOE are estimated ICCC-2007, Rome, Gang Soo Lee

  37. Effort-ratio among EALs (14) • Result of estimation • Effort-ratio of EALi = (effort-weight of EALi) / (effort-weight of EAL1) • We distribute evaluation activity of AST class to all EAL of CC v 2.3, because the activity of ST evaluation is included in all EALs of CC 3.1. • Effort-ratio of CAPi = (effort-weight of CAPi) / (effort-weight of CAP-A) ICCC-2007, Rome, Gang Soo Lee

  38. Combination of EALs and Product-types (1) • EALs and Product-types • Next table presents combination of effort-ratios of EAL and effort-ratios of product-types. ICCC-2007, Rome, Gang Soo Lee

  39. Combination of EALs and Product-types (2) ICCC-2007, Rome, Gang Soo Lee

  40. Combination of EALs and Product-types(3) • Application of EAL and Product-type • Base-line ratio and value • base-line : a specific product-type and EALs of which evaluation cost and duration information have available (e.g., EAL4 of Firewall product) • base-line ratio : a effort-ratio of specific product-type and EAL (e.g., EAL4 of Firewall product is 2.03.) • base-line value : substitute cost (or duration) value for base-line (e.g., EAL4 of Firewall product is $25,000). It is estimated from historical evaluation cost data of real CC evaluation cases. • If we assume base-line cost of EAL4 of Product-type X as $25,000, then estimated evaluation costs for each EALs are presented in the next table . ICCC-2007, Rome, Gang Soo Lee

  41. Combination of EALs and Product-types(4) • Estimation of Evaluation Duration • Evaluation cost (or evaluation Man-Month or Man-Day) is proportioned to evaluation duration * number of evaluators. • If we fix number of evaluators (e.g., 3 persons), then evaluation cost is proportioned to evaluation duration. • Thus, evaluation duration is estimated by means of the cost estimation method. • If we assume base-line duration of EAL4 of Product-Type X as 12 months (i.e., 260 working days), then estimated evaluation durations for each EALs are presented in the table of next page. ICCC-2007, Rome, Gang Soo Lee

  42. Combination of EALs and Product-types(5) ICCC-2007, Rome, Gang Soo Lee

  43. Analysis and evaluation (1) • In lower EAL (EAL2 and EAL3), CC 3.1 has more evaluation effort-amount than CC 2.3. • If a TOE is evaluated by using CC 3.1, then it will takes more cost or duration then those of CC 2.3 in evaluation of lower EALs. • In case of composed TOE, difference of evaluation effort-ratios among CAPs are not so high: CAP-A : CAP-B : CAP-C = 1 : 1.26 : 1.31 ICCC-2007, Rome, Gang Soo Lee

  44. Analysis and evaluation (2) Trend of evaluation effort-ratio among EALs of CC 2.3 and CC 3.1 ICCC-2007, Rome, Gang Soo Lee

  45. Analysis and evaluation (3) • In KISA’s evaluation fee and Germany BSI (The Bundesamt fur Sicherheit in der Informationstechnik)’s certification fee, evidences of effort-ratio among EALs does not available. • There is no open evidence to answer “why does certification fee of EAL 7 charged 9.1 times larger than EAL1 in Class I type products. in Germany BSI’s certification fee.” • BSI’s fee policy, example product of classes are: • Class I: smart card readers • Class II: PC security desktops, smart card applications software, general applications software • Class III: standard OS, smart card hardware and OS, firewalls ICCC-2007, Rome, Gang Soo Lee

  46. Analysis and evaluation (4) ICCC-2007, Rome, Gang Soo Lee

  47. Conclusion (1) • Contributions • Estimate relative evaluation effort-ratioamong EALs of CC 2.3 and CC 3.1 • effort-weight of evaluator action elements • adjustment of number of statementsin ‘content and presentation of evidence elements’ • effort-weight of assurance components • effort-ratio among EALs • amount of evaluation efforts of EALs of CC 2.3 and CC 3.1 • Estimate relative evaluation effort-ratio among product-types from real PP/ST • usage percentage of specific function for each product-type • function point (effort-weight) of security functions (class, family, components) ICCC-2007, Rome, Gang Soo Lee

  48. Conclusion (2) • hierarchical relation of components in a family • Estimate relative evaluation effort-ratio among combination of EALs and product-types • Survey and estimate a base-line value (e.g., EAL4 of DB product-type is 150,000$) • Estimate evaluation duration • The results, even they are estimated and empirical value, will useful for • TOE sponsors as well as evaluation project managers in a new CC 3.1 based evaluation scheme • CC based evaluation project management ICCC-2007, Rome, Gang Soo Lee

  49. Conclusion (3) • Further study • This research should be continued as CC is to be evolved. • Empirical study on evaluation of test and evaluation cost • Get more real cost and duration data from CC evaluation facility, even they are dealt with confidential business data. • Develop a new estimation model by using conventional Software Engineering and Project Management methods such as COCOMO, Function Point, and so on. • Relative complexity of security functional classes, families, components, as well as real products, packages, PPs, product-types • Relative complexity of security assurance class, family, component, as well as EALs ICCC-2007, Rome, Gang Soo Lee

  50. Conclusion (4) • Note • CC evaluation cost and duration is depended on evaluation environment (e.g., tool, evaluator, policy, developer, ….) • This presentation gives overview of my approach on estimation of CC evaluation cost and duration. • Research on estimation of CC evaluation cost and duration is open problem. • Acknowledgement • This work has been supported by Wan-suck Lee, KISA, and Ji-hoon Jung, NSRI. • This presentation is supported by Kab-seung Kou, a Ph.D student of Hannam Univ. ICCC-2007, Rome, Gang Soo Lee

More Related