A Fault Model and Mutation Testing of Access Control Policies

1. Evan Martin and Tao Xie Department of Computer Science North Carolina State University A Fault Model and Mutation Testingof Access Control Policies

2. WWW 2007, Banff, Alberta, Canada Outline Motivation XACML Policy Testing Techniques Coverage Criteria Request Generation Request Selection Fault Model & Mutation Testing Experimental Results

3. WWW 2007, Banff, Alberta, Canada Motivation Digital information is Easy to access Easy to search Sensitive information requires access control mechanisms A growing trend is to specify access control policies in a specification language such as XACML

4. WWW 2007, Banff, Alberta, Canada Problem How to ensure the correct specification of access control policies? What you specify is what you get, but not necessarily what you want Systematic testing of access control policies Complements policy verification, which requires properties and may not support full policy features Just like software testing + software verification

5. WWW 2007, Banff, Alberta, Canada Software Testing Test Inputs Test Outputs Expected Outputs Program Policy Testing Software Testing Requests Responses Expected Responses Policy

6. WWW 2007, Banff, Alberta, Canada XACML Policy Structure eXtensible Access Control Markup Language OASIS standard XML syntax for specifying policies, requests, and responses A flexible and expressive language but complex and verbose Key concepts A PolicySet holds other policies or policy sets. A Policy is expressed as a set of rules. ARule have targets and a set of conditions that determine if the rule applies to a given request. Both rule and policy Combining Algorithms exist to reconcile conflicts.

7. WWW 2007, Banff, Alberta, Canada XACML Example <?xml version="1.0" encoding="UTF-8"?> <PolicySet xmlns="urn:oasis:names:tc:xacml:1.0:policy" PolicySetId="college" PolicyCombiningAlgId="urn:oasis:names:tc:xacml:1.0:policy-combining-algorithm:permit-overrides"> <Description>A College Policy on Grades</Description> <Target> <Subjects><AnySubject /></Subjects> <Resources><AnyResource /></Resources> <Actions><AnyAction /></Actions> </Target> <Policy PolicyId="fac" RuleCombiningAlgId="urn:oasis:names:tc:xacml:1.0:rule-combining-algorithm:permit-overrides"> <Description>Faculty Policy</Description> <Target> <Subjects> <Subject> <SubjectMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal"> <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string"> Faculty </AttributeValue> <SubjectAttributeDesignator AttributeId="role" DataType="http://www.w3.org/2001/XMLSchema#string" /> </SubjectMatch> </Subject> </Subjects> <Resources> <AnyResource /> </Resources> <Actions> <AnyAction /> </Actions> </Target> Policy Set Target Policy Target

8. WWW 2007, Banff, Alberta, Canada XACML Example <Rule RuleId="fac-assign-view-grades" Effect="Permit"> <Target> <Subjects><AnySubject /></Subjects> <Resources> <Resource> <ResourceMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal"> <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">ExternalGrades</AttributeValue> <ResourceAttributeDesignator AttributeId="resource-class" DataType="http://www.w3.org/2001/XMLSchema#string" /> </ResourceMatch> </Resource> <Resource> <ResourceMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal"> <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">InternalGrades</AttributeValue> <ResourceAttributeDesignator AttributeId="resource-class" DataType="http://www.w3.org/2001/XMLSchema#string" /> </ResourceMatch> </Resource> </Resources> <Actions> <Action> <ActionMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal"> <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">Assign</AttributeValue> <ActionAttributeDesignator AttributeId="command" DataType="http://www.w3.org/2001/XMLSchema#string" /> </ActionMatch> </Action> <Action> <ActionMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal"> <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">Receive</AttributeValue> <ActionAttributeDesignator AttributeId="command" DataType="http://www.w3.org/2001/XMLSchema#string" /> </ActionMatch> </Action> </Actions> </Target> </Rule> </Policy> Rule Target

9. WWW 2007, Banff, Alberta, Canada XACML Example <Policy PolicyId="stu" RuleCombiningAlgId="urn:oasis:names:tc:xacml:1.0:rule-combining-algorithm:permit-overrides"> <Description>Student Policy</Description> <Target> <Subjects> <Subject> <SubjectMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal"> <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">Student</AttributeValue> <SubjectAttributeDesignator AttributeId="role" DataType="http://www.w3.org/2001/XMLSchema#string" /> </SubjectMatch> </Subject> </Subjects> <Resources><AnyResource /></Resources> <Actions><AnyAction /></Actions> </Target> <Rule RuleId="stu-recieve-extgrades" Effect="Permit"> <Target> <Subjects><AnySubject /></Subjects> <Resources> <Resource> <ResourceMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal"> <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">ExternalGrades</AttributeValue> <ResourceAttributeDesignator AttributeId="resource-class" DataType="http://www.w3.org/2001/XMLSchema#string" /> </ResourceMatch> </Resource> </Resources> <Actions> <Action> <ActionMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal"> <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">Receive</AttributeValue> <ActionAttributeDesignator AttributeId="command" DataType="http://www.w3.org/2001/XMLSchema#string" /> </ActionMatch> </Action> </Actions> </Target> </Rule> </Policy> </PolicySet> Rules can have a more complicated condition tag here

10. WWW 2007, Banff, Alberta, Canada Coverage Definition Rationale: like in program testing, when the policy part containing a fault is not “covered”, the error is often not exposed. Given a request q and a policy P, we say qcovers a rule m in P if m contributes to the decision of q. The rule mcontributes to the decision of q if all Cond are satisfied by q. The sequence of rules and combining algorithms may also affect coverage. Rule coverage of P by requests Q = #rules covered by at least one request in Q #rules in P Similarly define policy coverage/condition coverage [ICICS 06]

11. WWW 2007, Banff, Alberta, Canada Random Request Generation The example policy: Subjects: Student, Faculty Actions: Assign, Receive Objects: External Grades, Internal Grades Model the set of attribute values as a vector of bits and randomize the bits 1 0 1 0 0 1 [ICICS 06] Student Faculty Assign Receive ExtGrades IntGrades

12. WWW 2007, Banff, Alberta, Canada Random Request Generation The example policy: Subjects: Student, Faculty Actions: Assign, Receive Objects: External Grades, Internal Grades Model the set of attribute values as a vector of bits and randomize the bits 0 1 1 0 1 1 [ICICS 06] Student Faculty Assign Receive ExtGrades IntGrades

13. Request Generation via Change-Impact Analysis (Cirg) [SESS 07] Synthesize versions (all-to-negate-one) for Margrave [Kisler et al. 05] Access Control Policy Policy Versions 1. Version Synthesis Requests 2. Change-impact analysis Counter examples 3. Request generation WWW 2007, Banff, Alberta, Canada

14. WWW 2007, Banff, Alberta, Canada Greedy Algorithm for Request Selection [ICICS 06] Requests Policy Increased Coverage? Reduced Request Set Yes No Discard

15. WWW 2007, Banff, Alberta, Canada Fault Model and Mutation Testing Fault model used to model things that could go wrong when constructing a policy Fault model is the underlying foundation of mutation testing [DeMillo et al. 78] (similar to fault injection): Policy is iteratively mutated to produce numerous mutants – each containing one fault Usages Measure fault-detection effectiveness of test generation or test selection Select tests

16. WWW 2007, Banff, Alberta, Canada Policy Mutation Testing Requests Responses Policy Mutation Operators Mutator Differ? Mutant Killed! Mutant Responses Mutant Policy

17. Mutation Operators Op Description PSTT Policy Set Target True PSTF Policy Set Target False PTT Policy Target True PTF Policy Target False RTT Rule Target True RTF Rule Target False RCT Rule Condition True RCF Rule Condition False CPC Change Policy Combining algorithm CRC Change Rule Combining algorithm CRE Change Rule Effect • Each operator mutates a different policy element: policy set, policy, rule, condition, and/or their associated targets and effects. WWW 2007, Banff, Alberta, Canada

18. WWW 2007, Banff, Alberta, Canada IF (faculty AND assign AND grades) ELSE IF (student AND receive AND grades) Permit ELSE Deny The CRE mutation operator is performed on each rule and changes the decision effect (Permit  Deny) Change Rule Effect (CRE) Example Permit

19. WWW 2007, Banff, Alberta, Canada IF (faculty AND assign AND grades) ELSE IF (student AND receive AND grades) Permit ELSE Deny The CRE mutation operator is performed on each rule and changes the decision effect (Permit  Deny) Change Rule Effect (CRE) Example Deny

20. WWW 2007, Banff, Alberta, Canada Experiment How strong is the correlation between the basic coverage criteria and fault-detection capability? Does test selection based on the coverage criteria produce reduced request sets with low loss of fault-detection capability? What are the individual characteristics of each mutation operator?

21. WWW 2007, Banff, Alberta, Canada Metrics Policy, rule, and condition coverage Test count, t Reduced test count, tred Mutant-killing ratio, m Reduced mutant-killing ratio, mred

22. Policies used in the experiment WWW 2007, Banff, Alberta, Canada

23. Basic Coverage Results • Cirg performs at least as well as the random set for rule coverage except for the mod-fedora policy because of a policy error. • Average # of selected random tests is smaller than Cirg • Random achieves 0% coverage on the conference policy WWW 2007, Banff, Alberta, Canada

24. Mutant-killing Results • Cirg outperforms the random technique in terms of fault-detection capability with a far fewer number of requests WWW 2007, Banff, Alberta, Canada

25. Mutant-killing ratios by subjects 0% coverage => 0% mutant-kill Coverage criteria not bad for selection not great either WWW 2007, Banff, Alberta, Canada

26. WWW 2007, Banff, Alberta, Canada Mutant-killing ratios by operators Likely equivalent mutants

27. WWW 2007, Banff, Alberta, Canada Conclusions Policy testing complements policy verification in assuring policy correctness. Just like software testing + software verification We have developed coverage criteria, test selection, and test generation. This paper presents fault model and mutation testing for policies Used to measure fault-detection effectiveness of test generation or test selection Used to select tests

28. Questions? WWW 2007, Banff, Alberta, Canada

29. WWW 2007, Banff, Alberta, Canada Hypothesis We can achieve a significant reduction in request-set size for large randomly generated request sets while maintaining equivalent policy, rule, and condition coverage. Reducing a request set based on coverage will not proportionately decrease its fault detection capability. Request generation via Change-impact analysis (Cirg) will have a higher fault-detection capability.

30. WWW 2007, Banff, Alberta, Canada XACML Coverage Criteria [ICICS 06] • Policy coverage – A policy is covered by a request if the policy is applicable to the request and the policy is encountered before the PDP has fully resolved the decision for the request. • Rule coverage – A rule is covered by a request if the rule is applicable to the request and the rule is encountered before the PDP has fully resolved the decision for the request. • Condition coverage – A condition must be encountered and evaluate to true and false to be fully covered.