Privacy and Contextual Integrity: Framework and Applications

1. Privacy and Contextual Integrity:Framework and Applications Anupam Datta CMU Joint work with Adam Barth, John Mitchell (Stanford), Helen Nissenbaum (NYU) and Sharada Sundaram (TCS)

2. Problem Statement • Is an organization’s business process compliant with privacy regulations and internal policies? • Examples of organizations • Hospitals, financial institutions, other enterprises handling sensitive information • Examples of privacy regulations • HIPAA, GLBA, COPPA, SB1386 Goal: Develop methods and tools to answer this question

3. Privacy Project Space What is Privacy? [Philosophy, Law, Public Policy] Formal Model, Policy Language, Compliance-check Algorithms [Programming Languages, Logic] Implementation-level Compliance [Software Engg, Formal Methods] Data Privacy [Databases, Cryptography]

4. Project Overview • What is privacy? • Conceptual framework • Policy language • Privacy laws including HIPAA, COPPA, GLBA expressible • Compliance-check algorithms • Does system satisfy privacy and utility goals? • Case studies • Patient portal deployed at Vanderbilt Hospital • UPMC (ongoing discussions) • TCS

5. Contextual Integrity [N2004] • Philosophical framework for privacy • Central concept: Context • Examples: Healthcare, banking, education • What is a context? • Set of interacting agents in roles • Roles in healthcare: doctor, patient, … • Norms of transmission • Doctors should share patient health information as per the HIPAA rules • Purpose • Improve health

6. MyHealth@Vanderbilt Workflow Humans + Electronic system Health Answer Yes! except broccoli Appointment Request Secretary Health Question Health Question Now that I have cancer, Should I eat more vegetables? Doctor Patient Health Question Health Answer Utility: Schedule appointments, obtain health answers Nurse Privacy: HIPAA compliance+

7. Health Answer Yes! except broccoli Health Question Now that I have cancer, Should I eat more vegetables? MyHealth@Vanderbilt Improved Health Answer Appointment Request Secretary Doctor Patient Health Question Health Question • Message tags used for policy enforcement • Minimal disclosure Health Answer Nurse Responsibility: Doctor should answer health questions

8. Privacy vs. Utility • Privacy • Certain information should not be communicated • Utility • Certain information should be communicated • Tension between privacy and utility Workflows Violate Privacy Feasible Workflows Permissiveness “Minimum necessary” Violate Utility

9. Contextual Integrity Design-time Analysis: Big Picture Norms Purpose Business Objectives Privacy Policy Utility Checker (ATL*) Privacy Checker (LTL) Business Process Design Utility Evaluation Privacy Evaluation Assuming agents responsible

10. Auditing: Big Picture Business Process Execution Run-time Monitor Audit Logs Audit Algos Privacy Policies Utility Goals Policy Violation + Accountable Agent Agents may not be responsible

11. In more detail… • Model and logic • Privacy policy examples • GLBA – financial institutions • MyHealth portal • Compliance checking • Design time analysis (fully automated) • Auditing (using oracle) Language can express HIPAA, GLBA, COPPA [BDMN2006]

12. Health Question Now that I have cancer, Should I eat more vegetables? Model Inspired by Contextual Integrity Bob Alice • Communication via send actions: • Sender: Bob in role Patient • Recipient: Alice in role Nurse • Subject of message: Bob • Tag: Health Question • Message: Now that …. • Data model & knowledge evolution: • Agents acquire knowledge by: • receiving messages • deriving additional attributes based on data model • Health Question Protected Health Information contents(msg) vs. tags (msg)

13. Model [BDMN06, BDMS07] • State determined by knowledge of each agent • Transitions change state • Set of concurrent send actions • Send(p,q,m) possible only if agent p knows m K0 A13 A11 K13 A12 K11 ... K12 ... Concurrent Game Structure G = <k, Q, , , d, >

14. Logic of Privacy and Utility • Syntax  ::= send(p1,p2,m) p1 sends p2 message m | contains(m, q, t) m contains attrib t about q | tagged(m, q, t) m tagged attrib t about q | inrole(p, r) p is active in role r | t  t’ Attrib t is part of attrib t’ |    |  | x.  Classical operators | U | S | O Temporal operators | <<p>> Strategy quantifier • Semantics Formulas interpreted over concurrent game structure

15. Specifying Privacy • MyHealth@Vanderbilt In all states, only nurses and doctors receive health questions G  p1, p2, q, m send(p1, p2, m)  contains(m, q, health-question)  inrole(p2, nurse)  inrole(p2, doctor)

16. Specifying Utility • MyHealth@Vanderbilt Patients have a strategy to get their health questions answered  p inrole(p, patient)  <<p>> F  q, m. send(q, p, m)  contains(m, p, health-answer)

17. MyHealth Responsibilities • Tagging Nurses should tag health questions G p, q, s, m. inrole(p, nurse)  send(p, q, m)  contains(m, s, health-question)  tagged(m, s, health-question) • Progress • Doctors should answer health questions G p, q, s, m. inrole(p, doctor)  send(q, p, m)  contains(m, s, health-question)  F m’. send(p, s, m’)  contains(m’, s, health-answer)

18. Sender role Attribute Subject role Recipient role Temporal condition Gramm-Leach-Bliley Example Financial institutions must notify consumers if they share their non-public personal information with non-affiliated companies, but the notification may occur either before or after the information sharing occurs

19. Workflow Design Results • Theorems: Assuming all agents act responsibly, checking whether workflow achieves • Privacy is in PSPACE (in the size of the formula describing the workflow) • Use LTL model-checking algorithm • Utility is decidable for a restricted class of formulas • ATL* model-checking is undecidable for concurrent game structures with imperfect information, but decidable with perfect information • Idea: • Check that all executions satisfy privacy and utility properties • Definition and construction of minimal disclosure workflow Algorithms implemented in model-checkers, e.g. SPIN, MOCHA

20. Auditing Results • Who to blame? Accountability • Irresponsibility + causality • Design of audit log • Use Lamport causality structure, standard concept from distributed computing • Algorithms • Finding agents accountable for policy violation in graph-based workflows using audit log • Finding agents who act irresponsibly using audit log • Algorithms use oracle: • O(msg) = contents(msg) • Minimize number of oracle calls

21. Conclusions • Framework inspired by contextual integrity • Business Process as Workflow • Role-based responsibility for human and mechanical agents • Compliance checking • Workflow design assuming agents responsible • Privacy, utility decidable (model-checking) • Minimal disclosure workflow constructible • Auditing logs when agents irresponsible • From policy violation to accountable agents • Finding irresponsible agents • Case studies • MyHealth patient portal deployed at Vanderbilt University hospital • Ongoing interactions with UPMC Automated Using oracle

22. Future Work • Framework • Do we have the right concepts? • Adding time , finer-grained data model • Priorities of rules, inconsistency, paraconsistency • Compliance vs. risk management • Privacy principles • Minimum necessary one example; what else? • Improve algorithmic results • Utility decidability; small model theorem • Auditing algorithms • Privacy analysis of code • Current results apply to system specification • More case studies • Focusing on healthcare • Detailed specification of privacy laws • Immediate focus on HIPAA, GLBA, COPPA • Legal, economic incentives for responsible behavior

23. Publications/Credits • A. Barth, A. Datta, J. C. Mitchell, S. Sundaram Privacy and Utility in Business Processes, to appear in Proceedings of 20th IEEE Computer Security Foundations Symposium, July 2007. • A. Barth, A. Datta, J. C. Mitchell, H. Nissenbaum Privacy and Contextual Integrity: Framework and Applications, in Proceedings of 27th IEEE Symposium on Security and Privacy , pp. 184-198, May 2006. Work covered in The Economist, IEEE Security & Privacy editorial

24. Thanks Questions?

25. Additional Technical Slides

26. Related Languages • Legend:  unsupported o partially supported  fully supported • LPU fully supports attributes, combination, temporal conditions Utility not considered

27. Deciding Utility • ATL* model-checking of concurrent game structures is • Decidable with perfect information • Undecidable with imperfect information • Theorem: There is a sound decision procedure for deciding whether workflow achieves utility • Intuition: • Translate imperfect information into perfect information by considering all possible actions from one player’s point of view

28. Local communication game • Quotient structure under invisible actions, Gp • States: Smallest equivalence relation K1 ~p K2 if K1 K2 and a is invisible to p • Actions: [K]  [K’] if there exists K1 in [K] and K2 in [K’] s.t. K1 K2 • Lemma: For all LTL formulas  visible to p, Gp |= <<p>> implies G |= <<p>>

29. Auditing Results • Definitions • Policy compliance, locally compliant • Causality, accountability • Design of audit log • Algorithms • Finding agents accountable for locally-compliant policy violation in graph-based workflows using audit log • Finding agents who act irresponsibly using audit log • Algorithms use oracle: • O(msg) = contents(msg) • Minimize number of oracle calls

30. Policy compliance/violation Contemplated Action Judgment Policy Future Reqs History • Strong compliance [BDMN2006] • Action does not violate current policy requirements • Future policy requirements after action can be met • Locally compliant policy • Agents can determine strong compliance based on their local view of history

31. Causality • Lamport Causality [1978] “happened-before”

32. Accountability & Audit Log • Accountability • Causality + Irresponsibility • Audit log design • Records all Send(p,q,m) and Receive(p,q,m) events executed • Maintains causality structure • O(1) operation per event logged

33. Auditing Algorithm • Goal Find agents accountable for a policy violation • Algorithm(Audit log A, Violation v) • Construct G, the causality graph for v in A • Run BFS on G. At each Send(p, q, m) node, check if tags(m) = O(m). If not, and p missed a tag, output p as accountable • Theorem: • The algorithm outputs at least one accountable agent for every violation • of a locally compliant policy in an audit log • of a graph-based workflow that achieves the policy in the responsible model

34. Proof Idea • Causality graph G includes all accountable agents • Accountability = Causality + Irresponsibility • There is at least one irresponsible agent in G • Policy is satisfied if all agents responsible • Policy is locally compliant • In graph-based workflows, safety responsibilities violated only by mistagging • O(msg) = tags(msg) check identifies all irresponsible actions

35. MyHealth Example • Policy violation: Secretary Candy receives health-question mistagged as appointment-request • Construct causality graph G and search backwards using BFS Candy received message m from Patient Jorge. • O(m) = health-question, but tags(m) = appointment-request. • Patient responsible for health-question tag. • Jorge identified as accountable