SoK: Security and Privacy in Machine Learning

1. SoK: Security and Privacy in Machine Learning Nicolas Papernot∗, Patrick McDaniel∗, Arunesh Sinha†, and Michael P. Wellman† ∗ Pennsylvania State University † University of Michigan {ngp5056,mcdaniel}@cse.psu.edu, {arunesh,wellman}@umich.edu Presented by Jonny Dowdall CSE 914 2 Feb 2019

2. What is machine learning? Relevant terms A few math proofs Threats What are the major security threats in machine learning? Solutions What can we do about these threats? Machine Learning Objectives

3. Machine Learning An Over-Simplification

4. Machine Learning A computer’s ability to infer something based on experience rather than being explicitly programmed. An Over-Simplification

5. Machine Learning Given some input, produce some output. Supervised Learning

6. Machine Learning Train by providing explicit input and output pairs. Supervised Learning

7. Machine Learning Identify patterns in unlabeled data. Unsupervised Learning

8. Machine Learning No training in the conventional sense. Unsupervised Learning

9. Machine Learning Given a state, perform an action that leads you to an eventual desired state. Reinforcement Learning

10. Machine Learning No explicitly correct output for any input. Reinforcement Learning

11. Machine Learning Train by having an agent interact with an environment. Reinforcement Learning

12. Machine Learning Reward/penalize agent for reaching desired states. Reinforcement Learning

13. Machine Learning Typically, input data is represented as a vector (or matrix/tensor) of values. Call these input values features. Features

14. Machine Learning Some function transforms the input features into target output(s). Parameters

15. Machine Learning Function is typically a weighted sum of input features. Parameters

16. Machine Learning Call the weights parameters. Parameters

17. Machine Learning Start with random weights. Loss function

18. Machine Learning Grab a sample from the training data and compute some (bogus) output value(s). Loss function

19. Machine Learning The loss function tells us how far predicted output is from ground-truth output. Loss function

20. Machine Learning The gradient of the loss function tells us how to change our weights. Gradient

21. Machine Learning Next time we see that training sample, the loss should be smaller. Training

22. Machine Learning Rinse and repeat with all training samples until loss is low and weights stop changing. Training

23. Machine Learning Now, use these weights to predict values for new, unobserved inputs. This is our model. Inference

24. Confidentiality/Privacy Uncovering ML model itself, which may be confidential intellectual privacy. Uncovering users’ data, which may be private (model input/output). Integrity Exploiting the model to produce outputs that don’t correspond to patterns in training data. Availability Preventing access to an output or action induced by a model output. Threats (CIA)

25. Confidentiality/Privacy Uncovering ML model itself, which may be confidential intellectual privacy. Uncovering users’ data, which may be private (model input/output). Exploiting the model to produce outputs that don’t correspond to patterns in training data. Preventing access to an output or action induced by a model output. Integrity Threats (CIA)

26. White-box vs. Black-box Adversaries White-box Black-box • Access to the machine-learning model internals • Can see parameters/architecture • Only able to interact with model • Can provide input and view output

27. Papernot et al.

28. Confidentiality

29. Model extraction • Recover model parameters by testing many input output pairs (Tramer et al). • Requires access to class probabilities.

30. Privacy

31. Membership Attack • Test whether or not a data point is in the training set. • Exploit differences on model’s confidence to identify points that were trained on (Shokri). • Generate synthetic data until model produces an output with very high confidence.

32. Model Inversion • Fredrikson et al. present the model inversion attack. • Given the output of a model, predict the input. • For a medicine dosage prediction task, they show that given access to the model and auxiliary information about the patient’s stable medicine dosage, they can recover genomic information about the patient.

33. Analyzes the privacy guarantees of algorithms. Differential Privacy Framework

34. An algorithm’s output should not differ significantly statistically for two versions of the data differing by only one record (Dwork et al). Differential Privacy Framework

35. Add randomization to the machine learning pipeline. Solutions?

36. Randomly add noise to every input while training. Solutions? Local Privacy

37. E.g. Google Chrome collects user data with a probability q that the data is real and 1-q that the data is random. Resulted in meaningful and privacy-preserving statistics (Erlingsson) Solutions? Local Privacy

38. It was shown that adding random noise to the loss function during training provides differential privacy (Chaudhuri). Solutions? Perturbing loss

39. Randomly perturbing gradients before applying parameter updates guarantees even stronger differential privacy (Abadi). Solutions? Perturbing gradients

40. Randomly perturb output values. Solutions? Noisy output

41. This method degrades performance. Solutions? Noisy output

42. “Precisely quantifying the learning algorithm’s sensitivity to training points is necessary to establish differential privacy guarantees. For nonconvex models (e.g., neural nets), current loose bounds on sensitivity require that learning be heavily randomized to protect data—often at the expense of utility.”

43. Integrity

44. Training attack • Poisoning: input/label manipulation • Test distribution is different from training distribution • Model learns to behave in an unintended way • Example • Intentionally label genuine emails as spam

45. Assumption: Poisoned data points are typically outside the expected input distribution. Solution: Reduce the influence of outliers in training. Solutions?

46. Use a modified PCA algorithm: • Maximize the variance between training samples while reducing variance of outliers. (Rubinstien et al.) Solutions? Reduce outlier sensitivity

47. Adding a regularization term to the loss function, which in turn reduces the model sensitivity to outliers (Stempfel and L. Ralaivola). Solutions? Reduce outlier sensitivity

48. Inference attack • Adversarial example • Exploit weakness in model • Perturb a sample until it is misclassified • E.g. How can we get past malware detection by changing our malware as little as possible?

49. Inference attack • The more information provided in the output, the easier this is to exploit. • Especially easy when the system is an oracle. • Adversary can issue queries for any chosen input and observe the model output.

50. Inference attack • Lowd and Meek introduce ACRE learnability, which poses the problem of finding the least cost modification to have a malicious input classified as benign using a polynomial number of queries to the ML oracle.