From IF-Then Else To Deep Learning

1. From IF-Then ElseToDeep Learning Prof.dr. Bart Baesens Department of Decision Sciences and Information Management, KU Leuven (Belgium) Bart.Baesens@kuleuven.be Twitter/Facebook/YouTube: DataMiningApps www.dataminingapps.com

2. Presenter: Bart Baesens • Studied at KU Leuven (Belgium) • Business Engineer in Management Informatics, 1998 • PhD. in Applied Economic Sciences, 2003 • PhD. : Developing Intelligent Systems for CreditScoring Using Machine Learning Techniques • Professor at KU Leuven, Belgium • Research: Big Data & Analytics, Credit Risk, Fraud, Marketing, … • YouTube/Facebook/Twitter: DataMiningApps • www.dataminingapps.com • Bart.Baesens@kuleuven.be

3. My Example Publications

4. If-Then Else Rules • Expert-based Rules • Obtained from business experts using knowledge elicitation • Subjective but not to be frowned upon as inferior • Very popular in e.g. fraud detection • Data-based Rules • Obtained from data using analytical techniques (e.g. rule induction, decision trees) • Confirmatory versus novel patterns

5. The Analytics Process Model Baesens, 2015.

6. Example 1: Predicting Customer Churn Target Manually Engineered Features

7. Example 2: Predicting Customer Lifetime Value (CLV) Target Manually Engineered Features

8. Analytical industry benchmarks • Linear/Logistic Regression • Example applications: • Credit Scoring • Medical applications

9. Analytical industry benchmarks • Decision trees • Example applications • Credit scoring (pre-processing) • Churn prediction

10. Analytical industry benchmarks • Neural Networks Recency Frequency Output Monetary

11. Analytical Industry Benchmarks Weights Inputs Output • Neural Networks w1 Recency w2  Sum f(sum) Frequency w3 sum = wnIn Monetary w4 1 f: transformation function (tanh, exponential, linear, etc.)

12. Analytical industry benchmarks • Neural Networks

13. Analytical industry benchmarks • Neural networks (contd.) • Only 1 single hidden layer (shallow neural networks)! • Hidden layer extracts features from the data • Features are now automatically learned by the network (↔ manually engineered features) • But: computational requirements remain key concern! • Example applications: • Fraud detection (but not prevention!) • Response modeling • Forecasting (e.g. time series)

14. Analytical evaluation criteria • Accuracy • Interpretability • Operational Efficiency • Economical cost DEEP LEARNING Note: Profit Driven Analytics! • ProfLogit (Stripling, vandenBroucke, Snoeck, Baesens, 2017) • ProfTree (Höppner, Stripling, vandenBroucke, Baesens, Verdonck, 2017) • ProfARIMA (Van Calster, Baesens, Lemahieu, 2017)

15. Analytical Industry Benchmarks Lessmann S., Baesens B., Seow H.V., Thomas L.C., Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research, European Journal of Operational Research, 2015. Baesens B., Van Gestel T., Viaene S., et al. Benchmarking State of the Art Classification Algorithms for Credit Scoring, Journal of the Operational Research Society, 2003.

16. White versus Black Box applications • White box applications • Credit scoring (Basel, IFRS 9) • Medical diagnosis (FDA) • Fraud detection (and prevention!) • Black box applications • Response modeling • Image analytics (clustering, categorization, etc.) • Text analytics (clustering, categorization, etc.)

17. Deep Learning Triggers • Emergence of GPUs (Nvidia) and TPUs (Google) • Parallel computing • Cloud computing • NoSQL databases • Unstructured data • Software • TensorFlow (Google) • Keras, Theano, Caffe, Torch, Lasagna

18. Deep learning • Essentially a multi-layer neural network

19. Applications of Deep Learning • Image analytics • Audio/Video analytics • Textual analytics • Recommender systems • Music generation • Gaming

20. Deep Learning: Examples Image coloring Image captioning Object detection Translating text from image http://www.yaronhadad.com/

21. Deep Learning: Examples http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/ https://nl.mathworks.com/help/nnet/convolutional-neural-networks.html

22. Deep Learning: Examples http://www.kdnuggets.com/2017/05/deep-learning-extract-knowledge-job-descriptions.html?utm_content=buffera9696&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer predicting a job title for a job description

23. Deep Learning: Blooper • US Army wanted to use neural networks to automatically detect camouflaged enemy tanks • Trained a neural net on 50 photos of camouflaged tanks in trees, and 50 photos of trees without tanks • The neural network classified all test set photos 100% correctly • However, …. • http://www.kdnuggets.com/2017/08/visualizing-convolutional-neural-networks-open-source-picasso.html

24. Beyond the Buzz • Quotes from http://www.kdnuggets.com/2017/08/deep-learning-not-ai-future.html • “Too many startups and products are named “deep-something”, just as buzzword: very few are using DL really” • “Decision Trees like XGBoost are not making headlines, but silently beat DL at many Kaggle tabular data competitions.” • “Legal nightmare of DL decisions, that even if correct, can’t be explained when legally questioned” • “For many tasks, Deep Learning AI is or will become illegal, not compliant” • “GDPR, about automated decision-making, requires the right to an explanation, and to prevent discriminatory effects based on race, opinions, health, etc.” • “Fines for noncompliance are 4% of global revenue.”

25. Interpretable “Deep Learning” • Neural Network Rule Extraction • Extract rules that mimic the behavior of the neural network • Combine the performance of theneural network with the readabilityof If-Then Rules • Best of both Worlds approach! • Baesens, PhD Thesis, 2003.

26. Interpretable “Deep Learning” Baesens, Setiono, Mues, Vanthienen, 2003

27. Interpretable “Deep Learning” • IF Term > 12 Months AND Purpose = Cash Provisioning AND Savings Account ≤ 12.40 Euro AND Years Client ≤ 3 THEN Applicant = Bad • IF Term > 12 Months AND Purpose = Cash Provisioning AND Owns Property = No AND Savings Account ≤ 12.40 Euro THEN Applicant = Bad • IF Purpose = Cash Provisioning AND Income > 719 Euro AND Owns Property = No AND Savings Account ≤ 12.40 AND Years Client ≤ 3 THEN Applicant = Bad • IF Purpose = Second-Hand Car AND Income > 719 Euro AND Owns Property = No AND Savings Account ≤ 12.40 Euro AND Years Client ≤ 3 THEN Applicant = Bad • IF Savings Account ≤ 12.40 Euro AND Economic Sector = Sector C THEN Applicant = Bad • Default class: Applicant = Good

28. Interpretable “Deep Learning” Baesens, Setiono, Mues, Vanthienen, 2003

29. Conclusions (Bart’s perspective!) • What’s in a name! • Deep learning, machine learning, smart learning, etc. • Gap between academic research and business • Impact of Deep learning • Structured Data: No! • Unstructured Data: Yes! • Role of interpretability • Regulatory compliance (GDPR, antitrust law, Basel, Solvency, …) • Rule extraction (back to ‘If Then’ rules) • Graphical models (e.g. Bayesian networks)

30. Sign Up for Our New Book! www.pdbmbook.com