Advanced Methodologies for Predictive Data-Analytic Modeling

1. Advanced Methodologies forPredictive Data-Analytic Modeling Vladimir Cherkassky Electrical & Computer Engineering University of Minnesota – Twin Cities cherk001@umn.edu Presented at Chicago Chapter ASA, May 6, 2016 Electrical and Computer Engineering 1 1

2. Part 1: Motivation & Background Background - Big Data and Scientific Discovery - Philosophical Connections - Modeling Complex Systems Two Data-Analytic Methodologies Basics of VC-theory Summary 2 2

3. Growth of (biological) Datafrom http://www.dna.affrc.go.jp/growth/D-daily.html 3 3

4. Practical and Societal Implications Personalized medicine Genetic Testing: already available at $300-1K 4 4

5. Typical Applications Sparse High-Dimensional Data number of samples ~ n << d number of features Complex systems underlying first-principle mechanism is unknown Ill-posed nature of such problems  only approximate non-deterministic models • Genomics • Medical imaging (i.e., sMRI, fMRI) • Financial • Process Control • Marketing …… 5 5

6. What is Big Data? Traditional IT infrastructure Data storage, access, connectivity etc. Making sense / acting on this data Data  Knowledge  Decision making always predictive by nature Focus of my presentation - Methodological aspects of data-analytic knowledge discovery 6 6

7. Scientific Discovery Combines ideas/models and facts/data First-principle knowledge: hypothesis  experiment  theory ~ deterministic, causal, intelligible models Modern data-driven discovery: s/w program + DATA  knowledge ~ statistical, complex systems Two different philosophies 7 7

8. History of Scientific Knowledge Ancient Greece: Logic+deductive_reasoning Middle Ages:Deductive (scholasticism) Renaissance, Enlightment: (1) First-Principles (Laws of Nature) (2) Experimental science (empirical data) Combining (1) + (2)  problem of induction Digital Age: the problem of induction attains practical importance in many fields 8 8

9. Induction and Predictive Learning Induction: aka inductive step, standard inductive inference Deduction: aka Prediction

10. Problem of Induction in Philosophy Francis Bacon: advocated empirical knowledge (inductive) vs scholastic David Hume: What right do we have to assume that the future will be like the past? Philosophy of Science tries to resolve this dilemma/contradiction between deterministic logic and uncertain nature of empirical data. Digital Age: growth of empirical data, and this dilemma becomes important in practice. 10 10

11. Cultural and Psychological Aspects All men by nature desire knowledge Man has an intense desire for assured knowledge Assured Knowledge ~ belief in - religion (much of human history) - reason (causal determinism) - science / pseudoscience - data-analytic models (~ Big Data) - genetic risk factors …

12. Knowledge Discovery in Digital Age Most information in the form of digital data Can we get assured knowledge from data? Big Data ~ technological nirvana data + connectivity  more knowledge Wired Magazine, 16/07:We can stop looking for (scientific) models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.

13. More examples … Duke biologists discovered an unusual link btwn the popular singer and a new species of fern, i.e. - bisexual reproductive stage of the ferns; - the team found the sequence GAGA when analyzing the fern’s DNA base pairs 13 13

14. Real Data Mining: Kepler’s Laws • How planets move among the stars? - Ptolemaic system (geocentric) - Copernican system (heliocentric) • Tycho Brahe (16 century) - measured positions of the planets in the sky - use experimental data to support one’s view (hypothesis) • Johannes Kepler - used volumes of Tycho’s data to discover three remarkably simple laws

15. Kepler’s Laws vs. ‘Lady Gaga’ knowledge Both search for assured knowledge Kepler’s Laws - well-defined hypothesis stated a priori - prediction capability - humanintelligence Lady Gaga knowledge - no hypothesis stated a priori - no prediction capability - computer intelligence (software program) - popular appeal (to widest audience) 15 15

16. Lessons from Natural Sciences Prediction capability Prediction is hard. Especially about the future. Empirical validation/repeatable events Limitations (of scientific knowledge) Important to ask the right question -Science starts from problems, and not from observations (K. Popper) -What we observe is not nature itself, but nature exposed to our method of questioning (W.Heisenberg) 16

17. Limitations of Scientific Method When the number of factors coming into play in a phenomenological complex is too large, scientific method in most cases fails us. We are going to be shifting the mix of our tools as we try to land the ship in a smooth way onto the aircraft carrier. Recall:the Ancient Greeks scorned ‘predictability’ 17

18. Important Differences Albert Einstein: It might appear that there are no methodological differences between astronomy and economics: scientists in both fields attempt to discover general laws for a group of phenomena. But in reality such differences do exist. The discovery of general laws in economics is difficult because observed economic phenomena are often affected by many factors that are very hard to evaluate separately. The experience which has accumulated during the civilized period of human history has been largely influenced by causes which are not economic in nature. 18

19. Flexible Data Modeling Approaches Late 1980’s Artificial Neural Networks Mid 1990’s Data Mining Late 1990’s Support Vector Machines Mid 2000’s Deep Learning (reincarnated NNs) Early 2010’s Big Data NOTE 1: no clear boundary btwn science vs marketing NOTE 2: fragmentation and ‘soft’ plagiarism 19 19

20. Methodologies for Data Modeling The field of Pattern Recognition is concerned with the automatic discovery of regularities in data. Data Mining is the process of automatically discovering useful information in large data repositories. This book (on Statistical Learning) is about learning from data. The field of Machine Learning is concerned with the question of how to construct computer programs that automatically improve with experience. Artificial Neural Networks perform useful computations through the process of learning.  (1) focus on algorithms/ computational procedures (2)all fields estimate useful models from data, i.e. extract knowledge from data (the same as in classical statistics) Real Issues: what is ‘useful’? What is ‘knowledge’? 20 20

21. What is ‘a good model’? All models are mental constructs that (hopefully) relate to real world Two goals of data-analytic modeling: - explanation (of past/ available data) -prediction (of future data) All good models make non-trivial predictions Good data-driven models can predict well, so the goal is to estimate predictive models aka generalization, inductive inference  Importance of methodology/assumptions 21 21

22. The Role of Statistics Dilemma: Mathematical or natural science? Traditionally, heavy emphasis on parametric modeling and math proofs Conservative attitude: slow acceptance of modern computational approaches Under-appreciation of predictive modeling William Edwards Deming: The only useful function of a statistician is to make predictions, and thus to provide a basis for action. 22 22

23. BROADER QUESTIONS Can we trust models derived from data? What is scientific knowledge? First-principle knowledge vs empirical knowledge vsbeliefs Understanding uncertainty and risk Historical view: how explosive growth of data-driven knowledge changes human perception of uncertainty 23 23

24. Scientific Understanding of Uncertainty Very recent: most probability theory and statistics developed in the past 100 years. Most apps in the last 50-60 years Dominant approach in classical science iscausal determinism, i.e. the goal is to estimate the true model (or cause) ~ system identification Classical statistics: the goal is to estimate probabilistic model underlying the data, i.e. system identification 24 24

25. Scientific Understanding (cont’d) Albert Einstein: The scientist is possessed by the sense of universal causality. The future, to him, is every whit as necessary and determined as the past. Albert Einstein: God does not play dice Stephen Hawking: God not only plays dice. He sometimes throws the dice where they cannot be seen 25 25

26. Modeling Complex Systems First-principle scientific knowledge: - deterministic - simple models (~ few main concepts) This knowledge has been used to design complex systems: computers, airplanes etc. It has not been successful for modeling and understanding complex systems: - weather prediction/ climate modeling - human brain - stock market etc. 26 26

27. Modeling Complex Systems A. Einstein: When the number of factors coming into play in a phenomenological complex is too large, scientific method in most cases fails us. One need only think of the weather, in which case prediction even for a few days is impossible… Occurences in this domain are beyond the reach of exact prediction because of the variety of factors in operation, not because of any lack of order in nature. 27 27

28. How to Model Complex Systems ? Conjecture 1 first-principle /system identification approach cannot be used Conjecture 2 system imitation approach, i.e. modeling certain aspects of a system, may be used  statistical models Examples: stock trading, medical diagnosis 28 28

29. Three Types of Knowledge Growing role of empirical knowledge Classical philosophy of science differentiates only between (first-principle) science and beliefs (demarcation problem) Importance of demarcation btwn empirical knowledge and beliefs in modern apps 29 29

30. Beliefs vs Scientific Theories Men have lower life expectancy than women Because they choose to do so Because they make more money (on average) and experience higher stress managing it Because they engage in risky activities Because ….. Demarcation problemin philosophy 30

31. Popper’s Demarcation Principle Karl Popper: Every true (inductive) theory prohibits certain events or occurences, i.e. it should be falsifiable • First-principle scientific theories vs. beliefs or metaphysical theories • Risky prediction, testability, falsifiability 31

32. Popper’s conditionsforscientific hypothesis Should be testable Should be falsifiable Example 1: Efficient Market Hypothesis(EMH) The prices of securities reflect all known information that impacts their value Example 2: We do not see our noses, because they all live on the Moon 32

33. Observations, Reality and Mind Philosophy is concerned with the relationship btwn - Reality (Nature) - Sensory Perceptions - Mental Constructs (interpretations of reality) Three Philosophical Schools REALISM: - objective physical reality perceived via senses - mental constructs reflect objective reality IDEALISM: - primary role belongs to ideas (mental constructs) - physical reality is a by-product of Mind INSTRUMENTALISM: - the goal of science is to produce useful theories Which one should be adopted(by scientists+ engineers)??

34. Three Philosophical Schools • Realism (materialism) • Idealism • Instrumentalism

35. Application Example:predicting gender of face images Training data: labeled face images Male etc. Female etc. 35 35

36. Predicting Gender of Face Images Input ~ 16x16 pixel image Model ~ indicator function f(x) separating 256-dimensional pixel space in two halves Model should predict well new images Difficult machine learning problem, but easy for human recognition

37. Two Philosophical Views (Vapnik, 2006) System Identification (~ Realism) - estimate probabilistic model (of true class densities) from available data - this view is adopted in classical statistics System Imitation (~ Instrumentalism) - need only to predict well i.e. imitate specific aspect of unknown system; - multiplicity of good models; - can they be interpreted and/or trusted? 37 37

38. OUTLINE Background Two Data-Analytic Methodologies - inductive inference step - two approaches to statistical inference - advantages of predictive approach - Example: market timing of mutual funds Basics of VC-theory Summary 38 38

39. Statistical vs Predictive Modeling KNOWLEDGE, ASSUMPTIONS EMPIRICAL DATA STATISTICAL INFERENCE PREDICTIVE APPROACH PROBABILISTIC MODELING 39

40. Inductive Inference Step Inductive inference step: Data  model ~ ‘uncertain inference’ Is it possible to make uncertain inferences mathematically rigorous? (Fisher 1935) Many types of ‘uncertain inferences’ - hypothesis testing - maximum likelihood - risk minimization ….  each comes with its own methodology/assumptions 40 40

41. Two Data-Analytic Methodologies Many existing data-analytic methods but lack of methodological assumptions Two theoretical developments - classical statistics ~ mid 20-th century - Vapnik-Chervonenkis (VC) theory ~ 1970’s Two related technological advances - applied statistics (R. Fisher) - machine learning, neural nets, data mining etc. 41 41

42. Statistical vs Predictive Approach • Binary Classification problem estimate decision boundary from training data Assuming class distributions P(x,y) were known: (x1,x2) space 42 42

43. Classical Statistical Approach: Realism (1) parametric formof unknown distribution P(x,y) is known (2) estimate parameters of P(x,y) from the training data (3) Construct decision boundary using estimated distribution and given misclassification costs Estimated boundary Modeling assumption: Unknown P(x,y) can be accurately estimated from available data 43 43

44. Critique of Statistical Approach (Leo Breiman) The Belief that a statistician can invent a reasonably good parametric class of models for a complex mechanism devised by nature Then parameters are estimated and conclusions are drawn But conclusions are about - the model’s mechanism -not about nature’s mechanism Many modern data-analytic sciences (economics, life sciences) have similar flaws 44 44

45. Predictive Approach: Instrumentalism (1) parametric form of decision boundaryf(x,w) is given (2) Explain available data via fitting f(x,w), or minimization of some loss function (i.e., squared error) (3) A function f(x,w*) providing smallest fitting error is then used for predictiion Estimated boundary Modeling assumptions - Need to specify f(x,w) and loss function a priori. - No need to estimate P(x,y) 45 45

46. Classification with High-Dimensional Data • Digit recognition 5 vs 8: each example ~ 16 x 16 pixel image  256-dimensional vector x Medical Interpretation • Each pixel ~ genetic marker • Each patient (sample) described by 256 genetic markers • Two classes ~ presence/ absence of a disease • Estimation of P(x,y) with finite data is not possible • Accurate estimation of decision boundary in 256-dim. space is possible, using just a few hundred samples 46 46

47. Common Modeling Assumptions Future is similar to Past - training and test data from the same distribution - i.i.d. training data - large test set Prediction accuracy ~ given loss function - misclassification costs (classification problems) - squared loss (regression problems) - etc. Proper formalization~type of learning problem e.g., classification is used in many applications 47 47

48. Importance of Complexity Control Regression estimation for known parameterization Ten training samples Fitting linear and 2-nd order polynomial:

49. Statistical vs Predictive: issues Predictive approach - estimates certain properties of unknown P(x,y) that are useful for predicting y - has solid theoretical foundations (VC-theory) - successfully used in many apps BUT its methodology + concepts are different from classical statistics: - formalization of the learning problem (~ requires understanding of application domain) - a priori specification of a loss function - interpretation of predictive models may be hard - multiplicity of models estimated from the same data 49

50. Predictive Methodology (VC-theory) Method of questioning is - the learning problem setting(inductive step) - driven by application requirements Standard inductive learning commonly used (may not be the best choice) Good generalization depends on two factors - (small) training error - small VC-dimension ~ large ‘falsifiability’ 50 50