1 / 50

From Big Data to Little Knowledge

From Big Data to Little Knowledge. Vladimir Cherkassky University of Minnesota cherk001@umn.edu Presented at CodeFreeze , Jan 16, 2014. Electrical and Computer Engineering. 1. 1. Motivation : What is Big Data?. Traditional IT infrastructure Data storage, access, connectivity etc.

norina
Download Presentation

From Big Data to Little Knowledge

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. From Big Data to Little Knowledge Vladimir Cherkassky University of Minnesota cherk001@umn.edu Presented at CodeFreeze, Jan 16, 2014 Electrical and Computer Engineering 1 1

  2. Motivation: What is Big Data? Traditional IT infrastructure Data storage, access, connectivity etc. Making sense / acting on this data Data  Knowledge  Decision making always predictive by nature Objectives of my talk - Hype vs. Reality - Methodological aspects of data-analytic knowledge discovery 2 2

  3. Scientific Discovery Combines ideas/models and facts/data First-principle knowledge: hypothesis  experiment  theory ~ deterministic, causal, intelligible models Modern data-driven discovery: s/w program + DATA  knowledge ~ statistical, complex systems Two different philosophies 3 3

  4. History of Scientific Knowledge Ancient Greece: Logic+deductive_reasoning Middle Ages:Deductive (scholacticism) Renaissance, Enlightment: (1) First-Principles (Laws of Nature) (2) Experimental science (empirical data) Combining (1) + (2)  problem of induction Digital Age: the problem of induction attains practical importance in many fields 4 4

  5. Induction and Predictive Learning Induction: aka inductive step, generalization etc. Deduction: aka Prediction

  6. Problem of Induction in Philosophy Francis Bacon: advocated empirical knowledge (inductive) vs scholastic David Hume: What right do we have to assume that the future will be like the past? Philosophy of Science tries to resolve this dilemma/contradiction between deterministic logic and uncertain nature of empirical data. Digital Age: growth of empirical data, and this dilemma becomes important in practice. 6 6

  7. Cultural and Psychological Aspects All men by nature desire knowledge Man has an intense desire for assured knowledge Assured Knowledge ~ belief in - religion (much of human history) - reason (causal determinism) - science / pseudoscience - data-analytic models (~ Big Data) - genetic risk factors …

  8. Gods, Prophets and Shamans 8 8

  9. Uncertainty and Risk in Science Math, Logic and Science are about certainty ~ deterministic rules Probability and empirical data: involves uncertainty ~ inferior knowledge Causal Determinism dominates modern science True Scientific knowledge consists of deterministic Laws of Nature There is a single (true, causal) model that explains natural phenomenon 9 9

  10. Knowledge Discovery in Digital Age Most information in the form of digital data Can we get assured knowledge from data? Big Data ~ technological nirvana data + connectivity  more knowledge Wired Magazine, 16/07:We can stop looking for (scientific) models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.

  11. REALITY Many studies have questionable value - statistical correlation vs causation Some border nonsense - US scientists at SUNY discovered Adultery Gene !!! (based on a sample of 181 volunteers interviewed about their sex life) Economic forecasting, i.e. ‘predicting’ -unemployment rate, monthly job gain/loss... 11 11

  12. More examples … Duke biologists discovered an unusual link btwn the popular singer and a new species of fern, i.e. - bisexual reproductive stage of the ferns; - the team found the sequence GAGA when analyzing the fern’s DNA base pairs 12 12

  13. Real Data Mining: Kepler’s Laws • How planets move among the stars? - Ptolemaic system (geocentric) - Copernican system (heliocentric) • Tycho Brahe (16 century) - measured positions of the planets in the sky - use experimental data to support one’s view (hypothesis) • Johannes Kepler - used volumes of Tycho’s data to discover three remarkably simple laws

  14. Kepler’s Laws (1) The orbit is an ellipse with sun at its focus (2) The line joining a planet to the sun sweeps equal areas during the same time (3) The ratio P2/D3 is constant, where P is the orbit period and D is the orbit size. NO computers, statistics, machine learning or Big Data

  15. Kepler’s Laws vs. ‘Lady Gaga’ knowledge Both search for assured knowledge Kepler’s Laws - well-defined hypothesis stated a priori - prediction capability - humanintelligence Lady Gaga knowledge - no hypothesis stated a priori - no prediction capability - computer intelligence (software program) - popular appeal (to widest audience) 15 15

  16. Lessons from Natural Sciences Prediction capability Prediction is hard. Especially about the future. Empirical validation/repeatable events Limitations (of scientific knowledge) Important to ask the right question -Science starts from problems, and not from observations (K. Popper) -What we observe is not nature itself, but nature exposed to our method of questioning (W.Heisenberg) 16

  17. Limitations of Scientific Method When the number of factors coming into play in a phenomenological complex is too large, scientific method in most cases fails us. We are going to be shifting the mix of our tools as we try to land the ship in a smooth way onto the aircraft carrier. Recall:the Ancient Greeks scorned ‘predictability’ 17

  18. Important Differences Albert Einstein: It might appear that there are no methodological differences between astronomy and economics: scientists in both fields attempt to discover general laws for a group of phenomena. But in reality such differences do exist. The discovery of general laws in economics is difficult because observed economic phenomena are often affected by many factors that are very hard to evaluate separately. The experience which has accumulated during the civilized period of human history has been largely influenced by causes which are not economic in nature. 18

  19. Prediction in Social Systems The Bitcoin saga Illusion of predictability: 19 19

  20. Methodological Aspects ofData-Driven Knowledge Discovery Empirical Knowledge vs. First-Principles Method of Questioning: - Two Data-Analytic Methodologies - Statistical Modeling Assumptions Example: Market Timing of Mutual Funds Interpretation of Predictive Models 20 20

  21. Three Types of Knowledge Growing importance of empirical knowledge Demarcation problems: - first-principles vsempiricalvsbeliefs Assured knowledge ~ interpretable - first-principle ~ small number of concepts - empirical knowledge ??? 21 21

  22. Empirical Knowledge These methodological/philosophical issues need to be properly addressed • Can it be obtained from data alone? • How is it different from ‘beliefs’ ? • Role of a priori knowledge vs. data ? • What is ‘the method of questioning’ ? 22

  23. Induction and Predictive Learning Induction: aka inductive step, generalization etc. Deduction: aka Prediction

  24. Inductive Inference Step Inductive inference step: Data  model ~ ‘uncertain inference’ Is it possible to make uncertain inferences mathematically rigorous? (Fisher 1935) Many types of ‘uncertain inferences’ - hypothesis testing - maximum likelihood - risk minimization ….  each comes with its own methodology/assumptions 24 24

  25. Two Data-Analytic Methodologies Many existing data-analytic methods but lack of methodological assumptions Two theoretical developments - classical statistics ~ mid 20-th century - Vapnik-Chervonenkis (VC) theory ~ 1970’s Two related technological advances - applied statistics (R. Fisher) - machine learning, neural nets, data mining etc. 25 25

  26. Binary Classification Problem Given: training data (x,y) ~ i.i.d. samples from unknown distribution P(x,y) Estimate: a model or function f(x) that: - explains this data - can predict future data Classification problem:  Learning ~ function estimation

  27. Classical Statistics Goal of data modeling /Asking the right question - estimate unknown distribution P(x,y) Classical statistics approach (R. Fisher) - specify a parametric model for P(x,y) - estimate its parameters from training data Observed_Data ~ Model + Noise more data  better (more accurate) model Assumed parametric form of P(x,y) is based on first-principle knowledge, so it is true.

  28. Critique of Statistical Approach (Leo Breiman) The Belief that a statistician can invent a reasonably good parametric class of models for a complex mechanism devised by nature Then parameters are estimated and conclusions are drawn But conclusions are about - the model’s mechanism -not about nature’s mechanism Many modern data-analytic sciences (economics, life sciences) have similar flaws 28 28

  29. Risk Minimization Approach Goal of data modeling /Asking the right question ~estimate a model that will predict well Predictive Approach: estimate only properties of P(x, y) that are useful for predicting y Note: no need to estimate P(x, y) Requires specification of: - a set of possible models f(x,w) - loss function to measure prediction performance - proper formalization of the learning problem 29 29

  30. Standard Modeling Assumptions Future is similar to Past - training and test data from the same distribution - i.i.d. training data - large test set Prediction accuracy ~ given loss function - misclassification costs (classification problems) - squared loss (regression problems) - etc. Proper formalization (for an application) e.g., classification is used in many applications 30 30

  31. Predictive Methodology (VC-theory) Method of questioning is - the learning problem setting(inductive step) - driven by application requirements Standard inductive learning commonly used (may not be the best choice) Good generalization depends on two factors - (small) training error - small VC-dimension ~ large ‘falsifiability’ 31 31

  32. Timing of International Funds International mutual funds - priced at 4 pm EST (New York time) - reflect price of foreign securities traded at European/ Asian markets - Foreign markets close earlier than US market Possibility of inefficient pricing. Market timing exploits this inefficiency. Scandals in the mutual fund industry ~2002 Solution adopted: restrictions on trading 32 32

  33. Binary Classification Setting TWIEX ~ American Century Int’l Growth Input indicators (for trading) ~ today - SP 500 index (daily % change) ~ x1 - Euro-to-dollar exchange rate (% change) ~ x2 Output : TWIEX NAV (% change)~y next day Trading rule: D(x) = 0~Sell, D(x)=1 ~ Buy Model parameterization (fixed): - linear - quadratic Decision rule (estimated from training data):  Buy /Sell decision (+1 / 0) 33 33

  34. Methodological Assumptions When a trained model can predict well? (1) Future/test data is similar to training data i.e., use 2004 period for training, and 2005 for testing (2) Estimated model is ‘simple’ and provides good performance during training period i.e.,the trading strategy is consistently better than buy-and-hold during training period Loss function (to measure performance): where 34 34

  35. Empirical Results: 2004 -2005 data Linear model Training data 2004 Training period 2004  can expect good performance with test data 35 35

  36. Empirical Results: 2004 -2005 data Linear model Test data 2005 Test period 2005 confirmed good prediction performance 36 36

  37. Empirical Results: 2004 -2005 data Quadratic model Training data 2004 Training period 2004  can expect good performance with test data 37 37

  38. Empirical Results: 2004 -2005 data Quadratic model Test data 2005 Test period 2005 confirmed good test performance 38 38

  39. Interpretation vs Prediction Two good trading strategies estimated from 2004 training data Both models predict well for test period 2005 Which one is ‘true’? 39 39

  40. DISCUSSION Can this trading strategy be used now ? - NO, this market timing strategy becomes ineffective since ~ year 2008. The reason is changing statistical characteristics of the market - YES, it can be used occasionally. Hypocrisy of the mutual fund industry Story 1: markets are very efficient, so individual investors cannot trade successfully and outperform the market indices (such as SP500) Story 2: market timing is harmful for mutual funds, so such abusive trading activity should be banned Story 3: restrictions also apply to domestic funds 40 40

  41. Interpretation of Predictive Models Humans cannot provide interpretationeven if they can make good prediction Each input ~ 28 x 28 pixel image  784-dimensional input x Interpretation of black-box models Not unique/ subjective Depends on chosen parameterization (method) 41 41

  42. Classification with High-Dimensional Data • Digit recognition 5 vs 8: each example ~ 28 x 28 pixel image  784-dimensional vector x Medical Interpretation • Each pixel ~ genetic marker • Each patient (sample) described by 784 genetic markers • Two classes ~ presence/ absence of a disease • Estimation of P(x,y) with finite data is not possible • Accurate estimation of decision boundary in 784-dim. space is possible from just a few hundred samples, i.e. using Support Vector Machine (SVM) classifiers 42 42

  43. Interpretation of SVM models How to interpret high-dimensional models? (say, SVM model) Strategy 1: dimensionality reduction/feature selection  prediction accuracy usually suffers Strategy 2: approximate SVM model via a set of rules (using rule induction, decision tree etc.)  does not scale well for high-dim. models 43 43

  44. Dimensionality Reduction Reduce dimensionality (small # features) (a) 10 top ranked pixels using Fisher’s criterion (b) extract 3 principal components (via PCA) (2) Estimate RBF SVM model  Generalization performance degrades: 44 44

  45. Rule Induction (via ALBA method) Estimate SVM model using all 784 pixels (2) Interpret this SVM model via ALBA method (Active Learning Based Algorithm: Martens et al 2009)  Generalization performance degrades: 45 45

  46. SUMMARY (A) Predictive Data-Analytic Modeling: usually on the boundary btwntrivial and impossible Asking the right question ~ problem setting - depends on modeler’s creativity/ intelligence - requires application domain knowledge - cannot be formalized Modeling Assumptions (not just algorithm) Interpretation of black-box models - very difficult (requires domain knowledge) - multiplicity of ‘good’ models 46 46

  47. SUMMARY (B) Common misconception: data-driven models are intrinsically objective Explanation bias (favors simplicity+causality) psychological + cultural reasons Cognitive bias (favor only positive findings) When all these human biases are incorporated into data-analytic modeling: - many ‘interesting’ discoveries - little objective value - no real predictive value 47 47

  48. SUMMARY (C) Predictive learning methodology is useful for safeguarding against these biases It clearly differentiates between (1) The learning problem setting (~ creation of human mind, intelligent speculation - cannot be logically justified or derived from data) (2) The learning algorithm /software ~ particular implementation of (1) (3) Predictive data-analytic model ~ estimated from data; provides the only objective evaluationof the original intelligent speculation (1). Model (3) makes sense only in the context of (1). 48 48

  49. V. Cherkassky Predictive Learning, 2013 www.VCtextbook.com Available on Amazon.com • This book presents a very good introduction to machine learning forundergraduate students and practitioners. It differs from other textbooks in its original coverage of the philosophical aspects of inference and their relationship to machine learning theory. This will allow readers to develop a better understanding of generalization problems and learning algorithms. • - V. Vapnik, Columbia University

  50. References • V. Vapnik, Estimation of Dependencies Based on Empirical Data. Empirical Inference Science: Afterword of 2006 Springer • L. Breiman, Statistical Modeling: the Two Cultures, Statistical Science, vol. 16(3), pp. 199-231, 2001 • A. Einstein, Ideas and Opinions, Bonanza Books, NY 1954 • V. Cherkassky and F. Mulier, Learning from Data, second edition, Wiley, 2007 • V. Cherkassky and S. Dhar, Market timing of international mutual funds: a decade after the scandal, Proc. CIFEr 2012

More Related