Data-driven methods in Environmental Sciences Exploration of Artificial Intelligence Techniques

1. Data-driven methods in Environmental SciencesExploration of Artificial Intelligence Techniques Valliappa.Lakshmanan@noaa.gov lakshman@ou.edu

2. Data Driven Methods What is Artificial Intelligence? Common AI techniques Choosing between AI techniques Pre and post processing lakshman@ou.edu

3. What is AI? • Machines that perceive, understand and react to their environment • Goal of Babbage, etc. • Oldest endeavor in computer science • Machines that think • Robots: factory floors, home vacuums • Still quite impractical lakshman@ou.edu

4. AI vs. humans • AI applications built on Aristotlean logic • Induction, semantic queries, system of logic • Human reasoning involves more than just induction • Computers never as good as humans • In reasoning and making sense of data • In obtaining a holistic view of a system • Computers much better than humans • In processing reams of data • Performing complex calculations lakshman@ou.edu

5. Successful AI applications • Targeted tasks more amenable to automated methods • Build special-purpose AI systems • Determine appropriate dosage for a drug • Classify cells as benign or cancerous • Called “expert systems” • Methodology based on expert reasoning • Quick and objective ways to obtain answers lakshman@ou.edu

6. Data Driven Methods What is Artificial Intelligence? Common AI techniques Choosing between AI techniques Pre and post processing lakshman@ou.edu

7. Fuzzy logic • Fuzzy logic addresses key problem in expert systems • How to represent domain knowledge • Humans use imprecisely calibrated terms • How to build decision trees on imprecise thresholds lakshman@ou.edu

8. Fuzzy logic example Source: Matlab fuzzy logic toolbox tutorial http://www.mathworks.com/access/helpdesk/help/toolbox/fuzzy/fp350.html lakshman@ou.edu

9. Advantages of fuzzy logic • Considerable skill for little investment • Fuzzy logic systems piggy bank on human analysis • Humans encode rules after intelligent analysis of lots of data • Verbal rules generated by humans are robust • Simple to create • Not much need for data or ground truth • Logic tends to be easy to program • Fuzzy rules are human understandable lakshman@ou.edu

10. Where not to use fuzzy logic • Do not use fuzzy logic if: • Humans do not understand the system • Different experts disagree • Knowledge can not be expressed with verbal rules • Gut instinct is involved • Not just objective analysis • A fuzzy logic system is limited • Piece-wise linear approximation to a system • Non-linear systems can not be approximated • Many environment applications are non-linear lakshman@ou.edu

11. Neural Networks • Neural networks can approximate non-linear systems • Evidence-based • Weights chosen through optimization procedure on known dataset (“training”) • Works even if experts can’t verbalize their reasoning, or if there is ground truth lakshman@ou.edu

12. A example neural network Diagram from: http://www.codeproject.com/useritems/GA_ANN_XOR.asp lakshman@ou.edu

13. Advantages of neural networks • Can approximate any smooth function • The three-layer neural network • Can yield true probabilities • If output node is a sigmoid node • Not hard to train • Training process is well understood • Fast in operations • Training is slow, but once trained, the network can calculate the output for a set of inputs quite fast • Easy to implement • Just a sum of exponential functions lakshman@ou.edu

14. Disadvantages of neural networks • A black box • The final set of weights yields no insights • Magnitude of weights doesn’t mean much • Measure of skill needs to be differentiable • RMS error, etc. • Can not use Probability of Detection, for example • Training set has to be complete • Unpredictable output on data unlike training • Need lots of data • Need expert willing to do lot of truthing lakshman@ou.edu

15. Recap: • Fuzzy logic • Humans provide the rules • Not optimal • Neural network • Humans can not understand system • Optimal • Middle ground? • Genetic Algorithms • Decision Trees lakshman@ou.edu

16. Genetic algorithms • In genetic algorithms • One fixes the model (rule base, equations, class of functions, etc.) • Optimize the parameters to model on training data set • Use optimal set of parameters for unknown cases lakshman@ou.edu

17. An example genetic algorithm Sources: http://tx.technion.ac.il/~edassau/web/genetic_algorithms.htm http://cswww.essex.ac.uk/research/NEC/ lakshman@ou.edu

18. Advantages of genetic algorithms • Near-optimal parameters for given model • Human-understandable rules • Best parameters for them • Cost function need not be differentiable • The process of training uses natural selection, not gradient descent • Requires less data than a neural network • Search space is more limited lakshman@ou.edu

19. Disadvantages of genetic algorithms • Highly dependent on class of functions • If poor model is chosen, poor results • Optimization may not help at all • Known model does not always lead to better understanding • Magnitude of weights, etc. may not be meaningful if inputs are correlated • Problem may have multiple parametric solutions lakshman@ou.edu

20. Decision trees • Can automatically build decision trees from known data • Prune trees • Select thresholds • Choose operators • Disadvantages • Piece-wise linear, so typically less skilled than neural networks • Large decision trees are effectively a blackbox • Can not do regression, only classification • Advantages: • Fast to train • New advances: bagged, boosted decision trees approach skill of neural networks, but are no longer fast to train Root 30 50 T < 10C 20 15 T > 10C 10 35 Z > 45 18 2 Z < 45 2 13 V < 5 8 2 V > 5 2 33 lakshman@ou.edu

21. Radial Basis Functions Diagram from: A. W. Jayawardena & D. Achela K. Fernando 1998: Use of Radial Basis Function Type Artificial Neural Networks for Runoff Simulation, Computer-Aided Civil and Infrastructure Engineering 13:2 • Radial Basis Functions are a form of neural network • Localized gaussians • Linear sum of non-linear functions • Advantage: Can be solved by inverting a matrix, so very fast • Disadvantage: Not a general-enough model lakshman@ou.edu

22. Data Driven Methods What is Artificial Intelligence? Common AI techniques Choosing between AI techniques Pre and post processing lakshman@ou.edu

23. Typical data-driven application Input Data Which features? How do we find f() Features f(features) AI application in run-time Result lakshman@ou.edu

24. What is the role of the data? • Validation • Test known model • Technique: • Difference between model output and ground truth helps to validate the model • Calibration • Find parameters to model with desired structure • Technique: • Tuned fuzzy logic method • Genetic algorithms • Induction • Find model and parameters from just data • Technique: • Neural network methods, bagged/boosted decision trees, support vector machines, etc. lakshman@ou.edu

25. What is the problem to solve? • Do you have a bunch of data and want to: • Estimate an unknown parameter from it? • True rainfall based on radar observations? • Amount of liquid content from in-situ measurements of temperature, pressure, etc? • Regression • Classify what the data correspond to? • A water surge? • A temperature inversion? • A boundary? • Classification • Regression and classification aren’t that different • Classification: estimate probability of an event • A function from 0-1 lakshman@ou.edu

26. Which AI technique? • Do you have expert knowledge? • Humans have a “model” in their head? Should the final f() be understandable? • Create fuzzy logic rules from experts’ reasoning • Aggregate the individual fuzzy logic rules • Can tune the fuzzy rules based on data • Using regression, decision trees or neural networks for RMS error criterion • Genetic algorithms for error criteria like ROC, economic cost, etc. • Many times the original rules are just fine • Do you already know the model? • A power-law relationship? Gaussian? Quadratic? Rules? • Just need to find parameters to this model? • If linear, just use linear regression • If non-linear: use genetic algorithms • Use continuous GAs • Both of these can be used for regression (therefore, also classification) lakshman@ou.edu

27. Which AI technique (contd.) • Do you know nothing about the data? • Not the suspected equation/model (GA)? • Not the suspected rules (fuzzy logic)? • Use a AI technique that supplies its equations/rules • “black box”. • For classification, use: • Bagged decision trees or Support Vector Machines • If output is probabilistic, remember to apply Platt scaling • Summary statistics on bagged DTs can help answer “why” • Neural Networks • For regression, use: • Neural networks lakshman@ou.edu

28. Where do your data come from? • Observed data • Compute features • Choose AI technique • The 4 choices in the previous two slides • Simulated data: • Example: trying to replicate a very complex model • Throw randomly-generated data at model • Compute features • Choose AI technique: • GA for parametric approximations • NN when you don’t know how to approximate lakshman@ou.edu

29. Where do you get your inputs? • What type of data do you have? • Individual observations? • Sample them (choose at random) and use directly • Sparse observations in a time series? • Generate time-based features (1D moving windows) • Signal processing features from time series • Data from remotely sensed 2D grids? • Generate image-based features using convolution filters • Do you need: • Pixel-based regression/classification? • Use convolution features directly • Object-based regression/classification? • Identify regions using region growing • Use region-aggregate features lakshman@ou.edu

30. Typical data-driven application Observed data Signal/image processing;sampling Features normalize/create chromosome/ determine confidences f() FzLogic/GenAlg/NN/DecTree Platt method/region-average/threshold A data-driven application in run-time Result lakshman@ou.edu

31. Data Driven Methods What is Artificial Intelligence? Common AI techniques Choosing between AI techniques Pre and post processing lakshman@ou.edu

32. Preprocessing • Often can not use pixel data directly • Too much data, too highly correlated • May need to segment pixels into objects and use features computed on the objects • Different data sets may not be collocated • Need to interpolate to line them up • Mapping, objective analysis • Noise in data may need to be reduced • Smoothing • Present statistic of data, rather than data itself • Features need to be extracted from data • Human experts often good source of ideas on signatures to extract from data lakshman@ou.edu

33. Postprocessing • The output of an expert system may be grid point by grid point • May need to provide output on objects • Storms, forests, etc. • Can average outputs over objects’ pixels • May need probabilistic output • Scale output of maximum marginal techniques • Use a sigmoid function • Called Platt scaling lakshman@ou.edu

34. Summary • What is Artificial Intelligence? • Data-driven methods to perform specific targeted tasks • Common AI techniques • Fuzzy logic, neural networks, genetic algorithms, decision trees • Choosing between AI techniques • Understand the role of your data • Do experts understand the system? (have a model) • Do experts expect to understand the system? (readability) • Pre and post processing • Image processing techniques on spatial grids lakshman@ou.edu