Spatiotemporal Stream Mining using TRACDS Middle East Technical University October 31, 2012

1. Spatiotemporal Stream Mining using TRACDS Middle East Technical University October 31, 2012 Margaret H Dunham, Michael Hahsler, Yu Su, SudheerChelluboina, and Hadil Shaiba Computer Science and Engineering This work is supported by NSFIIS-0948893

2. IDA@SMUIntelligent Data Analysis Lab Team led by Margaret H. Dunham Michael Hahsler MissionAt IDA@SMU we create novel techniques inspired by knowledge discovery, data mining, machine learning, artificial intelligence and statistical analysis to work with data from various sources. Current Focus • Massive data stream modeling: TRACDSTM • Hurricane intensity prediction • Effective metagenomic classification for theHuman Genome Project • Recommender systems: R/Apache Mahout http://www.lyle.smu.edu/IDA

3. Outline • Spatiotemporal Stream Data • TRACDS • Hurricane Intensity Prediction • PIIH • PIIH online

4. From Sensors to Streams • Data captured and sent by a set of sensors is usually referred to as “stream data”. • Real-time sequence of encoded signals which contain desired information. It is continuous, ordered (implicitly by arrival time or explicitly by timestamp or by geographic coordinates) sequence of items • May be viewed as arriving in discrete time intervals. • Stream data is infinite - the data keeps coming. • Examples: Weather data, network data (VoIP), traffic data.

5. Stream Data Format • Events arriving in a stream • At any time, t, we can view the state of the problem as represented by a vector of n numeric values: Vt= <S1t, S2t, ..., Snt> Time

6. Modeling Stream Data • Summarization (Synopsis) of data • Temporal and Spatial • Dynamic • Continuous (infinite stream) • Concept Drift • Learn • Forget • Sublinear growth rate - Clustering

7. MM A first order Markov Chain is a finite or countably infinite sequence of events {E1, E2, … } over discrete time points, where Pij = P(Ej | Ei), and at any time the future behavior of the process is based solely on the current state A Markov Model (MM) is a graph with m vertices or states, S, and directed arcs, A, such that: • S ={N1,N2, …, Nm}, and • A = {Lij | i 1, 2, …, m, j 1, 2, …, m} and Each arc, Lij= <Ni,Nj> is labeled with a transition probability Pij = P(Nj | Ni).

8. Problem with Markov Chains • The required structure of the MC may not be certain at the model construction time. • As the real world being modeled by the MC changes, so should the structure of the MC. • Not scalable – grows linearly as number of events. • Our solution: • Extensible Markov Model (EMM) • Cluster real world events • Allow Markov chain to grow and shrink dynamically

9. EMM (Extensible Markov Model) • Time Varying Discrete First Order Markov Model • Continuously evolves • Nodes are clusters of real world states. • Learning continues during applicationphase. • Learning: • Transition probabilities between nodes • Node labels (centroid of cluster) • Nodes are added and removed as data arrives • Applications: • Anomaly/Rare Event Detection • Prediction • Classification

10. EMM Definition Extensible Markov Model (EMM): at any time t, EMM consists of an MC with designated current node and algorithms to modify it, where algorithms include: • EMMCluster, which defines a technique for matching between input data at time t + 1 and existing states in the MC at time t. • EMMIncrement algorithm, which updates MC at time t + 1 given the MC at time t and clustering measure result at time t + 1. • EMMDecrementalgorithm,which removes nodes from the EMM when needed.

11. EMM Cluster • Nearest Neighbor (or any clustering technique) • If none “close” create new node • Labeling of cluster is centroid of members in cluster or Clustering Feature • O(n) Here n is the number of states

12. EMM Sublinear Growth Servent Data

13. Growth Rate Automobile Traffic Minnesota Traffic Data

14. 2/3 1/2 N3 2/3 N1 2/3 1/2 N3 1/3 1/1 N2 N1 N1 1/2 2/3 1/3 1/1 N2 1/3 N2 N1 1/3 N2 N3 1/1 1 N1 1/1 2/2 1/1 N1 EMM Learning <18,10,3,3,1,0,0> <17,10,2,3,1,0,0> <16,9,2,3,1,0,0> <14,8,2,3,1,0,0> <14,8,2,3,0,0,0> <18,10,3,3,1,1,0.>

15. 1/3 1/3 1/3 1/6 1/6 N1 N1 N3 N3 1/3 2/2 1/3 1/6 N2 1/3 1/2 N6 N6 N5 N5 EMM Forgetting

16. Outline • Spatiotemporal Stream Data • TRACDS • Hurricane Intensity Prediction • PIIH • PIIH online

17. Traditional Stream Clustering Standard Data Stream Clustering ignores temporal aspect of data

18. Stream Clustering • Clusters change over time – they move • Some techniques use micro clusters/reclustering • Reclustering is often off line (batch while stream data comes). • STREAM • Partitions stream data into segments • Clusters each segment (k-medians) • Iteratively reclusters the centers of these clusters S. Guha, A. Meyerson, N. Mishra, R. Motwani, and L. O'Callaghan. “Clustering data streams: Theory and practice.” IEEE Transactions on Knowledge and Data Engineering, 15(3):515-528, 2003.

19. Temporal Relationship Among Clusters in Data Streams

20. TRACDS NOTE • TRACDS is not: • Another stream clustering algorithm • TRACDS is: • A new way of looking at clustering • Built on top of an existing clustering algorithm • TRACDS may be used with any stream clustering algorithm

21. TRAC-DS Overview

22. TRACDS Clustering Operations

23. TRACDS Example C EMM http://www.lyle.smu.edu/IDA/TRACDS

24. Outline • Spatiotemporal Stream Data • TRACDS • Hurricane Intensity Prediction • PIIH • PIIH online

25. Lower 9th Ward of New Orleans, Louisiana, Feb 27, 2006Photographer: Mackenzie Schott

26. Hurricanes Hurricanes are tropical cyclones with sustained winds of at least 64 kt (119 km/h, 74 mph) . The major issues in forecasting hurricanes are predicting their tracks of movement and their intensities. Compared with prediction of track movement, intensity prediction is still relatively inaccurate. Time step [0h, 12h, 24h, …, 120h]

27. Hurricane Intensity Prediction • Hurricane Intensity: • Maximum sustained surface wind. • Highest average wind speed within 1 minute and10m above surface. • Rapid Intensification • 24-h increase in maximum wind speed >= 30knots. “Maximum Sustained Wind”. Wikipedia.Wikimedia foundation,27 August 2011. Web. 4 December 2011. Retrieved from http://en.wikipedia.org/wiki/Maximum_sustained_wind. “Rapid Intensification,” accessed on 10/24/12, http://www.hurrnet.com/tutorial/forecasts/intensity/rapid.htm .

28. Predicting Intensity • Statistical models predict intensity based on measured stream data. • Current state of storm • History of this storm • How similar storms behaved in past • Regression models are the most popular. • NOAA (branch of U.S. Government) • collects stream data. • Yearly updates it models based on data from previous year • Makes predictions in a quasi-real time manner.

29. Hurricane Intensity Prediction • Category 5 - 175 mph • Damage: estimated $125 billion • Fatalities: >1,800 • “Hurricane Katrina – Most Destructive Hurricane Ever to Strike the U.S.”, August 28, 2005, February 12, 2007, http://www.katrina.noaa.gov/ . “Objective: Improve forecast skill to accuracy and confidence levels required for decision‐making and risk management” NOAA’s National Weather Service Strategic Plan 2010-2020 • Very difficult to predict Intensity (rapid intensification) • National Hurricane Center (NHC) uses • Dynamical models: computational intensive and slow • Statistical models: Statistical Hurricane Intensity Prediction Scheme (SHIPS) • Current Storm – SANDY http://www.nhc.noaa.gov/archive/2012/SANDY_graphics.shtml Path of Hurricane Katrina (2005)Color shows intensity 10/31/2012, METU

30. Remote Sensing • Storm features are gathered from the earth's observations using remote sensing. • Real time data are gathered every few hours and stored in largedatabases. • Historical data of more than 20 years of the earth's behavior is stored in the database. • Methods: • Satellite • Buoy • Ship • Aircraft

31. Outline • Spatiotemporal Stream Data • TRACDS • Hurricane Intensity Prediction • PIIH • PIIH online

32. Hurricane Data The data contains 16 predictors. The dataset is formed by time ordered 12 hour interval records and contains the hurricane data from seasons 1982 to 2003. 1982 16 predictors hurricane 1 0h, 12h, 24h, … 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 25,0,1,-5.83,668,0,140,14.9,-53.5,13.25,40.5,23,6.6,27,372.5,19600 25,0,1,-5.83,708,0,140,12.7,-53.45,13.65,37.5,17.5,5.69,4,317.5,19600 30,5,1,-3.58,682,150,135,12.75,-53.35,13.25,34,1.5,5.79,15,382.5,18225 35,5,1,-4.9,674,175,130,14.2,-53.35,13.4,33,-12,6.66,-13,497,16900 50,15,1,0.44,681,750,113.52,17.1,-53.15,13.2,35,-20,8.32,-7,855,12885.79 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 30,0,0.99,-7.02,656,0,124.55,19.05,-52.55,14.75,51,0.5,6.68,45,571.5,15512.49 30,0,0.98,-7.02,675,0,123.75,17.3,-52.6,14.15,54,5,6.63,22,519,15314.28 35,5,0.98,-4.16,722,175,119.55,17.9,-52.6,14.65,58,10,7.43,34,626.5,14292 65,30,0.97,4.09,635,1950,88.77,19.15,-52.1,14.7,54.5,27.5,8.63,33,1244.75,7879.26 75,10,0.97,6.25,724,750,70.08,17.8,-52.15,12.55,54,48.5,8.61,45,1335,4910.92 95,20,0.96,9.17,641,1900,37.59,14.85,-52.9,11.1,56.5,55,7.87,15,1410.75,1413.13 95,0,0.96,7.2,691,0,33.33,15.6,-53.45,9.25,51.5,44.5,8.97,32,1482,1110.98 95,0,0.95,0.82,713,0,35.62,17.9,-53.25,7.85,47,38,10.72,31,1700.5,1268.43 95,0,0.95,2.4,813,0,28.12,20.85,-52.65,7.25,45,45,12.84,63,1980.75,790.65 115,20,0.93,10.65,635,2300,-11.1,24.45,-52.7,4.55,41.5,57.5,15.81,24,2811.75,123.2 110,-5,0.93,14.51,622,-550,-26.24,30.7,-53.55,1.15,40.5,50.5,21.2,28,3377,688.71 90,-20,0.91,18.15,613,-1800,-17.97,37.05,-53.95,0,46,29.5,27.08,42,3334.5,322.99 70,-20,0.91,21.86,668,-1400,1.01,40.3,-53.7,0,52.5,20,30.72,41,2821,1.02 70,0,0.89,26.22,688,0,2.35,45.05,-52.7,0.25,50.5,37.5,35.18,31,3153.5,5.5 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 …… 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 Hurricane Data hurricane 2 0h, 12h, 24h, … … … hurricane 274 0h, 12h, 24h, … 2003 Intensity

33. Construct EMM

34. Use EMM for Prediction

35. EMM, TRACDS and Hurricane Data • Approach: Using TRACDS algorithms, construct multiple EMMs. One will be built for each time point into the future for which predictions are to be made: 12 hours, …, 120 hours. • NOAA provides 16 different features or predictors (attribute values). • Clustering isperformed based on a distance calculation from input feature vector to centroid of clusters in EMMs. • However the importance of these to intensity prediction is not uniform. • How can we determine weight for each feature? Used during clustering.

36. Weighted Feature Learning -Extensible Markov Model (WFL-EMM) WFL-EMM assumes that the different predictors contribute differently during the prediction. 1 Weights for predictors 0 V1 = <20 50 100 30 25 4 10> V2 = <20 80 50 20 10 10 10> …… In WFL-EMM, a weight vector u = <u1, …, un > to indicate the weights for different predictors, where ui∈[0, 1] . ui =1 means the ith predictor is important and ui =0 implies that the ith predictor is ignored.

37. Weighted Feature Learning -Extensible Markov Model (WFL-EMM) GA Learning Process The question is how to locate a fitness weight vector u = <u1, …, un > for hurricane intensity predictions. Genetic algorithm (GA) is introduced in WFL-EMM to find the best fitness weight vector, which gives the smallest error of the prediction.

38. Weighted Feature Learning -Extensible Markov Model (WFL-EMM) Given a weights vector u = <u1, …, un >. Two steps of data transformation • Normalization: normalize all the predictor within the range of [0, 1] • First standardize the predictor values by where and sd(x) are the mean and standard deviation of the ith predictor. Then a non-linear normalization maps zi to interval [0, 1], where is damping coefficient. • Transformation: Assume a normalized record d = <d1,…,dn>. Then the • record is transformed as d’ = < u1 d1,…,un dn>.

39. Weighted Feature Learning -Extensible Markov Model (WFL-EMM)

40. Weighted Feature Learning -Extensible Markov Model (WFL-EMM) GA Learning Process • The question is how to locate a fitness weight vector u = <u1, …, un > for hurricane intensity predictions. • These weights are used during the clustering and applied to the distance/similarity measure used for clustering • Genetic algorithm (GA) is introduced in WFL-EMM to find the best fitness weight vector, which gives the smallest error of the prediction .

41. Weighted Feature Learning -Extensible Markov Model (WFL-EMM) GAs try to locate a fitness solution from the a solution space. Weight vector u = <u1, …, un > spans a vector space [0, 1]n since each uiis a real value ranged in [0, 1]. Solution space Fitness solution

42. Weighted Feature Learning -Extensible Markov Model (WFL-EMM) GA Learning Process Genetic algorithm evolution Each time, two chromosomes are selected randomly from the ith population with a probability proportional to their fitness, where a chromosome is a Gray code string of a weight vector u. Chromosome 1 Chromosome 2 Population i

43. Weighted Feature Learning -Extensible Markov Model (WFL-EMM) GA Learning Process Genetic algorithm evolution Chromosome 1 Chromosome 2 crossover mutation inversion Randomly select a break point in a chromosome and then exchange the position of the two pieces. Randomly alter one or more bits in the offspring based on a given probability. New chromosome Calculate the fitness of the obtained chromosome and place it into the population i+1

44. Weighted Feature Learning -Extensible Markov Model (WFL-EMM) GA Learning Process Fitness of the chromosome A chromosome is first decoded into a weight vector u. Apply this obtained u to generate a GEMM by using the training data. Then the fitness is calculated by either mean absolute deviation (MAD) or root mean square error (RMSE) based on the testing data. The best fitness weight vector u is located during the evolution of a GA. Fitness where

45. Results - Experiment 2: Evaluating WFL-EMM by using k-fold cross validation technique over the dataset from 1982 to 2003 (set MAD as fitness).

46. Results It is interesting to look at the weights of the features because these weights reveals information about what the main drivers of intensity change might be.

47. Learn feature weights using Genetic Algorithm. Weights for features over time.

48. PIIH – Prediction Intensity Interval Model for Hurricanes TRACDSTM • Historic hurricane data • Features • Current wind speed • Various temperatures • Time of the year • Direction of movement • GOES Satellite Data (IR) • Currently 23 features from the Statistical Hurricane Intensity Prediction Scheme (SHIPS) Data stream clustering + temporal order model

49. Prediction using PIIH – Irene (2011) Current features of hurricane

50. Prediction using PIIH – Irene (2011) Current features of hurricane Aggregate possible future scenarios into a prediction