1 / 32

Learning To Predict rare events in event sequences

Learning To Predict rare events in event sequences. By Gary M Weiss and Haym Hirsh Presented by Veena Raja Rathna. Contents. Aim & Introduction to the problem Basic problem Formulation Definitions & Evaluation metrics Learning methods

nile
Download Presentation

Learning To Predict rare events in event sequences

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Learning To Predict rare events in event sequences By Gary M Weiss and Haym Hirsh Presented by Veena Raja Rathna

  2. Contents • Aim & Introduction to the problem • Basic problem Formulation • Definitions & Evaluation metrics • Learning methods • Example & Results TimeWeaver • Reference

  3. AIM Introduction to the problem • To predict rare events from sequences of events which contain non numerical features • An event sequence is a time stamped observations described by a fixed set of features • Why • Predicting events with categorical data is an important real world problem • Not Suited to be solved by existing statistical and machine learning methods

  4. Background • Classical time series - Predict next ‘n’ successive observations from a history of past observations • Statistical techniques not applicable to event prediction because they require numerical features do not support predicting a specific ‘event’ within window of time.

  5. Examples • Telecommunication equipment failure • Predicting Fraudulent credit card transactions • Start of Transcription in DNA sequences

  6. Basic Problem Formulation • Definitions • Event- Et: is a time stamped observation which occurs at time ‘t’ and is described by a set of feature value pairs. • Event Sequence- S: is a time ordered sequence of events.S=Et1,Et2,….Etn • Domain Object- D: events are associated with D which is a generator of events

  7. Definitions • Target Event Xt :is the event to be predicted and is specified by a set of feature value pairs. • Warning Time W:is the lead time necessary for a prediction to be useful • Monitoring Time M: determines the maximum amount of time prior to target event for which a prediction is considered correct.

  8. Problem now reduces to • Learning a prediction procedure P that correctly predicts the target events. • P is a function that maps an event sequence to a boolean prediction value. • P:Et1,Et2,Et3,…Etn-->{+,-} • A target event is predicted if at least one prediction is made within its prediction period • A prediction is correct if it falls within the prediction period of some target event

  9. Evaluation Measures for Event Prediction • Recall = #Target Events Predicted /Total target events It is the percentage of target events correctly predicted • Simple Precision = TP/TP+FP TP - true predictions FP - false Predictions Simple precision is the percentage of predictions that are correct

  10. Evaluation Metrics • Normalized precision = #Target Events Predicted/#Target Events Predicted +FP Replaces # of correct predictions with target events correctly predicted • Reduced Predictions = #Target Events predicted/# Target Events Predicted+Discounted FP • A prediction is active for a period equal to its monitoring time

  11. Basic Learning Method • Identify prediction patterns : The space of PP is searched to identify a set ‘C’ of candidate PP.Each pattern c  C should do well at predicting a subset of target events. • Generate prediction rules : An ordered list of PP is generated from C. Prediction rules are then formed by creating a disjunction of the top n PP,thereby creating solutions with different precision/recall values. • PP is a sequence of events connected by ordering primitives that define sequential or temporal constraints between consecutive events

  12. Ordering primitives are • Let A B C D represent individual events • ‘Wildcard’ “*” matches any number of events so the PP A*D matches ABCD • ‘next’ “.” matches no events so the PP A.B.C only matches ABC • ‘unordered’ “|” allows events to occur in any order and is commutative so that the PP A|B|C will match ,amongst others,CBA

  13. Example • “|” has the highest precedence so the pattern “A.B*C|D|E” matches A followed by B,followed sometime later by a C,D and E in any order. • Each feature in an event is permitted to take on the “?” value that matches any feature value. • PP also has an integer valued pattern duration

  14. Learning Method • A PP matches an event sequence if • 1) events within the PP matches events within an event sequences • 2) ordering constraints in the PP are obeyed • 3) events involved in the match occur within the pattern duration • This language enables flexible and noise tolerant prediction rules to be constructed, such as • if 3(or more) A events & 4( or more B events occur within an hour, then predict the target event.

  15. GA for identifying PP • Use GA to identify a diverse set of PP. • Each individual in the GA’s population represents part of a complete solution and should perform well at classifying a subset of the target events. • GA used is steady state GA where only a few individuals are modified each ‘iteration’ .

  16. Basic steps in GA • 1. Initialize population • 2. while stopping criteria not met • 3. select 2 individuals from the population • 4. apply mutation operator to both individuals with Pm; • else apply crossover operator • 5. evaluate the 2 newly formed individuals • 6. Replace 2 existing individuals with the new ones • 7.done

  17. Selection and Replacement Strategy • GA’s selection and replacement strategy must balance two opposing criteria • 1. They must focus the search in the most profitable areas of the search space. • 2. Maintain a diverse population to avoid premature convergence and to ensure that the individuals in the population collectively cover most of the target events.

  18. Selection & Replacement Strategy • Fitness of the PP is based on both its precision and recall and is given by fitness = ((* )+1)precision recall ------------------------------------- (* )precision+recall • To encourage diversity we use a strategy called sharing that rewards individuals based on how different they are from other individuals in the population

  19. Selection & Replacement Strategy • Individuals are selected proportional to their shared fitness = fitness/ count • count measures the degree of similarity of individual j to p individuals comprising the population count j=(1 -distance(j k))3 The similarity of 2 individuals is measured using a phenotypic distance metric that measures the distance based on the performance of the individual

  20. Observations • The more similar an individual to the rest of the individuals in the population, smaller the distances & greater the count. • Replacement strategy also uses shared fitness. • Individuals are chosen for deletion inversely proportional to their shared fitness

  21. Creating Prediction Rules • Greedy Algorithm shown is used to form a list of prediction rules S from a set of candidate patterns C • Precision, recall and prediction vector information computed in the first step for each PP are used. • Step 11 requires access to the training set and is the most time-intensive step

  22. Algorithm • 1. C=patterns returned from the GA;S={}; • 2. while C!=0 do • 3. for c  C do • 4. if(increase_recall(S+c,S)<=THRESHOLD) • 5. then C=C-c; • 6. Elseeval=PF*(c.precisionS.precision)+increase_recall(S+c,S); • 7. done • 8. Best={c C,x  C|c.eval>=x.eval} • 9. S=S||best;C=C=best; • 10. recompute S.precision on the training set; • 11. done

  23. Observations • Builds solution with increasing recall by heuristically selecting the best PP in C using eval function in line 6. • PF controls the importance of precision vs recall.PPs that do not increase the recall by at least THRESHOLD are discarded. • Both THRESHOLD AND PF affect the complexity of t he learned concept and can prevent overfitting of the data

  24. TimeWeaver • This paper describes timeweaver a GA based ML system that predicts rare events • For AT & T ,the specific task is to predict failure of 4ESS h/w components from alarm messages. • Problem Formulation • Each 4ESS generated alarm is an event with 3 features--device,severity ,code • each 4ESS is a domain object that generates an event sequence and the target event is any event with code set to ‘FAILURE’

  25. Experiments • Training set 110,000 alarms reported from 55 4ESS switches. • Test set contained 40,000 alarms from 20 different 4ESS switches.This data included 1200 alarms which indicated equipment failure. • THRESHOLD was 1% and PF was 10 • Pattern 351:<TMSP,?,MJ>*<?,?,MJ>*<?,?MN> indicates that within 351 sec time period a major severity alarm on a TSMP device is followed by a major alarm and minor alarm

  26. Results • Varying the warning time demonstratesthat it is much easier to predict failures when short warning time is required. • Increasing the monitoring time significantly improves TW’s ability to predict failures. • Larger prediction period leads TW to focus its attention on ‘spurious correlations’ in data.

  27. Comparision with other methods • Tw was compared to C4.5rules and RIPPER - 2 rule induction systems and FOIL a system that learns logical definitions from relations. • Class distribution of the generated examples is skewed; prevented C4.5rules and RIPPER from predicting any failure. • TW yields results with precision 3-5 times higher for a given recall value than various thresholding strategies used by ANSWER system. Concept space is much more expressive.

  28. Example • {100:a,b},{104:c,c},{105:d,a},{108:c,c},{110:a,d}, {111:crash,c},{115:d,a},{118:,c,c},{119:a,d},{124:a,b}! Format: Integer valued timestamp,colon,comma separated list of feature values.2 feature values per event and each event can take on the values a,b,c or d.The first can also take on the value “crash”. The target event is any event with ‘crash’ as the first feature. Warning time is 2 secs and monitoring time is 8 secs. TW-GA training on this might produce PP: 4:|c,c|*|c,c|.

  29. References • Learning to predict rare events in categorical time series data • http://paul.rutgers.edu/~gweiss/thesis/timeweaver.html

More Related