1 / 62

How Fast Data Is Turned into Fast Information and Timely Action

How Fast Data Is Turned into Fast Information and Timely Action. Lucas Jellema. Oracle OpenWorld 2014, San Francisco, CA, USA. Audience Challenge. Audience Challenge. Audience Challenge. Audience Challenge. Audience Challenge. Audience Challenge. Filter. Pattern Detection.

rhea
Download Presentation

How Fast Data Is Turned into Fast Information and Timely Action

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How Fast Data Is Turned into Fast Information and Timely Action Lucas Jellema Oracle OpenWorld 2014, San Francisco, CA, USA

  2. Audience Challenge

  3. Audience Challenge

  4. Audience Challenge

  5. Audience Challenge

  6. Audience Challenge

  7. Audience Challenge

  8. Filter PatternDetection Agregate

  9. Fast Data Example 14,0 16,1 14,1 16,1 16,0 13,1 14,0 16,0 13,1 13,0 14,1 16,0 14,1 13,0 14,1 16,0 13,1 14,0 Smart Processing • Information • Conclusion • Alert • Recommendation • Action

  10. Demonstration: Live Tennis • Tennis Tournament • Many matches played in parallel • The data that is produced: • At a rate of up to 10 events/minute Match Id, Player [whoscored] 14,0 16,1 14,1 16,1 16,0 13,1 14,0

  11. Demonstration: Live Tennis • The information, conclusions &actions we are looking for: • Scoreboard per game, set, match • Match start andcompletion (action: inform next players for that court) • Interrupted match (action: go and check out the reason for the interruption) Fast Data Smart Processing • Scoreboard • Match start andcompletion • Interrupted match

  12. Real Time – from event to UIPush through Web Sockets msg Smart Processing WebLogic Fast Data WebSocket Server Oracle Event Processor msg event CQL queries

  13. WebSocketPowered Scoreboard

  14. OEP applicationtoprocessfast tennis data • Preparation • Define event definitions • Createlocal, in memory cache withstatic, enriching data • Gather (in this case generate) tennis data through adapter • Create Event Sinktoconsumeallfindingsandpublishto console TennisMatchEvent matchId player

  15. Match Level events

  16. Rally’s to games TennisMatchEvent matchId player • The first playerto have won more than 4 points • and have won two or more points more than his opponent

  17. Detectinterrupted matches by ‘finding’ missing events • When a match is interrupted, obviously no more ‘rally point events’ are produced • Detecting the absence of these events for a match [that has begun] is equivalent todetectinganinterruption of the match • Unless the match is complete becausesomeone won

  18. Detectinterrupted matches by ‘finding’ missing events

  19. Complete EPN diagram for Tennis Tournament Processor • A single OEP applicationthatconsumes fine grained rally point events andperformsthree-stage aggregationandenrichment New Match TennisMatchEvent matchId player Match Finish Interrupted Match Set Won Game Won

  20. Overview • What is [special about] Fast Data? • Continuous, Volume|Velocity|Variety, Real Time • Challenges • Volatile, non persistent • Data => Information, Conclusion, Alert, Recommendation, Action • Strategies • Smart gathering • Discard – filter, aggregate, pattern(andalso look for missing events!) • Promote (process, enrich) • Visualize • Technology/Tools • Demonstration/Cases

  21. Fast Data • Tweet • Feed • Beat • Signal • Measurement • Message • Mail • Notification • Tick • Pulse

  22. New theme(thatbringsitalltogether)

  23. Some event producingdevices

  24. Most of these events….

  25. Fast Data Processing Fast Data Smart Processing • Information • Conclusion • Alert • Recommendation • Action

  26. Fast Data ProcessingMulti-stage cleansing & aggregation Fast Data Smart Processing • Information • Conclusion • Alert • Recommendation • Action

  27. Data captured Analysis completed Typical Flow andAdditional Challenge… Fragmented event entities Business event Action taken Business Value TIME

  28. The V-factor VALUE Volume Velocity Variety

  29. Keystrategy • Discard – as early as possible (close to the source) • Ignore irrelevant events • Filter out unneededattributes • Takes samples instead of entirestream • Aggregate: merge multiple, correlated events intoone

  30. Fast Data Processing:Oracle Event Processor Smart Processing RMI File REST HTTP Channel JMS Business Rule Database Custom (Java) SOA Suite EDN Coherence JMX Fast Data RMI File REST HTTP Channel JMS Database Custom (Java) SOA Suite EDN Coherence JMX QuickFix (financial) Oracle Event Processor

  31. Oracle Event Processor events • Light weight, real-time (sub-sub-second), in-memory, continuous query engine • Available in embedded form – withcorrespondinglicence • Interactswithmany different channels – inboundandoutbound • Has internal caches toenrich events andtemporarilyretain events • Uses CQL to: • Filter, aggregate, enrichanddetectpatterns (including missing events) Event Processor CQL

  32. Fast Data ProcessingFusionMiddlewareTooling Smart Processing SOA Suite 12c EDN Fast Data RMI File REST HTTP Channel JMS Business Rule JMX Database Custom (Java) RMI File REST HTTP Channel JMS Database JMX Custom (Java) Coherence Oracle Event Processor

  33. Fast Data ProcessingFusionMiddlewareTooling Fast Data Smart Processing • Information • Conclusion • Alert • Recommendation • Action BI RTD BAM BPMSuite EDN SOA Suite ADF Coherence NoSQL Task BPEL OEP ODI Golden Gate

  34. Business User FriendlyExploration of Fast Data: Stream Explorer

  35. Credit Card TheftDetection • Severalsituations in the past • Credit card is stolen in the main terminal building • Severalpurchases are made in shops on the way fromthat area to the main exit • Purchasesbetween $200-$500 dollar • Purchases made within 5 minutes of eachother • Sometimes the purchases are made in notentirely the direct route to the exit EXIT Main Terminal

  36. Credit Card TheftDetection • Severalsituations in the past • Credit card is stolen in the main terminal building • Severalpurchases are made in shops on the way fromthat area to the main exit • Purchasesbetween $200-$500 dollar • Purchases made within 5 minutes of eachother • Sometimes the purchases are made in notentirely the direct route to the exit • To catch the perpetrator • Consume the credit card purchase event stream for airport shops • Spot situationswherethree or more purchases of $200-$500 are made within 5 minutes fromeachotherandroughly in the terminal => exit physical order • Publishan event to alert security staff • Towatch for anyfurtherpurchaseswiththat credit card • Toinform show staff for that credit card • Tosendstaffto the exit totryandapprehend the thief(perhapsbased on the shoppingbags he is carryingfrom the shops he bought stuff at)

  37. Catch me ifyoucan EXIT Main Terminal

  38. Catch me ifyoucan EXIT $440 $250 $300 $380 Main Terminal

  39. CQL todetect‘funny string of transactions’

  40. Real Time – from Event toTaskOEP => SOA Suite 12c EDN Smart Processing SOA Suite 12c Fast Data Oracle Event Processor event event EDN

  41. Real Time – from Event toTaskOEP => SOA Suite 12c EDN Smart Processing SOA Suite 12c Fast Data Task BPEL Oracle Event Processor event event BPMN EDN event Medi-ator

  42. From OEP findingto EDN Business Event triggering the SOA Suite

  43. Human consumers • Slow at data processing • Notelectronicallyconnected • Visuallyoriented (1 picture > 1000 words) • Frequently (thoughperhapsdecreasinglyso) the actor or decision maker • Interactalong human communicationchannels • Usevisualizationto present findings, conclusions, recommended actions • And as a second tier of fast data processing:Highlight (filter), aggregate, patterns, extrapolate/interpolate, missing elements • Sometimes take over fromhumansandjust take action

  44. VisualizeandAggregate

  45. Real Time – from event to UIBusiness Activity Monitoring Smart Processing WebLogic Fast Data BAM Oracle Event Processor event msg msg JMS

  46. Real Time – from event to UIADF DVT Visualizations Smart Processing WebLogic Fast Data ADF DVT Oracle Event Processor event msg msg JMS

  47. Visualizephysicallocations of [string of] suspicious transactions

  48. Summary • Fast Data (events): Vast, Continuous, Velocity, Variety • Wanted: Near real time conclusions, recommendations, alerts, actions • Strategy: • Discard – as early as possible (Filter, Aggregate) • Enrich, CorrelateandPattern Match, Missing Events, Retain, Publishhigher level, more coarsegrained business event • Repeatthiscycle multiple times (such as rally point, game, set, match) • Technology for Fast Data processing: Oracle Event Processor & CQL • Interactswith JMS, EDN, RMI, HTTP (/REST), JMX, Database, Coherence • New: Stream Explorer – business friendly, industrypatternbasedfast data explorationsandvisualization • To assist humans in Fast Data and Information Processing: Visualization • Filter, Aggregate, Enrich, Pattern Match (1 picture > 1000 words) • Technology: BAM (Dashboard andRule processing), ADF Data Visualization • Also: turn findingsinto actions using Human Task, BPEL and BPM via the SOA Suite 12c Event Delivery Network (EDN)

More Related