Uncertainty in Automation: Anomaly Detection in Event-Based Systems

1. Uncertainty in Automation:Anomaly Detection in Event-Based Systems Dawn Tilbury Linday Allen (PhD) and John Broderick University of Michigan

2. Outline • Example problem: inconsistent logical behavior • Developed solution: Anomaly detection • Model generation using observed data • Performance assessment of models using known “good” and “bad” behavior • On-line anomaly detection • Industrial Application • Academic assumptions meet industry realities • Resolution and results • Lessons learned

3. OPC Tag OPC Tag Response Response s1 Example from our testbed Correct, typical behavior Incorrect behavior PartReady Part1 Release Pallet PartReady Part1 LoadPart1 • No model of entire system’s correct behavior • Manual inspection required to find this anomaly • Laborious, offline

4. Events All Events Op. Feedback Anomalies Approach: Anomaly detection using model generation Plant Plant • Goal: diagnosis of system level event-based faults in mfg systems without use of pre-existing formal model Anomaly Fault diagnosis detection Controller Controller Logic Control System Operator • Method: • Generate models based on training data • Detect anomalies on-line by comparing traces to models • Advise the operator when anomaly occurs

5. Knowns & Unknowns • Known: • Resources in system • Robots • CNC machines • Pallets • Measurable: • OPC tag changes • Communication events between controllers • Unknown: • Formal model of the systemCould be constructed but is time-consuming and error prone • Logic control codeWritten by different people at different times in different languages • Correct event orderMany different orders may be acceptable

6. Resource Info ok efad... Model Generation ok abfc... adbe... ok Solution approach Given resource information and strings of “ok” events Given a new string, determine whether the models accept it (weight by model performance) Given some “ok” and “not ok” strings, compute the performance of each model abfe... 6 31 22 ok Performance Assessment abfe... Fault Detection OR Create a set of models that can generate these strings abfe... not ok eafd... not acea... If not, where is the anomaly

7. Anomaly Detection Method • Inputs: • Streams of events from system in operation • Resource information, including mapping of events to resources • Outputs: • Set of models that represent the system behavior • Model performance on training data • On-line detection: • Score for each string – anomalous or not

8. G2 G1 M1 M6 2 3 4 1 Machining Cell: Physical Set-Up • Problem: G2 will have raw parts and at least one CNC available, but G2 incorrectly waits • Resources: • Gantries, CNCs, buffer at hand-off • Events: PLC data recorded via Ford data collection system Entry Hand-off Exit Reject

9. Driving logic Data collection set-up • Data from each machine & gantry • Bits include: Cycle End, Good/Bad Cycle, Wait Aux, Blocked, and Starved • PLC message generated each time particular bit changes occur • Approx. 11,000 parts worth of data IT System FunctionBlock FunctionBlock DrivingLogic Driving Logic PLC PLC (270,000 PLC messages)

10. Identified Inconsistencies • What we thought we would get: • Well-defined strings of events • Events that acquire/release resources recorded • Unique mapping of PLC bits to events • Many strings, starting from the initial state, labeled as “good” or “bad • What we got: • Not every event triggers a message  multi-bit change (order is uncertain) • Not all resource events captured in data collection • Some bits used for multiple purposes • One huge log file with no defined “beginning”

11. Resolution of Inconsistencies

12. Word 18 bits 8-10 give the CNC ID Ford data Gantry waiting: word 19 bit 9 is high

13. Lessons learned • Sometimes you can adapt/improve your method to handle given uncertainties • Multiple models when system model unknown • Multiple bit changes  uncertain event order • Initial state unknown • Advise operator instead of closing the loop • Sometimes you have to decrease the uncertainty by improving the system • Consistent bit/event mapping • Unobservable events for acquiring resources