1 / 39

Experimental Analysis of Sequential Decision Making Algorithms for Port of Entry Inspection Procedures

Saket Anand David Madigan Richard Mammone Saumitr Pathak Fred Roberts. Experimental Analysis of Sequential Decision Making Algorithms for Port of Entry Inspection Procedures. RUTGERS UNIVERSITY. Port of Entry Inspection Procedures.

sumana
Download Presentation

Experimental Analysis of Sequential Decision Making Algorithms for Port of Entry Inspection Procedures

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Saket Anand David Madigan Richard Mammone Saumitr Pathak Fred Roberts Experimental Analysis of Sequential Decision Making Algorithms for Port of Entry Inspection Procedures RUTGERS UNIVERSITY

  2. Port of Entry Inspection Procedures • Goal: Find ways to intercept illicit nuclear materials and weapons destined for the U.S. via the maritime transportation system subject to minimization of delays, manpower and equipment utilization • Find inspection schemes that minimize total “cost” including “cost” of false positives and false negatives

  3. Mobile VACIS: truck-mounted gamma ray imaging system Port of Entry Inspection Procedures • Formulation of the problem as a complex sequential decision making problem • Containers have attributes • Sample attributes: • Does ship’s manifest set off an “alarm”? • What is the neutron or Gamma emission count? Is it above threshold? • The Decision Maker’s Problem: • Which attributes to inspect? • Which inspections next based on previous results? • Approach: • Builds on ideas of Stroud and Saeger at Los Alamos National Laboratory

  4. 0110 F(0110) = 0/1 Sequential Decision Making Problem • Simplest Case: Attributes are in state 0 or 1 • Sensors measure presence/absence of attributes • Then: Container is a binary string like 0110 • So: Classification is a Boolean decision function F that assigns each binary string to a category (0 or 1). Category 0 = “ok” and 1 = “suspicious” • Example: 00001111 ; F(000) = F(001) = F(010) = F(011) = 0, F(100) = F(101) = F(110) =F(111) = 1,

  5. Binary Decision Tree Approach • Binary Decision Tree: • Nodes are sensors (A,B,C, etc.) or categories (0 or 1) • Stroud and Saeger enumerate all “complete” and monotone boolean functions and calculate the least expensive corresponding binary decision trees.

  6. Sensitivity Analysis of Stroud Saeger Experiments • Aim – Experimental Analysis of the robustness of the optimal binary decision tree (BDT) implementing the inspection scheme found by the Stroud-Saeger1 approach • How sensitive are the optimal Binary Decision Trees to variations in the cost and sensor parameters? 1 Stroud, P. D. and Saeger K. J., “Enumeration of Increasing Boolean Expressions and Alternative Digraph Implementations for Diagnostic Applications”, Proceedings Volume IV, Computer, Communication and Control Technologies, (2003), 328-333

  7. Cost Function used for Evaluating the Decision Trees. CTot =CFalsePositive *PFalsePositive + CFalseNegative *PFalseNegative+ Cutil CFalsePositive is the cost of false positive (Type I error) CFalseNegative is the cost of false negative (Type II error) PFalsePositive is the probability of a false positive occurring PFalseNegative is the probability of a false negative occurring Cutil is the expected cost of utilization of the tree. Note: In this cost model, other costs such as fixed costs and costs due to delay are not considered

  8. Ti P(Yi|X=1) P(Yi|X=0) Characteristics of a typical sensor Probability of Error for Individual Sensors • For ith sensor, the type 1 (P(Yi=1|X=0)) and type 2 (P(Yi=0|X=1)) errors are modeled using Gaussian distributions. • State of nature X=0 represents absence of a bomb. • State of nature X=1 represents presence of a bomb. • Yi represents the outcome (count) of sensor i. • Σi is variance of the distributions • PD = prob. of detection, PF= prob. of false positive Ki i

  9. A A B 0 B 0 C 1 C 1 1 0 1 0 Probability of Error for The Entire Tree State of nature is one (X = 1), presence of a bomb State of nature is zero (X = 0), absence of a bomb Probability of false positive (P(Y=1|X=0)) for this tree is given by Probability of false negative (P(Y=0|X=1)) for this tree is given by P(Y=1|X=0) = P(YA=1|X=0) * P(YB=1|X=0) + P(YA=1|X=0) *P(YB=0|X=0)* P(YC=1|X=0) Pfalsepositive P(Y=0|X=1) = P(YA=0|X=1) + P(YA=1|X=1) *P(YB=0|X=1)*P(YC=0|X=1) Pfalsenegative

  10. Stroud Saeger Experiments • Stroud-Saeger ranked all trees formed from 3 or 4 sensors A, B, C and D according to increasing tree costs. • Used cost function defined above. • Values used in their experiments: • CA = .25; P(YA=1|X=1) = .9856; P(YA=1|X=0) = .0144; KA = 4.37; ΣA = 1; • CB = 1; P(YC=1|X=1) = .7779; P(YB=1|X=0) = .2221; KB = 1.53; ΣB = 1; • CC = 10; P(YD=1|X=1) = .9265; P(YC=1|X=0) = .0735; KC = 2.9; ΣC = 1; • CD = 30; P(YD=1|X=1) = .9893; P(YD=1|X=0) = .0107; KD = 4.6; ΣD = 1; • Here, Ci = unit cost of utilization of sensor i, Ki is the sensor discrimination power and Σi is the relative spread factor for sensor i. • Also fixed were: CFalseNegative, CFalsePositive, P(X=1)

  11. Stroud Saeger Experiments: Our Sensitivity Analysis • CFalseNegativewas varied between 25 million and 500 billion dollars • Low and high estimates of direct and indirect costs incurred due to a false negative. • CFalsePositive was varied between $180 and $720 • Cost incurred due to false positive (4 men * (3 -6 hrs) * (15 – 30 $/hr) • P(X=1)was varied between 3x10-9 and 1x10-5

  12. Stroud Saeger Experiments: Our Sensitivity Analysis • First set of Computer experiments: n = 3; (use sensors, A, C and D) • Experiment 1: Fix values of two of CFalseNegative,CFalsePositive, P(X=1) and vary the third through their interval of possible values. • Experiment 2: Fix a value of one of CFalseNegative,CFalsePositive, P(X=1) and vary the other two. • Do 10,000 experiments each time. • Look for the variation in the highest ranked tree.

  13. 493 Variation of CTot vs. CFalseNegative • P(X=1) and CFalsePositive were kept constant at the specified value and CTot was • computed for 10,000 randomly selected values of CFalseNegative in the specified range. • Randomly selected fixed parameter values

  14. 426,807,420,776 Variation of CTot vs. CFalsePositive • P(X=1) and CFalseNegative were kept constant at the specified value and CTot was • computed for 10,000 randomly selected values of CFalsePositive in the specified range. • Randomly selected fixed parameter values

  15. 447,470,143,842 352 Variation of CTot vs. P(X=1) • CFalsePositive and CFalseNegative were kept constant at the specified value and CTot was • computed for 10,000 randomly selected values of P(X=1) in the specified range. • Randomly selected fixed parameter values

  16. Variation of CTot wrt CFalseNegative and CFalsePositive • Randomly selected fixed parameter values CTot =CFalsePositive *P(X=0)*P(Y=1|X=0) + CFalseNegative *P(X=1)*P(Y=0|X=1)+ Cutil

  17. 669 Variation of CTot wrt CFalseNegative and P(X=1) • Randomly selected fixed parameter values CTot =CFalsePositive *P(X=0)*P(Y=1|X=0) + CFalseNegative *P(X=1)*P(Y=0|X=1)+ Cutil

  18. 82,737,009,757 Variation of CTot wrt CFalsePositive and P(X=1) • Randomly selected fixed parameter values CTot =CFalsePositive *P(X=0)*P(Y=1|X=0) + CFalseNegative *P(X=1)*P(Y=0|X=1)+ Cutil

  19. a c 1 d a 1 0 a c 1 1 c d d 0 1 d 0 1 1 0 0 1 Structure of trees which came first Rank with 3 sensors (A, C and D) Tree number 37 Boolean Fn: 00011111 Tree number 49 Boolean Fn: 01010111 Tree number 55 Boolean Fn: 01111111 In the 10,000 experiments, only 3 out of the 60 Binary Decision Trees ever attained first rank

  20. Stroud Saeger Experiments: Our Sensitivity Analysis • Second set of computer experiments: n = 4; (use sensors, A, B, C, D). • Experiment 1: Fix values of two of CFalseNegative,CFalsePositive, P(X=1) and vary the third through their interval of possible values. • Experiment 2: Fix a value of one of CFalseNegative,CFalsePositive, P(X=1) and vary the other two. • Do 10,000 experiments each time. • Look for the variation in the highest ranked tree.

  21. 189 Variation of CTot vs. CFalseNegative • P(X=1) and CFalsePositive were kept constant at the specified value and CTot was • computed for 10,000 randomly selected values of CFalseNegative in the specified range. • Randomly selected fixed parameter values

  22. 240,407,400,315 Variation of CTot vs. CFalsePositive • P(X=1) and CFalseNegative were kept constant at the specified value and CTot was • computed for 10,000 randomly selected values of CFalsePositive in the specified range. • Randomly selected fixed parameter values

  23. 406,238,290,733 298 Variation of CTot vs. P(X=1) • CFalsePositive and CFalseNegative were kept constant at the specified value and CTot was • computed for 10,000 randomly selected values of P(X=1) in the specified range. • Randomly selected fixed parameter values

  24. Variation of CTot wrt CFalseNegative and CFalsePositive • Randomly selected fixed parameter values CTot =CFalsePositive *P(X=0)*P(Y=1|X=0) + CFalseNegative *P(X=1)*P(Y=0|X=1)+ Cutil

  25. 454 Variation of CTot wrt CFalseNegative and P(X=1) • Randomly selected fixed parameter values CTot =CFalsePositive *P(X=0)*P(Y=1|X=0) + CFalseNegative *P(X=1)*P(Y=0|X=1)+ Cutil

  26. Variation of CTot wrt CFalsePositive and P(X=1) 47,484,728,943 • Randomly selected fixed parameter values CTot =CFalsePositive *P(X=0)*P(Y=1|X=0) + CFalseNegative *P(X=1)*P(Y=0|X=1)+ Cutil

  27. a b 1 d c a 1 d 0 1 b 1 c 1 1 d 0 1 0 1 Structure of trees and corresponding Boolean Expressions for n = 4 Tree number 11785 Boolean Fn: 0111111111111111 Tree number 11605 Boolean Fn: 0101011111111111 In the above experiments, ≤ 10 out of the 11,808 Binary Decision Trees ever attained first rank

  28. a b b a c d c 1 b d d 1 0 0 1 1 c d 0 1 d 1 0 0 1 1 0 0 Structure of trees and corresponding Boolean Expressions for n = 4 Tree number 9133 Boolean Fn: 0001010111111111 Tree number 8965 Boolean Fn: 0001010101111111 In the above experiments, ≤ 10 out of the 11,808 Binary Decision Trees ever attained first rank

  29. 1 PD Operating Point EER 0 PF 1 Ki Ti P(Yi|X=1) P(Yi|X=0) Receiver Operating Characteristic (ROC) Curve • The ROC curve is the plot of the probability of correct detection (PD) vs. the probability of false positive (PF) • Sensor threshold is varied to select an operating point; trade off between the PD and PF • Each sensor has an ROC curve and the combination of sensors into a decision tree has a composite ROC curve • Equal Error Rate (EER) is the operating point on the ROC curve where, PF=1 - PD

  30. Sensitivity to Sensor Performance Following experiments have been done using sensors A, B, C and D as described below by varying the individual sensor thresholds Tifrom -4.0 to +4.0 in steps of 0.4. These values were chosen since they gave us a ROC curve for the individual sensors over a complete range P(Yi=1|X=0) and P(Yi=1|X=1) PF for the ith sensor is computed as: P(Yi=1|X=0) = 0.5 erfc[Ti/√2] PD for the ith sensor is computed as: P(Yi=1|X=1) = 0.5 erfc[(Ti-Ki)/(Σi√2)] CA = .25; KA = 4.37; ΣA = 1; CB= 1; KB = 1.53; ΣB = 1 CC = 10; KC = 2.9;ΣC = 1; CD = 30; KD = 4.6;ΣD = 1 where Ci is the individual cost of utilization of sensor i, Ki is the discrimination power of the sensor and Σi is the spread factor for the sensor

  31. Performance (ROC) of Binary Decision Tree number 37 (3 sensors) • The lines represent performance characteristics (ROC curve) of sensors A, C and D. • The green dots represent the performance characteristics (P(Y=1|X=0), P(Y=1|X=1)) of the tree over all combinations of sensor thresholds (Ti). • 15 out of 60 trees attained first rank

  32. Performance (ROC) of Binary Decision Tree number 37 (3 sensors) • Assuming performance probabilities (P(Y=1|X=1) and P(Y=1|X=0)) to be monotonically related (in the sense that P(Y=1|X=1) can be called a monotonic function of P(Y=1|X=0)), we can find an ROC curve for the tree consisting of the set containing maximum P(Y=1|X=1) value corresponding to given P(Y=1|X=0) value. • The blue dots represent such an ROC curve, the “best” ROC curve for tree 37.

  33. Performance (ROC) of Binary Decision Tree number 445 (4 sensors) • The lines represent performance characteristics (ROC curve) of sensors A, B, C and D. • The green dots represent the performance characteristics (P(Y=1|X=0), P(Y=1|X=1)) of the tree over all combinations of sensor thresholds (Ti). • Only 244 of 11,808 trees attained first rank

  34. Performance (ROC) of Binary Decision Tree number 445 (4 sensors) • Assuming performance probabilities (P(Y=1|X=1) and P(Y=1|X=0)) to be monotonically related (in the sense that P(Y=1|X=1) can be called a monotonic function of P(Y=1|X=0)), we can find an ROC curve for the tree consisting of the set containing maximum P(Y=1|X=1) value corresponding to given P(Y=1|X=0) value. • The blue dots represent such an ROC curve, the “best” ROC curve for tree 445.

  35. Conclusions from Sensitivity Analysis • Considerable lack of sensitivity to modification in parameters for trees using 3 or 4 sensors. • Very few optimal trees. • Very few boolean functions arise among optimal trees. • Binary Decision Trees perform better than individual sensors

  36. Pallet VACIS • Explain why conclusions are so insensitive to variation in parameter values. • Explore the structure of the optimal trees and compare the different optimal trees. • Develop less brute force methods for finding optimal trees that might work if there are more than 4 attributes. • Develop methods for • approximating the optimal tree. Some Research Challenges

  37. Acknowledgement • Supported by Naval Research and National Science Foundation • The authors thank Phil Stroud Kevin Saeger Rick Picard for providing data, code and ideas

  38. Thank you http://dimacs.rutgers.edu/Workshops/PortofEntry/ Saket Anand – anands@caip.rutgers.edu Fred S. Roberts – froberts@dimacs.rutgers.edu

  39. Monotone and Complete Boolean Functions • Monotone Boolean Functions: • Given two strings x1x2…xn, y1y2…yn • Suppose that xi≤ yi for all i implies that • F(x1x2…xn) ≤F(y1,y2…yn). • Then we say that F is monotone. • Then 11…1 has highest probability of being in category 1. • Complete Boolean Functions: • Boolean function F is complete if and only if F can be calculated by finding all n attributes and knowing the value of the input string on those attributes • Example: F(111) = F(110) = F(101) = F(100) = 1, F(000) = F(001) = F(010) = F(011) = 0. • F(abc) is determined without knowing b (or c). • F is incomplete.

More Related