Quantification of the non- parametric continuous BBNs with expert judgment

1. Quantification of the non- parametriccontinuous BBNs with expert judgment Iwona Jagielska Msc. Applied Mathematics

2. Outline of the presentation I PART1. Introduction II PART2. Method of eliciting conditional rank correlations3. Comparison of algorithms to calculate multivariate normal probabilities 4. Presentation of elicitation software UniExp III PART5. Building the Maintenance Performance Model Model variables Dependence relation6. Results 7. Conclusions and recommendations

3. 1. Introduction CATS – casual model for Air Transport Safety – motivation and purpose - three sectors of human performance ATC Model, Flight Crew Performance Model Maintenance Performance Model

4. 2. Way of assessing dependence relations • Conditional Rank correlations  Conditional probabilities of exceedence • Why normal copula? • - Advantages • known relation between partial and rank correlation • equal conditional and partial correlations • possess zero independent property • - Disadvantages • no analytical form for multivariate cumulative distribution function “Suppose that the variable X3 was observed above its qth quantile. What is the probability that also X4 will be observed above its qth quantile? “ P1= P ( FX4(X4) > q | FX3(X3) > q )

5. 2. Way of assessing dependence relations • we can calculate relationship between rank correlation and conditional probability To see the conditional probability as a function of rank correlation we integrate bivariate normal density over the given region .

6. 2. Way of assessing dependence relations “ Suppose that not only variable X3 but also X2 was observed above their qth quantile. What is the probability that also X4 will be observed above its qth quantile? ” P2= P ( FX4(X4) > q | FX3(X3) > q, FX2(X2) > q ) To find the conditional probability we integrate trivariate normal density over the given regionwith covariance matrix . We assess the higher order conditional rank correlation in the similar way.

7. 3.1. Algorithms to calculate multivariate normal probabilities • Proposed numerical integration methods: • Algorithm I and II – by Genz • - first we apply transformation to simplify integration region • later randomized quasi Monte Carlo method is used • different choice of quasi points • in algorithm I we specify number of points; algorithm II assign number of points, s.t. the requested accuracy is provided • Algorithm III and IV • - based on successive subdivisions of integration region, where each subdivision is • used to provide a better approximation of the integrand • - polynomial rule is used to approximate integrand on each subregions • - error estimate – difference between two polynomial rules of different order • - algorithm IV may involve some simplification routines (change of • variables)

8. 3.2. NumericalComparison TAi – time of calculation for algorithm i PAi – probability obtained by algorithm i EAi – estimated error of approximation provided by algorithm i 700, 1500 – number of quasi random point in Alg I; 10-5 requested accuracy for Alg II dimension = 4, determinant = 0.5271 dimension = 7, determinant = 0.489

9. 3. NumericalComparison – brief summary • Algorithms III and IV are unpractical for large scale applications since they require long time for numerical calculations • time for hypercube [0.5, inf]7 is more than 700seconds • when the procedure of subdivision of integration region is applied, algorithm do not provide the total error • In Algorithm I user needs to specify number of quasi random points used to calculation; there is no control of provided error of estimation; time of calculation depends on the number of points, not of covariance matrix • Time of calculation for Algorithm II is sometimes grater than for Algorithm I; time depends on covariance matrix; number of quasi random points depends on requested accuracy of solution At this moment Algorithm II is used in the software UniExp as the most accurate one; Algorithm I also has future potencial for implementation.

10. 4. Software elicitation tool - UniExp 1 Step – input of nodes and connections

11. 4. Software elicitation tool - UniExp 2 Step – elicitation of conditional rank correlations

12. 4. Software elicitation tool - UniExp Values of Rank Correlations can be found in RankCorrelationValues.txt file

13. 5. Maintenance Performance Model

14. 5. Maintenance Performance Model – dependence relation Elicitation with single expert we asked – 4 questions about marginal distributions – classical method of expert judgment 7 questions about conditional probabilities of exceedance All variables are negatively correlated with variable human error

15. 5. Maintenance Performance Model At the bottom of each histogram the expectation and standard derivation are shown. Unconditional expected value of human error is 0.266/10000

16. 6. Maintenance Performance Model - conditioning Number of years of experience = 3 expected value of human error increases 0.266/10000 -> 0.309/10000

17. 6. Maintenance Performance Model - conditioning Requiring at least 6 hours of sleep provides decrease of expected human error from 0.266/10000 to 0.152/10000 Moreover E(HE|WorkCond=1,Alert=6) = 0.248/10000 while E(HE|WorkCond=1)=0.398/10000

18. 7. Conclusions and recommendations • Calculation of multivariate normal probabilities is not an easy task in case of high dimension; there is still need to develop more fast (and also accurate) algorithm for higher dimension • Include Algorithm I in UniExp software; together with making UniExp to worked outside the Matlab environment • Combining experts opinion to obtain better results • Collect data describing to marginal distribution in Maintenance Performance Model • Discover other possible influential factors in Maintenance Performance Model Any other propositions?

19. Questions ???

20. Additional Slides

21. A1. Covariance matrixes used in numerical tests Dimension 4 determinant = 0.5271 determinant = 0.1099

22. A1. Covariance matrixes used in numerical tests Dimension 7 determinant = 0.4890 determinant = 0.1102

23. A2. Determinant of covariance matrix as the measure of spread from distribution Dimension 2 determinant = 1

24. A2. Determinant of covariance matrix as the measure of spread of distribution Dimension 2 determinant = 0.51, =0.714

25. A2. Determinant of covariance matrix as the measure of spread of distribution Dimension 2 determinant = 0.0199, =0.141

26. 3.2. NumericalComparison TAi – time of calculation for algorithm i PAi – probability obtained by algorithm i EAi – estimated error of approximation provided by algorithm i 700, 1500 – number of quasi random point in Alg I; 10-5 requested accuracy for Alg II Test 1 – identity covariance matrix dimension = 4 dimension = 7

27. 3. NumericalComparison Test 2 – covariance matrix with determinant 0.5 dimension = 4, determinant = 0.5271 dimension = 7, determinant = 0.489

28. 3. NumericalComparison Test 3 – covariance matrix with determinant 0.1 dimension = 4, determinant = 0.1099 dimension = 7, determinant = 0.1102

29. 5. Maintenance Performance Model • Motivation – • part of CATS model • build to describe the causal factors influencing the maintenance crew • Methodology • non-parametric BBN • Quantification • - Nodes – variables which can influence the human performance among the maintenance crew; marginal distribution – data or Classical Method (Expert Judgment) • - Conditional rank correlations – obtained from experts through the dependence probabilities of exceedance

30. 5. Maintenance Performance Model – model variables