ARL-HRED

1. MA&D ARL-HRED FAMU-FSU Simulation Group Output Analysis Overview 7 Dec 2005

2. Who we are • Dr. James Simpson, Principal Investigator Associate Professor of Industrial Engineering Florida State University • Graduate Assistants: • Lisa Hughes • Nicholas Done • Wayne Wesley • Michelle Zeisset

3. What we do • Bring operations research expertise • Not directly involved in product development so come from perspective of a potential user • Military background adds realism Improve IMPRINT as a decision-making tool

4. Our history with IMPRINT

5. Areas of concentration Output Variable Assessment • Multiple runs • Useful metrics Data Compilation • Efficient data reporting • Graphical and tabular reports User Support • GUI • Analysis tools

6. Study approach • Use small simple models • Compare observed results to expected results • Interpret results in context of a question a potential user may want to answer • Validate and demonstrate suggested improvements using realistic models

7. Areas of concentration Output Variable Assessment • Multiple runs • Useful metrics Data Compilation • Efficient data reporting • Graphical and tabular reports User Support • GUI • Analysis tools

8. Need for multiple runs Mean: 27.9 Std dev: 3.7 Min: 10.8 Max: 33.8 25 20 15 10 5 0 16 16 18 18 20 20 22 22 24 24 26 26 28 28 30 30 32 32 34 34 % of Time Overloaded (C/G) N = 1 frequency 25 N = 100 20 15 frequency 10 5 0 Combat model (Advanced operations module)

9. How many runs? Stryker model (Maintenance model)

10. Multiple run output:Frequency histogram Number of Times Overloaded 30 25 20 15 Frequency 10 5 0 100 179 258 36.8 52.6 68.4 84.2 131.6 147.4 194.8 210.6 226.4 242.2 115.8 163.2 BFV (VACP operations module)

11. Which events led to results? 4 4 1 0 : Start 0 : Start 5 P P 2 8 8 3 3 P P 1 1 2 2 M M 3 10 7 R R 6 4 11 12 5 6 14 20: END 20: END P P 7 15 8 16 • 3 Function-level probability nodes act as switches Function Network of BFV

12. Histogram broken down by function Histogram of Number of Times Overloaded, Color-Coded by Trace Histogram of Number of Times Overloaded 30 25 20 15 F11 Frequency F10 10 F8 F4 5 0 100 179 258 36.8 52.6 68.4 84.2 131.6 147.4 194.8 210.6 226.4 242.2 115.8 163.2 BFV (VACP operations module)

13. Tabular report by function BFV (VACP operations module)

14. Areas of concentration Output Variable Assessment • Multiple runs • Useful metrics Data Compilation • Efficient data reporting • Graphical and tabular reports User Support • GUI • Analysis tools

15. Compiled per run report Operator Overload Report                                                 2 vs. 3 study (Goal orientation operations module)

16. Maximum workload peak report Tasks contributing to max workload peaks Workload Peak Summary Measures Task i # times % times 167 30 83.33% 168 3 8.33% E[max workload peaks] = 67.81 209 3 8.33% E[# of max workload peaks] = 1.47 210 21 58.33% E[# of ongoing tasks] = 8.67 211 12 33.33% 212 2 5.56% Var[max workload peak] = 19.23 219 36 100.00% Var[# of max workload peaks] = 0.68 222 36 100.00% Var[# of ongoing tasks] = 0.43 223 27 75.00% 227 36 100.00% 228 29 80.56% 229 36 100.00% 232 36 100.00% multiple runs, 36 max workload peaks through 100 runs 2 vs. 3 study (Goal orientation operations module)

17. Contingency table Key Average Occurrences/ 1 Man-Hours Total Occurrences/ 2 Man-Hours for MOS Total Occurrences/ Man-Hours for 3 MOS/ORG Level 95% Confidence Interval for total Occurrences/ 4 Man-Hours for MOS/ORG Level Overall Average and 95% Confidence Interval for 5 Occurrences/ Man-Hours C Corrective OR P Preventive RE n = 5 replications Simple study (Maintenance model)

18. Areas of concentration Output Variable Assessment • Multiple runs • Useful metrics Data Compilation • Efficient data reporting • Graphical and tabular reports User Support • GUI • Analysis tools

19. User support: output analysis • Study objective • Preliminary output assessment • System analysis modified and tailored to the needs of user • Summary performance • Comparative model study • Model characterization • Sensitivity analysis/Model validation • Model optimization or enhancement

20. Analysis guidelines

21. Example: multiple factor study • Factor values may vary depending upon characteristics such as manufacturer or material type. EXAMPLE: A certain component is made by two different companies. Component A has a MOUBF of 1 hour. Component B is a higher quality product and therefore has a higher MOUBF of 3 hours. Simple study (Maintenance model)

22. Example: multiple factor study • Significant Factors: • MOUBF • MTTR • Statistical Prediction Model: Total Direct Man hours = 68.7 - 1.4*(MOUBF) + 2.2*(MTTR) + 2.3*(# of Maintainers) + 3.16*(# of Systems) + 3.24*(Length of Run) • Prediction: • # of Maintainers • # of Systems • Length of Run • MOUBF = 2 hr; MTTR = 20 min; # maint = 3; # sys = 7; run length = 30 • Estimated total direct man hours = 236.12 Simple study (Maintenance model)

23. MA&D Collaborative Discussion

24. Workload level histogram overloaded NOT overloaded % of Time at Workload Levels for Driver 25.00% 100 runs 20.00% 21.05% 15.00% % of Time 78.95% 10.00% 5.00% 0.00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% 105% 110% 115% 120% 125% 130% 135% 140% 145% 150% 155% 160% 165% 170% 175% 180% 185% 190% 195% 200% >200% Workload Level 2 vs. 3 study (Goal orientation operations module)