Understanding Simulators Using Tree-Query Language

1. Understanding Simulators Using Tree-Query Language New idea: Treatment learners Tim Menzies1, Ying Hu1, James D. Kipers2 1SE, ECE, UBC, Canada 2Com Sci, Miami, Oxford, Ohio, USA tim@menzies.com, huying_ca@yahoo.com, kiperjd@muohio.edu

2. Business motivation • IV&V practitionersseek: • Cheapest questions from IV&V to the projects • Conversations take time • Data collection = $$$ • Least changes proposed by IV&V to the projects • Organizational change is hard • Fewest monitors required for the IV&V effort • Monitoring = on-going data collection = $$$ • Method: • Knowledge farming • Treatment learning

3. Knowledge Farming • If data is plentiful: • Then data mining (e.g. Khoshgoftaar & Allen) • Else, farm it: • Plant a seed • Quickly build (or borrow) a model • Grow the data • Monte Carlo (ish) simulations • Harvest • Summarize • Happy surprise: • When we explore a large space of “maybes”… • …Stable conclusions exist.

4. stableRequirements if effectiveReviews @ 0.3 and requirementsUsed @ 0.3 …. and (workProductsIdentified @ 0.3 or … or softwareTracking @ 0.3). Planting some seeds • ARRT • The CMM2 model- • 350 lines of Prolog • Unsure of influence weights? • Jiggle! • Unsure which of N to do? • At runtime, pick at random! • The COCOMO-R risk mitigation model • Risk of development time over-run • Unsure of internal tuning parameters? • Pick randomly from tunings in the literature • Unsure of inputs (e.g. SLOC)? • Pick randomly from possible ranges. • Generate sketches of local intuitions • Qualitative models, balanced score cards, fault trees,… • Unsure if X or not X? Add them both in!

5. Harvest: improve the scores sunny, 85, 86, false, none (21=$2) sunny, 80, 90, true, none sunny, 72, 95, false, none rain, 65, 70, true, none rain, 71, 96, true, none rain, 70, 96, false, some (22=$4) rain, 68, 80, false, some rain, 75, 80, false, some sunny, 69, 70, false, lots (23=$8) sunny, 75, 70, true, lots overcast, 83, 88, false, lots overcast, 64, 65, true, lots overcast, 72, 90, true, lots overcast, 81, 75, false, lots hours on course outlook temp humidity wind Lots  some Lots  none Delta(outlook.overcast)=10= ((8-2)*(4-0))+((8-4)*(4-0)) (4+0+0) Outlook= overcast Humidity= 90.. 96%

6. If no change, Then lots of golf 6/(6+3+5)= 43% If outlook=overcast, Then lots of golf 4/(4)= 100% Least change: bribe DJs to lie about the weather assumes causality, controllability BTW- deltaf >> tree query (faster, simpler). Harvesting from data: golf If humidity=90..97 Then lots of golf 1/(4)= 25% Least monitor: watch the humidity- alert if rising over 90%

7. Harvesting from seed1: ARRT • ARRT: manual risk balancing • TAR2+ARRT: • 88 possible actions (28 8  billion*billion*billioncombinations) • Random combinations  <benefit,cost> • Seek actions high benefit, low cost Treated (found automatically in 88 secs): Untreated: do! P697=Y P919=Y P753=Y P692=N don’t! worst best worst best

8. Same Not 2 Lower cost of requirements used (via B?) (A) Using ultra-lightweight formal methods such as proposed by Leveson et.al. (B) sharing requirements documents around the development team in some searchable hypertext format (C) Build test stubs Harvesting from seed2: CMM-2 Lower cost of formal reviews at milestones (via A?) Do periodicsoftwarereviews Lower cost of unit testing (via C?) Warning: general conclusions may not apply to specific projects (as we shall see)

9. Inputs = all ranges Harvesting from seed3: COCOMO-R • Sced=4: Time =160% of schedule • Pcap=4: programmer capability > 90th percentile • Pmat=2: CMM level 2 • Inputs = KC1= changes1 U now1 • Sced=2: Time = 100% of schedule • Acap=2: analyst capability  55th percentile

10. Qualitative constraint model of a circuit : 9 switches, 9 bulbs, 3 batteries Many unknowns: E.g. bulbs may be blown/ok Score each run by # shining light bulbs (max=9) Run I: Find top treatment T learnt from run I. T must be: Acceptable to users And Possible Constrain run I+1 with T 1 question culls 90% of options Runs=35228 Runs= 3264 Leave Sw2c open Runs= 648 And close Sw1c Runs= 32 Andclose Sw3c (not done) Harvesting from seed4: a qualitative model

11. Discussion • For coarse-grain decision making • Don’t fret what you don’t know • Simulate across the space of don’t knows • Actions = top ranked treatment(s) • Monitors = lowest ranked treatment(s) • To ask the fewest questions: • Explore issues in order of treatment ranking • Generality: • Average case analysis: use with care for safety critical sub-systems • Works in many domains • Assumes low-hanging fruit • Elsewhere: theoretically, empirically, many maybes mean (mostly) the same thing

12. Questions or comments? (Some random other slides follow)

13. Software • The TAR2 treatment learner • Visual C++/Windows • Fast: • Thousands of examples in a few seconds • Free: • If you tell us what you are doing with it • Simple: • Easy to use (good documentation) • But we’re happy to help • Dr. Tim Menzies, professor, UBC, tim@menzies.com • Ms. Ying Hu, masters student, UBC

14. Generality TAR2 works when domains generates a small number of outstandingly large deltaf values