Yoav Hollander - yoav.hollander@gmail blog.foretellix

1. Verification Tribes Their habits and terminologies Yoav Hollander - yoav.hollander@gmail.comblog.foretellix.com

2. Agenda • The verification tribes • Terminologies and beliefs • Is some unification possible?

3. The verification tribes • There is no “one verification community” • Lots of tribes • Different terminologies • Different belief systems • I am originally from the HW verification tribe • Specifically, from the dynamic verification sub-tribe • But I am engaging in “verification tribe anthropology” • Examples of misunderstandings • DUT, HIL, Verification, VE, CDV, Monte-Carlo • The results of these misunderstandings are hilarious • Even in HVC HW module SoC / LL SW Servers ML IAS Mil/Aero Security Plant Urban Others

4. The current state of verification Cloud of misunderstandings Verification concepts and methodologies Verification areas HW module Dynamic: Directed, CDV, Scenario-based, MBT SoC / LL SW Formal ML IAS DUT / VE Specs Mil/Aero Modeling Generation / randomization Plant Servers Regression mgmt Checking Security Urban Coverage Failure clustering Others Execution platforms Coverage Auto-fill Repeatability

5. Some common confusions • Dynamic, formal, MBT • Simulation, simulator • Manual, automated • directed, random, CDV, scenario-based • Test vs. run, seed, repeatability • Fuzzing • Coverage auto-fill, CDG, MBT HW module SoC / LL SW Servers ML IAS Mil/Aero Security Plant Urban Others

6. Confusions related to hierarchical verification • Verification vs. validation • Performance bugs vs. functional bugs • DUT/VE boundary • Conceptual bugs • Root cause Landing-related system Level 1 Level 2 Cockpit Airplane Airport Level 3 Equipment Pilot Co-pilot Air-traffic Controller … Level 4 … Auto-pilot Level 5 GPS logic Navigation logic …

7. Beyond common terminology / concepts • I hope we can remove this cloud of confusion • But why stop there? • Is a Universal Verification Framework (UVF) possible? • I.e. a single tool-suite for verifying all areas

8. An ideal Universal Verification Framework HW module UVF tool-suite SoC / LL SW ML • Basic framework • Basic facilities: • Generation • Checking • Coverage • Modeling • Regression mgmt. • … • Methodology • Documentation • GUI • Extension mechanism IAS Mil/Aero Plant Servers Security Area-specific extensions Urban Others

9. So, how about this UVF idea? • This is a crazy idea • The areas are too different • It may not even be possible to create a verification framework for a single area – too many sub-areas • But an interesting crazy idea • Lots of reasons why it can’t happen • Lots of reasons why it should happen • We (the verification community) should consider • Perhaps we can at least move towards it HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

10. Moving towards UVF HW module Dynamic: Directed, CDV, Scenario-based, MBT Formal SoC / LL SW UVF ML DUT / VE IAS Specs Mil/Aero Modeling Generation / randomization Plant Regression mgmt Servers Checking Security Coverage 1: current state 2: common concepts / methodologies 3: Architecture and some modules 4: Universal Verification Framework Urban Failure clustering Others Execution platforms Coverage Auto-fill Progress Repeatability

11. Why UVF is probably impossible (for now) • The areas are too different • Risk inventing the umbrella / kite combo • There are perhaps better alternatives • Adding verification to a per-sub-area non-verification tool • Universal modeling languages / simulators • Various stand-alone verification-related point tools • Let’s see how different the areas are …

12. A short tour through the tribes • HW module • Cost of failure high, 50% of engineering effort is verification • Well-developed methodology, mainly using CDV • This is one area where there is a per-area framework HW module SoC / LL SW ML IAS VE “Banana” Verification plan Mil/Aero Coverage Verification Environment and tests Plant DMA module Servers Security Errors Urban Others

13. A short tour through the tribes • SoCs + low level SW • Less developed than HW module • Space is too big and undefined, not all is interesting • So need CDV + scenarios / use cases HW module SoC / LL SW ML DUT RAM CPU IAS Mil/Aero Bus Plant Ethernet controller video controller B video controller A camera controller Servers Security built-in display Urban testbench Others external display agent Ethernet agent

14. A short tour through the tribes • Machine Learning (ML) • This is “verifying a module implemented via ML” • E.g. classification (e.g. via neural networks), reaction • Hard to verify: black box, may change on-the-fly • Nobody has a good scheme HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

15. A short tour through the tribes • Intelligent Autonomous Systems (IAS) • Includes autonomous robots, autonomous vehicles, UAVs • “Infinite world” and hard to spec • The VE includes people • Lots of HW-in-the-loop / people-in-the-loop verification • Nobody has a good scheme HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

16. A short tour through the tribes • Mil/Aero • Includes airplanes, missile systems, … • Many layers of specs, danger of spec bugs • Simulation mainly for training (but also for verification) • Simulation often on multiple, different machines HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

17. A short tour through the tribes • Plant • Includes power plants, airports, hospitals and so on • People / human procedures are part of the DUT • Looking for functional / safety bugs and also for performance / optimality bugs HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

18. A short tour through the tribes • Servers • This includes high-availability multi-server systems (e.g. database systems) • Very hard to create controllable / repeatable verification environment, but some do HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

19. A short tour through the tribes • Security • Includes just the verification of security • Mainly vulnerability detection • A hot topic currently • Mainly “negative testing” • Mainly checking for just crashes / memory leaks • Lots of success with “fuzzing” (which is CDV-like) • Success in auto-filling of code/implementation coverage • SAGE, AFL, … HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

20. A short tour through the tribes • Urban • Includes transportation / infrastructure systems: Bridges, stoplight designs, parking designs, new road designs, … • A smaller system grafted into a bigger, complex system • Looking almost exclusively for performance / optimality bugs • Lots of other kinds of “social simulations” HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

21. A short tour through the tribes • Others • Includes everything not mentioned above – a huge range • OS, device drivers, compilers, medical equipment, communications equipment, non-autonomous cars, web pages, apps, banking SW, insurance SW, … HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

22. Why UVF may still happen • As verification becomes more important, UVF becomes more compelling HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

23. Maslow’s hierarchy of verification needs Need auto-check, Scenario definitions, coverage auto-fill / MBT, regression mgmt., projection to vPlan, parametric verification, repeatability, … Here UVF begins to make sense … Now we need coverage Randomize with constraints Better verification Automate the running Here you only care about APIs / mechanics of your sub-area Just run something, then debug

24. Consider Autonomous Vehicles • Currently verification much less than 50% (I assume) • Lots of work on safety engineering, not verification • But this could change quickly. Here is one scenario • AVs bring US fatal car accidents from ~100/day to ~20/day • A few times/day a lawyer can say “Your bug killed my client’s child” • So bugs become headlines, then coverage, … • So regulators require strict verification requirements • If that happens, AV verification may jump to above 50% • So will need much better verification • So UVF will become more attractive • Also, regulators may require uniform inter-industry safety norms

25. Why UVF may still happen • As verification becomes more important, UVF becomes more compelling • It succeeded in one area (HW modules) HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

26. Why UVF may still happen • As verification becomes more important, UVF becomes more compelling • It succeeded in one area (HW modules) • It can save a huge amount of repetitive effort HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

27. The area / facility matrix HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

28. Why UVF may still happen • As verification becomes more important, UVF becomes more compelling • It succeeded in one area (HW modules) • It can save a huge amount of repetitive effort • Bigger systems need multiple areas HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

29. Why UVF may still happen • As verification becomes more important, UVF becomes more compelling • It succeeded in one area (HW modules) • It can save a huge amount of repetitive effort • Bigger systems need multiple areas • Many of the differences are not absolute, but rather a matter of degree HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

30. Reconciling the differences between areas External inputs Faults Human behavior DUT • Positive vs. negative testing • Gen / check tradeoffs • Black-box / white-box • Performance / functionality verification • Coverage hole filling strategy • Very different execution platforms • Special modeling requirements HW module SoC / LL SW ML IAS Mil/Aero Plant Servers Security Urban Others

31. So, how far can we go? HW module Dynamic: Directed, CDV, Scenario-based, MBT Formal SoC / LL SW UVF ML DUT / VE IAS Specs Mil/Aero Modeling Generation / randomization Plant Regression mgmt Servers Checking Security Coverage 1: current state 2: common concepts / methodologies 3: Architecture and some modules 4: Universal Verification Framework Urban Failure clustering Others Execution platforms Coverage Auto-fill Progress Repeatability

32. Summary • We can progress towards a common terminology • You can contribute to better inter-tribe understanding! • See also blog.foretellix.com • UVF seems impossible • But the upside is too big to miss completely • So let’s try to advance it • Look for unified conceptual framework / methodology • Investigate “architecture + some modules” • Look for the next single-area framework • Other ideas? • Thank youyoav.hollander@gmail.com