Making a Simple, Structured and Efficient Testbench Step-by-step

1. Making a Simple, Structured and Efficient TestbenchStep-by-step Espen Tallaksen www.bitvis.no Making a simple, structured and efficient Testbench, Step-by-step

2. About Bitvis • Leading Vendor Independent Design Centre in Norway • FPGA and Embedded SW services for customers • From specification to final product – or any phase in between • Good overview of pitfalls, time wasters and risks • Focus on methodology, quality, efficiency and customers • Located in Asker outside Oslo Making a simple, structured and efficient Testbench, Step-by-step

3. Products and courses from Bitvis • Products from Bitvis • 'Bitvis Utility Library' (Free and Open source, Directlydownloadable)- Currentlybeing used worldwide • 'UVVM' (Universal VHDL VerificationMethodology) (UVL for VHDL)To be released 2014, Q4 • 'RegisterWizard', For generationof SW, Doc. and VHDL (bus IF, regs, etc.)To be released 2014, Q4 • Courses from Bitvis • 'FPGA Development Best Practices' - A twodaycourse- A pragmaticapproach to improvingquality and efficiency.- So far Denmark, Sweden and Norway. May be held anywhereonrequest • See ourwebsite for more offershttp://bitvis.no/services/fpga-courses/ Making a simple, structured and efficient Testbench, Step-by-step

4. Design stage Spec. Cost of corrections Ease of correction &debugging Volum/Field Why Testbenches and Simulation • Far more control and observability • Far faster iterations • May have a structured bottom-­up verification. • Detect bugs that cannot or most probably will not be detected in a lab-test • Most bugs can be found with short simulations. Making a simple, structured and efficient Testbench, Step-by-step

5. Verification: State ofthe FPGA community • Even for quite simple modules (e.g. IRQC, SPI, I2C, UART): • Average verification workload: 3 days to 3 weeks • Often inadequate coverage • Multiple Design, Synthesis, P&R iterations on full FPGA • A low quality testbench • Hopeless to understand for anyone else • Difficult to extend • Terrible to modify • Tedious debugging Why? How can we improve? Making a simple, structured and efficient Testbench, Step-by-step

6. The Verification Paradox • For proper simulation of FPGAs: • More lines of verification code than for design source • More time consuming than design • Yet - a far lower effort is spent on verification • Lower effort on partitioning • Lower effort on structuring • Lower effort on documentation A good TB is key to success: • Simple 500 MH project: Saves 50-100 MH • Complex 5000 MH project: Saves 500-1500 MH • Significantly improves TTM, LCC, Quality Making a simple, structured and efficient Testbench, Step-by-step

7. Different TB requirements (1) • The required level of verification differs a lot • Low verification complexity • single data path FPGA with some additional functionality • modules with no or very simple corner cases E.g. GPIO, simple IRQC, CRC, FEC, decoder... • Higher verification complexity:  Still needs basic infrastructure – as a platform Need more advanced verification solutions on top Low verification complexity  Simple TestbenchBut – how simple? Making a simple, structured and efficient Testbench, Step-by-step

8. How simple should a TB be? • Understanding somebody’s design module is often very hard • Understanding somebody’s testbench is normally impossible • And modifying it… ? Testbench Issues: • Purpose: To verify DUT requirements • Focus: Sufficient functional coverage with a minimum effort • Test Sequencer Requirements:  Simple to write  Simple to understand and modify - by anyone  Simple to execute, debug and understand reports & results Making a simple, structured and efficient Testbench, Step-by-step

9. The development stages • Ideally: - Specify  Design  Simulate  Synthesize  Test- Verification should start as early as possible • But: Specification changes all the time…  Design must be extended or modified  Testbench or test cases must be extended or modified  Simulations must be re-executed  and the result must be checked again • TB Readability, Flexibility and Extendibility is important • Simulation result checking and readability is important Making a simple, structured and efficient Testbench, Step-by-step

10. Verification stages • Will go through, stage-by-stage • Make a verification specification – but avoid overhead and aim for single source • Define your TB architecture and concept- but consider what is already available • Implement TB architectureIf available – start TB implementation from a template • Implement test cases- In sensible steps and starting ASAP But also focus on: • Simulation execution, Reporting, Debugging, Results • Iterations Making a simple, structured and efficient Testbench, Step-by-step

11. IRR(0) IRQC IRQC irq_source(0) IPR(0) clk clk arst arst igr irq2cpu IPR(1..n) ITR(0) (to CPU) bus interface bus interface ICR(0) /n /n irq_source(n) irq_source(n) irq2cpu irq2cpu IER(0) irq2cpu_ack irq2cpu_ack (Handling of one single irq_source.To be repeated for all sources.) S D Q Q D Q R R IRQ2CPU_ALLOWED IRQ2CPU_ENA Note: Uppercase names indicate software accessible registers (or dummy registers) irq2cpu_ack (from CPU) The IRQC Businterface Making a simple, structured and efficient Testbench, Step-by-step

12. IRR(n) irq_source(n) IPR(n) igr ITR(n) irq2cpu D S Q Q ICR(n) D Q R R IER(0) IRQ2CPU_ENA IRQ2CPU_ALLOWED irq2cpu_ack 1: Make a verification specification • A verification specification is always needed, but • The verification spec. should not be too extensive/detailed • Required details could be added in a later spec. iteration • A separate verification spec. document is normally not needed Verification specification IRQC: • Check defaults on output ports • Check register defaults and access (write + read) • Check register trigger/clear mechanism • Check interrupt sources, IER, IPR and irq2cpu • Check autonomy for all interrupts • Check irq acknowledge and re-enable Making a simple, structured and efficient Testbench, Step-by-step

13. 2: Define your TB architecture • Two main categories for almost all modules- Defined by degree of interaction on interfaces 1. Basic TB infrastructure only 2. TB for simultaneous handling of multiple interfaces • IRQC has very simple interfaces • Very little and no critical interaction • No contention issues • Still – A bad design may always fail for corner cases • Hence IRQC TB Simple Verification Simple TB arch./concept  Basic TB infrastructure Making a simple, structured and efficient Testbench, Step-by-step

14. TB_IRQC DUT (IRQC) Test sequencer Support processes Supportprocedures & functions Simple TB architecture • Avoid all complex issues • No support for handling simultaneous interfaces • No queuing of commands • Use a single, simple, understandable test sequencer • Required support • Logging • Alert handling & reports • Continuous actions • Actions not handled by sequencer • Min. interaction with sequencer • Repeated actions • Avoid unstructured copying • Provide more info for multi-usage Making a simple, structured and efficient Testbench, Step-by-step

15. TB_IRQC DUT (IRQC) Test sequencer Support processes Supportprocedures & functions 3: Implement TB architecture Ready to implement the first tests TB workload so far: 10-30 min. *** Inside TB architecture: ------------------------------------------------ -- Main process and test sequencer ------------------------------------------------ p_main: process begin log(ID_LOG_HDR, "Start simulation TB_IRQC"); clock_ena <= true; -- start clock generator *** *** Actual test sequence. To be filled in *** --========================================================== log(ID_LOG_HDR, "SIMULATION COMPLETED"); clock_ena <= false; -- to gracefully stop the simulation assert false report "End of simulation. (***Ignore this provoked failure.)" severity failure; wait; -- to stop completely end process p_main; Assuming no template is available… • Generate TB entity with DUT instantiated • Push-button in several tools (MsD, Emacs, etc.) • Add support process for clock generation • Allow enable/disable from seq. • Add test sequencer process • Main process in TB • Controls everything! • From TB initialization • To termination of simulation Making a simple, structured and efficient Testbench, Step-by-step

16. TB_IRQC DUT (IRQC) Test sequencer Support processes Supportprocedures & functions 4a: Implement First tests set_inputs_passive(VOID); apply_reset(VOID); log(ID_LOG_HDR, "Check defaults on output ports"); --------------------------------------------------------------- check_value(irq2cpu, '0', ERROR, "irq2cpu must be default inactive"); check_value(dout, x"00", ERROR, “dout must be default inactive"); Inside p_main process test sequencer process) • First tests are important • To verify compilation and elaboration OK • To see that our Testbench is up and running • To actually verify our first tests of the DUT • First test: Output defaults • Very simple test • Allows module to be integrated • First example on active support • Using support procedures The above code will result in the following log/transcript: BV: 0 ns irqc_tb All inputs set passive BV: 60 ns irqc_tb Pulsed reset-signal - active for 10T BV: BV: 60 ns irqc_tb Check defaults on output ports BV:---------------------------------------------------------------------- BV: 60 ns irqc_tb check_value(sl 0)=> OK. irq2cpu must be default inactive BV: 60 ns irqc_tb check_value(slv x00)=> OK. dout must be default inactive Progress information Detailed progress information. Only interesting initially and for debug… Making a simple, structured and efficient Testbench, Step-by-step

17. Verbosity control • Method to controlamountofinformation • To allowonlyselectedgroupsofmessages • Allowsreduction/increaseofinformation – withoutmodifyingthecode (comment/uncomment) • Makes it possible to get a different setofmessagesdependingoncurrentsimulationfocus - e.g.: • Debugging testcase or verificationcomponent • Debugging a specificinterfaceon DUT • Debugging a data flowthrough DUT • General simulation progress report. • E.g. Modelsimvsim options:-version Print the version of the simulator -quiet Do not report 'Loading...' messages +sdf_verbose Display SDF annotator status messagesverbosity switches turning on or off a given set of messages Making a simple, structured and efficient Testbench, Step-by-step

18. Priority based verbosity control • Every message has a given priority. • Typically 1 for a high priority message and 6 for a low priority message. • Every possible message from any part of the testbench has a given priority - normally defined when writing the message. E.g. log(2, "Packet header received”) • Verbosity level will determine which messages to show • E.g. a verbosity level of 2 means - Only priority 1 and priority 2 messages are let through (i.e. shown in the transcript or log); - whereas all other messages are blocked. • Messages could be defined anywhere • In Sequencer, BFMs, verification components, other processes, etc.. • SystemVerilog’s verbosity control is basically priority based Making a simple, structured and efficient Testbench, Step-by-step

19. ID based verbosity control • ID based verbosity allows log messages with different IDs to be shown or blocked depending on whether a given ID is enabled or disabled. • Every message has a given ID • Could be a string, a number, an enumerated, etc…. • There are no prioritiesE.g. ID=58 and ID=216 are not prioritised in any way • Example: log(216, "Packet header received”) • Verbosity control will determine which messages to show • Test sequencer may enable or disable message IDs dynamically depending on which messages are of interest. • SystemC’s verbosity control is ID based Making a simple, structured and efficient Testbench, Step-by-step

20. Verbosity Control – Using Enumerated • ID based verbosity control. • Enumerated IDs • A set of predefined IDs. E.g. ID_LOG_HDR or ID_BFM • Example: log(ID_BFM, "UART byte received") • Verbosity control • Enable an ID using enable_log_msg(ID_BFM) • Disable an ID using disable_log_msg(ID_BFM) • Only enable or disable from the test sequencer! Making a simple, structured and efficient Testbench, Step-by-step

21. Why ID based verbosity • Far more flexible and controllable • Real priorities are dependent on the situation • application • development stage • problem at hand • ID’s allows full control of verbosity • May still define numeric priorities locally • E.g. to indicate your assumed priorities • May then choose to use numeric priority or ID • ID-based verbosity is a superset of priority-based. Making a simple, structured and efficient Testbench, Step-by-step

22. The need for verbosity control Verbositycontrol is not required for the simplest TBs, but... • Provides an advantage - even for most really simple TBs • More efficient to usethe same methodology for simple, medium and advanced TBs • Most TBs normally end up more advancedthanassumed • Yieldsno overhead in eventhe simplest test sequencer Now – back to simple TB Making a simple, structured and efficient Testbench, Step-by-step

23. Back to TB: Do not waste time • Most designers are unstructured wrt. verification • Do not wait with your testbench and verification • Do not dive into your first test case with no structure • Do not use stupid testbench generators • Do not write stimuli sequences by pure assignments • Do not check results by checking outputs in the wave view • Identify the need for subprograms – ASAP • Immediately write subprogram when detecting repeated code • Write a “self checking” testbench • Requires good check subprograms • Good Comments/documentation is NEVER wasted • Provide in sequencer and in log/transcript Making a simple, structured and efficient Testbench, Step-by-step

24. IRR(n) irq_source(n) IPR(n) igr ITR(n) irq2cpu D S Q Q ICR(n) D Q R R IER(0) IRQ2CPU_ENA IRQ2CPU_ALLOWED irq2cpu_ack Identify the need for subprograms *** Inside p_main process (test sequencer process) log_hdr(“Reset"); --------------------------------------------------------------- set_inputs_passive(VOID); apply_reset(VOID); log_hdr("Check defaults on output ports"); --------------------------------------------------------------- check_value(irq2cpu, '0', ERROR, "irq2cpu must be default inactive"); check_value(pif_irqc_dout, X"00", ERROR, “pif_irqc_dout must be default inactive"); • Obvious subprogram candidates for IRQC • Register Access • Signal checkers • Interrupt source pulsing? • Interrupt acknowledge pulsing? • (Report/log method) • (Alert-handling) • (reset, set_passive, …) • Split in two categories • Local – IRQC-dedicated • Declare locally • Common – non dedicated • Declare in common package • Share with others Local procedure – dedicated to inputs Evaluate whether this should be a common, slightly more general procedure… Common procedures. Logging and checking is required in all testbenches. Making a simple, structured and efficient Testbench, Step-by-step

25. 4b: Next test: Register Access • First check register defaults • Then general access (Write + Read) • Provide Check-procedure – using Read • Make dedicated procedures • Will be common for many register interfaces  Bus access procedures are used to set up signal sequences to access internal registers  BFM : Bus Functional Model Making a simple, structured and efficient Testbench, Step-by-step

26. BFM / TLM Purpose • Handle transactions at a high level • E.g. Read, Write, Send packet, Config, etc • More understandable for anyone • Simpler code & Improved overview • Uniform style, method, sequence, result • Easy to add several very useful features Example: BFM for a CPU access to a module's registerE.g. write 0xF0 (“11110000”) into a register at address 0x22 (“100010”) cs <= ’1’;we <= ’1’;addr <= ”00100010”;data <= ”11110000”;wait until rising_edge(clk);wait until falling_edge(clk);cs <= ’0’;we <= ’0’; Replaced bywrite(x”22”, x”F0”); Making a simple, structured and efficient Testbench, Step-by-step

27. Major BFM quality differences • Inside a ”normal” BFM • Pure Read, Write or Check transaction • Additionally - Inside some BFMs • Syncronization of access to the relevant clock • Additionally - Inside good BFMs • Normalisation of inputs • Sanity-check on inputs • Configuration of behaviour • Logging of all accesses – with parameters and result • Severity control and alert handling • Verbosity control to potentially suppress log Making a simple, structured and efficient Testbench, Step-by-step

28. Making common, general BFMs function normalise( *** parameters (incl. implicitly constrained vectors) ) is *** Constants and Variables begin *** checks for legal constrained ranges *** check if (restriction = ALLOW_WIDER_SOURCE) then check_value(***Complex check and reporting***); *** corresponding checks for ALLOW_SHORTER_SOURCE, etc. *** Adapt value to target signal range and return end; • Value and Signal widths are initially unknown • Hence use unconstrained input vector for addr & data • But unconstrained vectors do not define direction/range e.g constant data_exp : in std_logic_vector; • Unconstrained data must often be “interpreted” • Comparisons, Bit-manipulation, etc… • Must Normalise • Must Check for sanity – for early problem detection Making a simple, structured and efficient Testbench, Step-by-step

29. Make and run first BFM - Summary As soon as register interface is implemented: • Make BFMs for write, read, check – in a common package • Then make simplified BFMs • First check defaults, and then write followed by check procedure sbi_write ( constant addr : in unsigned; constant data_in : in std_logic_vector; signal cs : inout std_logic; signal addr : inout unsigned; signal wr : inout std_logic; signal rd : inout std_logic; signal din : inout std_logic_vector; signal rdy : in std_logic; signal clk : in std_logic; constant clk_period : in time; constant msg : in string; constant scope : in string; *** more *** ) is begin normalise, sanity check, adapt, synch, write, wait?, alert?, log? end; procedure write ( constant addr : in unsigned; constant data_in : in std_logic_vector; constant msg : in string := "" ) is begin sbi_write(addr, data_in, pif_irqc_cs, pif_addr, pif_wr, pif_rd, pif_din, pif_rdy, clk, C_T_CLK, msg , C_SCOPE, ***more***); end; write(x"1", x"0F", "IER"); check(x"1", x"0F", error, "Pure readback"); Only if allowed by verbosity control ifcheck ok BV: 172 ns. irqc_tb SBI write(x1, x00000F) completed. IER BV: 192 ns. irqc_tb SBI check(x1, ==> x00000F) completed. BV: Pure readback ifchecknot ok BV: 172 ns. irqc_tb SBI write(x1, x00000F) completed. IER BV: BV:============================================================== BV: ERROR: BV: 192 ns. irqc_tb BV: value was: 'x00000E'. expected 'x00000F'. BV: (From SBI check(x1, x0F, Pure readback)) BV:============================================================== Not shown (or counted) if set to ignore Error Will break simulation if stop limit is reached Making a simple, structured and efficient Testbench, Step-by-step

30. TB: Workload so far? Note: Assuming a TB infrastructure for log/alert/checksOtherwise only slightly more, but far less log/report/debug) Making a simple, structured and efficient Testbench, Step-by-step

31. 4c: Continuing with TB for IRQC log(ID_LOG_HDR, "Check register trigger/clear mechanism", C_SCOPE); ------------------------------------------------------------ write(C_ADDR_ITR, x"AA", "ITR : Set interrupts"); check(C_ADDR_IRR, x"AA", ERROR, "IRR"); write(C_ADDR_ITR, x"55", "ITR : Set more interrupts"); check(C_ADDR_IRR, x"FF", ERROR, "IRR"); write(C_ADDR_ICR, x"71", "ICR : Clear interrupts"); check(C_ADDR_IRR, x"8E", ERROR, "IRR"); write(C_ADDR_ICR, x"85", "ICR : Clear interrupts"); check(C_ADDR_IRR, x"0A", ERROR, "IRR"); write(C_ADDR_ITR, x"55", "ITR : Set more interrupts"); check(C_ADDR_IRR, x"5F", ERROR, "IRR"); write(C_ADDR_ICR, x"5F", "ICR : Clear interrupts"); check(C_ADDR_IRR, x"00", ERROR, "IRR"); ***Etc… • Checking register interactions • Just a series of Write and Check procedures • Checking signal stability • E.g. interrupt to CPU stable on or off • Waiting for an event- inside a “window” • E.g. interrupt to occur wrt. activated source *** time_tmp is set at a time from which the interrupt should be stable check_stable(irq2cpu, (now - time_tmp), error, "No previous active irq2cpu"); await_value(irq2cpu, '1', 0 ns, C_T_CLK, ERROR, "IRQ2CPU expected by now"); Making a simple, structured and efficient Testbench, Step-by-step

32. Making local procedures Then evaluate whether this should be a common, slightly more general procedure… • Local procedures should be made for any repeated actions for which a common (central) procedure does not make sense • E.g. pulsing the interrupt acknowledge: procedure pulse_irq2cpu_ack( constant dummy : in t_void ) is variable initial_value : std_logic := irq2cpu_ack; begin check_value(std_match(irq2cpu_ack, '0'), tb_warning, “Test seq.”, "irq2cpu_ack='1' when pulse_irq2cpu_ack() called"); wait until falling_edge(clk); irq2cpu_ack <= '1'; wait until rising_edge(clk); wait for (C_T_CLK / 4); irq2cpu_ack <= '0'; log(ID_SEQUENCER_SUB, “Test seq.”, "Pulsed irq2cpu_ack from 0 to 1 to 0"); end; In addition to the actual signal toggling – several tasks should be added. Check that ack is not already active Synchronise pulse on and off Report the pulse to the log Dedicated msg ID Making a simple, structured and efficient Testbench, Step-by-step

33. 4d: Completing the Test case • Add missing sections acc. to verif. spec. • Fill inn all sections • Section headers  Log headers • Details inside sections  Std. log messages • End simulation by reporting all alerts Making a simple, structured and efficient Testbench, Step-by-step

34. Report summaries • report_alert_counters() • Reports all alert counters =========================================================================== BV: *** SUMMARY OF ALL ALERTS *** BV: ====================================================================== BV: REGARDED EXPECTED IGNORED Comment? BV: NOTE : 0 0 0 ok BV: TB_NOTE : 0 0 0 ok BV: WARNING : 0 0 0 ok BV: TB_WARNING : 0 0 0 ok BV: MANUAL_CHECK : 0 0 0 ok BV: ERROR : 0 0 0 ok BV: TB_ERROR : 0 0 0 ok BV: FAILURE : 0 0 0 ok BV: TB_FAILURE : 0 0 0 ok BV: ====================================================================== BV: >> No mismatch between counted and expected serious alerts BV: ====================================================================== Making a simple, structured and efficient Testbench, Step-by-step

35. Verbositycontrol – revisited Bitvis: 0.0 ns TB seq. Start Simulation of TB for IRQC Bitvis: -------------------------------------------------------------------------- Bitvis: Bitvis: 110.0 ns TB seq. Check defaults on output ports Bitvis: -------------------------------------------------------------------------- Bitvis: Bitvis: 110.0 ns TB seq. Check register defaults and access (write + read) Bitvis: -------------------------------------------------------------------------- Bitvis: Bitvis: 217.5 ns TB seq. Check register trigger/clear mechanism Bitvis: -------------------------------------------------------------------------- Bitvis: Bitvis: 337.5 ns TB seq. Check interrupt sources, IER, IPR and irq2cpu Bitvis: -------------------------------------------------------------------------- Bitvis: Bitvis: 637.5 ns TB seq. Check autonomy for all interrupts Bitvis: -------------------------------------------------------------------------- Bitvis: Bitvis: 2167.5 ns TB seq. Check irq acknowledge and re-enable Bitvis: -------------------------------------------------------------------------- Bitvis: Bitvis: 2367.5 ns TB seq. Check Reset Bitvis: -------------------------------------------------------------------------- Bitvis: Bitvis: ===================================================================== Bitvis: *** SUMMARY OF ALL ALERTS *** etc…… • Log-result for TB for IRQC • Only ID_LOG_HDR enabled: • 32 lines • Log headersonly • All IDs enabled: • 500 lines • All details • IRQC is an extremely simple module… Making a simple, structured and efficient Testbench, Step-by-step

36. Reduced design time • Early structured TB & test cases improve efficiency • Forces you to think about the spec. in a different way • Avoids wasting time on “simple/stupid” simulations • Do NOT make initial “force/check” verification • Yields a good TB & Test case structure - right away • and provides a “shell” for filling in tests as needed • Early verification of selected issues is useful • May add verification tasks as IRQC is being designed • No time wasted – as all verification is useful • Continuous TB update on spec/design-changes • Late changes are far simpler to handle Making a simple, structured and efficient Testbench, Step-by-step

37. Reduced debug time • Far better simulation debug support • Logging sequences and every single action • Possibility to reduce amount of logging • Good alert messages with mismatch report • And preceeding progress report • Far better quality on lab-releases • Minimised need for lab debug • Easier to make new test cases to trace lab problems Making a simple, structured and efficient Testbench, Step-by-step

38. Documentation • Single source for all documentation/commenting • May easily extract required documentation Making a simple, structured and efficient Testbench, Step-by-step

39. Bitvis Utility Library • Open source VHDL library – Released April 2013 • Initial version VHDL 2002/08 compliantonly • VHDL '93 versionreleased August 2013 • Importantfeatures • Logging and Verbositycontrol • Alert handling and reporting • Simple randomisation • Basic testbench checkingand awaitprocedures • Simplicity is key • Strongfocusonallowing real simple infrastructureusage • Project adaptablebehaviour and log layout • Quick Reference for all methods is provided • Advanced methodversionsallowaddedcomplexity Making a simple, structured and efficient Testbench, Step-by-step

40. Verification effort for a single module/FPGA Coverage Total of 4 hours for complete TB for IRQC... Structured approachusing an already available infrastructure 100% Structured approach Typical Project Approved No “wasted” time up frontStraight on coding Time (MH) Making a simple, structured and efficient Testbench, Step-by-step

41. Conclusions for my TB for IRQC • A uniform and structured methodology • More compact and understandable code • Single source for spec, code comments and logging • Far easier to write, read, modify and extend a test case • Easy to execute and understand output/status • Debugging is much faster with a good progress report • Total of 4 hours for complete TB for IRQC  Very efficient TB implementation and design debug Making a simple, structured and efficient Testbench, Step-by-step

42. Making a Simple, Structured and Efficient TestbenchStep-by-step The end. Making a simple, structured and efficient Testbench, Step-by-step