1 / 25

Specification System for Multi-Platform Test Suite Configuration

Explore the new Dyninst test suite architecture and its features, status, and future plans. Learn about the idealized Dyninst Test, multiple platforms, compilers, languages, build parameters, and more. The new infrastructure simplifies writing test cases and automating the test process with detailed specifications.

jennamcreynolds
Download Presentation

Specification System for Multi-Platform Test Suite Configuration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey

  2. Talk Outline • Test suite challenges and basis for our new architecture • The specification system • A walk through the new Dyninst test suite architecture • Other features, status notes, and future plans

  3. Idealized Dyninst Test Dyninst mutatee mutator output • Mutator program • Uses Dyninst to analyze or modify mutatee • Mutatee program • Simply verify that modification occurred • Output • passed, failed, or skipped Simple, right? test

  4. Multiple platforms Architecture operating system operating system version 32 vs. 64 bit Multiple compilers Native GNU Other Multiple languages C C++ Fortran Multiple build parameters Optimization levels Stripped vs. unstripped Multiple test parameters Create vs. attach Multiple mutator-mutatee mappings Not That Simple And not all combinations are valid

  5. Our Goal • Write simple test cases: • Modify specification file • Don’t modify code when adding a test • Automatically do the rest • Build the test programs, with appropriate variations and parameters • Run the test with appropriate parameters • Collect results and insert into database mutator() { insertSnippet(…) . . . } foo() { . . . }

  6. New Infrastructure • Specification language and compiler • Specifications for almost all aspects of test suite • Test suite build system • Standard build system • Makefiles are output of spec. compiler • Test driver • Tests are plugins • List of tests to run is output of spec. compiler • Results parser and database User only needs to write test cases and test specifications

  7. Specifications • Languages • Test cases • Mutator sources • Mutatee sources • Mutator-mutatee mappings • Build configurations • Run configurations • Test suite glue • Standard libraries • Constraints • What? • Platforms • Architecture • Operating system • File name conventions • Compiler availability • Compilers • Language • Standard flags • Option-parameter string mapping • How? • We use simple tuples, written as Prolog clauses • Prolog lets us naturally express relationships between tuples

  8. Test Suite Architecture Test sources Build Stage Run Stage Data Collection Specifications

  9. Test Suite Architecture spec compiler Test sources Build Stage Run Stage Data Collection spec file

  10. Specification Compiler Tuples Scripts compiler spec mutator tuple standard libs Prolog Python test suite glue makefile mutatee tuple test tuple mutator spec mutatee spec test lists test spec Specifications

  11. Test Specification mutator source mutatee source valid compilers mutator-mutatee mapping build and run constraints mutator(‘example’, [‘example.C’]). mutatee(‘example’, [‘example_mutatee.c’]). compiler_for_mutatee(‘example’, Compiler) :- compiler_language(Compiler, ‘c’); compiler_language(Compiler, ‘c++’). mutator_mutatee(‘example’, ‘example’). test_runs_everywhere(‘example’). test_runmode(‘example’, ‘both’). test_start_state(‘example’, ‘stopped’).

  12. Test Suite Architecture spec compiler Test sources Build Stage Run Stage Data Collection spec file

  13. Test Suite Architecture mutator compiler mutatee compilers build scripts (makefiles) spec file spec compiler Test sources Run Stage Data Collection

  14. Build scripts • Specification compiler generates makefile • Mutator built as shared library plugin • example.so • Multiple mutatees built with build parameters encoded into names • example.mutatee_g++_none • example.mutatee_g++_low • . . .

  15. Test Suite Architecture mutator compiler mutatee compilers build scripts (makefiles) spec file spec compiler Test sources Run Stage Data Collection

  16. Test Suite Architecture mutator compiler mutatee1 mutatee2 mutatee3 mutatee compilers spec file spec compiler test driver src mutator .so Test sources Data Collection build scripts (makefiles) run scripts (test lists) Dyninst compiler test driver

  17. Run scripts • Prolog generates tests as tuples • (example, example.mutatee_gcc_low, createProcess, stopped) • Python script creates lists of tests that are compiled into the test driver • Test driver executes lists of tests and produces results output

  18. Test Suite Architecture mutator compiler mutatee1 mutatee2 mutatee3 mutatee compilers spec file spec compiler test driver src mutator .so Test sources Data Collection build scripts (makefiles) run scripts (test lists) Dyninst compiler test driver

  19. Test Suite Architecture parser mutator compiler mutatee3 mutatee2 mutatee1 mutatee compilers spec file spec compiler test driver src mutator .so Test sources output log build scripts (makefiles) database run scripts (test lists) Dyninst compiler test driver

  20. Collecting Output • Test output “example –mutatee example.mutatee_g++_low” All tests passed • Output is human readable • Added to database • (example, g++, low, createProcess): passed

  21. Flexible Testing Structures • Build each test as separate binary • High isolation, slow performance • Build several tests into a single binary • Balance test isolation vs. performance

  22. Not Just Tests • Specifications also used for • Platforms • Specify architecture, operating system, filename conventions, compiler availability* • Compilers • Specify languages*, option - command line parameter mapping, standard flags, platform availability* • Languages • Specify filename conventions, compilers*

  23. Example Platform % Linux 2.6 i386 specification % paragraph platform(‘i386’, ‘linux’,‘linux2.6’, ‘i386-unknown-linux2.6’). compiler_platform(‘gcc’, ‘i386-unknown-linux2.6’). object_suffix(‘i386-unknown-linux2.6’, ‘.o’). library_prefix(‘i386-unknown-linux2.6’, ‘lib’). library_suffix(‘i386-unknown-linux2.6’, ‘.so’).

  24. Example Compiler % g++ specification paragraph comp_lang(‘g++’, ‘c++’). compiler_platform(‘g++’, Platform) :- platform(_, OS, _, Platform), OS \= ‘windows’. compiler_optimization_translate(‘g++’, ‘high’, ‘-O2’). comp_std_flags(‘g++’, ‘$(CXXFLAGS)’). comp_mutatee_flags(‘g++’, ‘$(MUTATEE_CXXFLAGS_GNU)’). mutatee_link_options(‘g++’, ‘$(MUTATEE_LDFLAGS_GNU)’).

  25. Status • Infrastructure functional • Converted 2/3 of old tests; working on the rest • Will add SymtabAPI tests to new infrastructure • Tech. report in early summer • Future: integrate into NMI Build & Test system

More Related