1 / 16

Automatic Testing in Madagascar

Automatic Testing in Madagascar. Jim Jennings Houston, Texas July, 2010. Test Everything!. Outline. Madagascar testing purposes, goals, methods, and components Give it a try!. Madagascar Testing Purposes. Regression testing for developers Regression testing for users

dirk
Download Presentation

Automatic Testing in Madagascar

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Automatic Testing in Madagascar Jim JenningsHouston, TexasJuly, 2010

  2. Test Everything!

  3. Outline • Madagascar testing purposes, goals, methods, and components • Give it a try!

  4. Madagascar Testing Purposes • Regression testing for developers • Regression testing for users • Reproducibility testing • Data flow testing • Computational example comparisons

  5. Madagascar Testing Goals • Make it convenient • Make it automated • Make it flexible • Make it usable

  6. Madagascar Testing Goals • Make it convenient • Make it automated • Make it flexible • Make it usable • Make it fun!

  7. Madagascar Built-in Test Base • Over 300 computational examples in $RSFSRC/book • Over 3,000 figures in a separate repository, https://rsf.svn.sourceforge.net/svnroot/rsf/figs • The basic Madagascar testing method is to run the examples and compare the resulting figures to those downloaded from the repository.

  8. … But it’s not as simple as it sounds • Some of the examples require external data. • Some of the examples require components that might not be installed. • Some of the examples take a long time to run. • Some of the examples use a lot of disk space. • Round off differences make some examples not exactly reproducible.

  9. … But it’s not as simple as it sounds • Some of the examples require external data. • Some of the examples require components that might not be installed. • Some of the examples take a long time to run. • Some of the examples use a lot of disk space. • Round off differences make some examples not exactly reproducible. • We need an easy user-adjustable way to select examples for routine testing.

  10. Madagascar Examples Metadata • Metadata stored for each example in an .rsfproj file: • Madagascar tools used • External data used • Total disk space used • Metadata for $RSFSRC/book examples stored in repository. • Metadata generated automatically as part of “scons lock” operation. • You can store metadata for your own examples.

  11. Madagascar Testing Components • sfbooklist • Make an inventory of examples • Select examples according to optional filters • Run an optional command for each selected example • sffiglist • Make an inventory of figures in one example • List and display figures that are different or missing • sftestlist • Make a summary of test results • $RSFSRC/admin/quick_test.csh • Do everything for a pre-selected collection of examples

  12. sfbooklist Try these in $RSFSRC: > sfbooklist book • Make an inventory of $RSFSRC/book > sfbooklist levels=2 book/geostats • Make an inventory of $RSFSRC/book/geostats > sfbooklist levels=2 command=ls book/geostats • Run a simple command in each $RSFSRC/book/geostats/*/* > sfbooklist levels=2 command=scons book/geostats • Build each $RSFSRC/book/geostats/*/* example

  13. sffiglist Try these in $RSFSRC/book/geostats/simulate/rfield: > sffiglist list=all • Inventory of all figures > sffiglist list=miss • List only missing or different figures --- Delete a figure and try it again --- Change something in SConstruct and try it again > sffiglist list=miss show=miss • Show the different figures > sffiglist list=miss rsftest=y • Save test results in an .rsftest file

  14. sftestlist Try these in $RSFSRC: > sfbooklist levels=2 command="sffiglist rsftest=y" \book/geostats • Compare figures in each $RSFSRC/book/geostats/*/* example > sftestlist book • Make summary of tests in $RSFSRC/book/*/*/* > sftestlist levels=2 book/geostats • Make summary of tests in $RSFSRC/book/geostats/*/*

  15. $RSFSRC/admin/quick_test.csh Try this in $RSFSRC: > admin/quick_test.csh

  16. Test Everything!

More Related