1 / 26

Demonstrating and Testing the BML Compliance of BML Realizers

Demonstrating and Testing the BML Compliance of BML Realizers. Herwin van Welbergen, Yuyu Xu , Marcus Thiebaux, Wei-Wen Feng, Jingqiao Fu, Dennis Reidsma, Ari Shapiro. What is this talk about. Testing the adherence to the BML standard by realizers

taini
Download Presentation

Demonstrating and Testing the BML Compliance of BML Realizers

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Demonstrating and Testing the BML Compliance of BML Realizers Herwin van Welbergen, Yuyu Xu, Marcus Thiebaux, Wei-Wen Feng, Jingqiao Fu, Dennis Reidsma, Ari Shapiro

  2. What is this talk about • Testing the adherence to the BML standard by realizers • (and, by this, also testing for weaknesses in the standard) • Better software engineering practices for virtual humans • Daily automated testing (regression, acceptance, …)

  3. BML Realizer Interface:SAIBA Framework

  4. BML Realizer Interface: BML goes in

  5. BML Realizer Interface: Feedback comes out • bml1 starts (globaltime=10) • gaze1 starts (localtime=0) • gaze1 ready (localtime=1) • speech1 start (localtime=1) • … • speech1 end (locatime=10) • bml1 ends (globaltime=20)

  6. Why automatic testing of realizers? • BML Realizers are complex software components • They form the backbone of virtual human applications by research groups • Yet their testing has so far been limited to time consuming manual inspection of the execution of BML scripts

  7. Why automatic testing of realizers? • Automatic tests are automatic • No wasted developer time • Automatic tests can be run often • Early error detection

  8. How can we test a realizer? • A realizer provides standardized interface • BML goes in • Feedback comes out • Several realizers implement this interface • We can make automatic tests for this interface • These tests can be used for all realizers implementing the interface

  9. What can be tested? • Message flow and behavior execution • Adherence to time constraints • Error handling • Acceptance testing

  10. Typical test case • Send BML to a realizer, capture all feedback • Wait until the realizer has executed the BML • Verify assertions on the received feedback

  11. Example test case <bml id="bml1"> <speech id="speech1" start="6"> <text>Hey punk <sync id="s1" />what do ya want?</text> </speech> <head id="nod1" action="ROTATION" rotation="X" start="speech1:s1"/> </bml> Whatcan we assertabout the expected feedback? - All default sync points remain in order forallbehaviors - The headnodwill start within |ε| seconds of the start of “what” - Speech1 starts within |ε| seconds of t=6 - Therewillbe no exceptions or warnings - The block will end … …

  12. Example test case <bmlid="bml1"> <speech id="speech1" start="6"> <text>Hey punk <sync id="s1" />what do ya want?</text> </speech> <head id="nod1" action="ROTATION" rotation="X" start="speech1:s1"/> </bml> @Test public void testSpeechNodTimedToSync() { realizerPort.performBML(readTestFile("testspeech_nodtimedtosync.xml")); waitForBMLEndFeedback("bml1"); assertSyncsInOrder("bml1", "speech1", "start", "ready", "stroke_start", "stroke", "s1", "stroke_end", "relax", "end"); assertAllBMLSyncsInBMLOrder("bml1", "nod1"); assertBlockStartAndStopFeedbacks("bml1"); assertRelativeSyncTime("bml1", "speech1", "start", 6); assertLinkedSyncs("bml1", "speech1", "s1", "bml1", "nod1", "start"); assertNoExceptions(); assertNoWarnings(); }

  13. RealizerTester • A generic testing framework that can test all BML Realizers • Provides a set of tests + assertions • And functionality to easily author new tests • Utility methods (e.g. waitForBMLEndFeedback) • Custom assertions (e.g. assertLinkedSyncs) • Code and tests released on SF, together with a nice set of example movies to showcase various realizers

  14. Testing Elckerlyc • Using RealizerTester • Additional tests for Elckerlyc specific functionality • Regression testing • Acceptance testing • Running all tests takes some time (~3 min.) • This might discourage frequent testing by developers • => Run tests automatically on a continuous integration server • Notify developers that commit a build that fails tests • Keep track of test result history

  15. Testing Elckerlyc • RealizerTester does not help in identifying the exact location of errors • It tests a Realizer as a black box • Elckerlyc uses additional white box testing at a smaller granularity to identify the exact location of errors • >1000 unit + mid range tests, running in <10s • RealizerTester helps in identifying locations that require more unit testing • Regular visual inspection is still valuable!

  16. Testing in SmartBody • Visual regression testing • Record a baseline • Does it (approximately) look like the baseline? • Automatically check difference with baseline • If different • Bug: fix • New functionality: create new baseline

  17. Testing in SmartBody: example

  18. Testing in SmartBody • Visual regression testing: • checks whether the correct motion is generated rather than whether the correct signals are sent • is Realizer and character dependent • does not provide acceptance testing • RealizerTester is complementary to visual regression testing

  19. Testing SmartBody using RealizerTester • Dealing with BML versions • Took 1 day to implement • Found someunclarities in BML (e.g. feedback ordering) nowaddressed in 1.0 • Found some minor implementation issues in SmartBody • Did not find any interpretation differences in the constraintsatisfaction

  20. Observations: au 1

  21. Observations: gaze

  22. Observations: longer performance

  23. Observations • Designing test cases + assertions • Highlighted several cases in which the current BML specification lacked detail or was unclear • Setting up a video corpus • Highlighted some expressivity issues in the BML standard • Shows that several behaviors are executed in a semantically equivalent manner • Created a healthy competition between the developers • Motivated developers to move to better BML compliance

  24. Conclusions • The modularity proposed by SAIBA enables the reuse of testing functionality • Designing a generic testing framework and a shared video repository helps move the standard forward • BMLRealizerTester provides a starting point for a BML compliance test suite

  25. Furtherwork • XML format to author tests using BML and assertions expressed? • Add more realizers (PLEASE CONTACT US!) • Full BML compliance is not required • Support for feedback is required • Includenewestversion of BML standard discussed last Wednesday • Continue extendingourrepository of niceexamplemovies

  26. Thanks for your attention http://realizertester.sourceforge.net/ • contact: hvanwelbergen@techfak.uni-bielefeld.de

More Related