webMethods Testing at Corporate Express

1. webMethods Testing at Corporate Express Sonam Chauhan Corporate Express sonam.chauhan@ce.com.au

2. Overview • Introduction • Change Process • General Requirements for Testing • Old Testing Process • New Testing Process • Implementation Tips • Future Steps and Trends • References

3. Introduction • Constant pressure for change • Changes need to be tested for bugs • Testing has business impact • Good testing costs money • Bad testing costs more money! • Need to find the “Golden Mean”

4. Introduction • webMethods issues that relate to testing • Dev/Test/UAT/Live Builds  Much Variation • Due to IS’s multiple packages feature • Source Code Management  Rudimentary • Support planned for interfacing to SCMs • Complex Change Deployments  Risky • Partially addressed by wMDeployer • Testing Support  Absent • wM aren’t adding it

5. Development Change Process Requirements Specification Requirements Definition Changes ValidatedChanges Review and Testing Change Deployment (Dev Test  UAT  Live)

6. General Requirements for Testing • Like development, testing is an engineering skill • Allocate limited resources for max. benefit. • Black-box Integration testing for Regressions  Best return for your testing buck • High level tests simulate live use • Tests can be external to code • Easy to retrofit to existing codebase • Regression are the costliest bugs • All changes must “first do no harm” • This is not Unit Testing • Unit tests can’t assure system functionality • This is integration/system-level testing

7. General Requirements for Testing • Developers can be testers! • Commitment to quality • Make test development part of development process • Make reviews of the test code part of code review process • Automate testing • Automated Regression Testing Harness • Without automation, testing is tedious work • Automation increases productivity and efficiency • Permanently add hard-won insight to corporate memory • And replay it on demand • Essential for managing change in complex systems • Active testing catches problems early • Load testing often very important • Some bugs can only be "teased out" under load.

8. Old Testing Process • Implemented Circa 2001 • Mostly manual • Repository of test cases • Test case input submitted to system being tested • System output compared to stored test case outputs and “Diff” generated (image at right) • “Diff” manually inspected and signed off

9. Old Testing Process • Advantages • Thorough Inspection of Changes • A human eyeballs all differences • Disadvantages • Expensive • Due to tedium, testing is only done when required • Not Scalable • Testing time increases significantly as platform grows more complex • Limited Coverage • No load testing • Fixed issues are not re-tested • Errors • Quality of QA suffers as most testing is manual and tedious

10. New Testing ProcessThe ‘metatest’ Testing Architecture • Implemented Circa 2004 • Drivers: • Increasing Complexity • Need for Automation • ‘metatest’ Testing Architecture • Perl command line engine • Open source components • Controls test engines that execute parameterized test scripts • Supports multiple test engines, test parallelization (in addition to threading within tests), test retargeting and test load variation • Test scripts execute sequence of steps, with assertions made at end of each step. • Entire test runtime and test script database in SCM system (CVS) • Effort: 1 man-months (excluding test scripts) with one developer (me) • Enhanced over a year

11. New Testing ProcessSample Test Script – JMeter Test Engine • JMeter [4] test scripts are executed by JMeter instances controlled by metatest Load testing part of same test script Documents retrieved from TN database to test document transform assertions

12. New Testing Process • Advantages • Cheap • Open-source software, no new hardware cost • Completely test automation - no tedium • Scalable • No time increase with platform complexity - tests run in parallel • Coverage • Includes load testing • Can run selective tests, or run regression test an entire system in 5 minutes • No Human Errors • Quality of QA improves • New Benefits • Detailed test results, Test Driven Development, Continual testing • Disadvantages • Typically, only specific assertions are automatically proven • All differences not examined – this may be mitigated by using ‘full-document’ assertions regular expressions adjusting for variances

13. Automatic archival of detailed results Data useful for: Load and stress performance validation (eg: image on the right) [5] Capacity planning Trend Detection Analysing software component failure rates New analytical methods can be applied to stored data in future New Testing ProcessNew Benefit – Detailed Test Results

14. While not Unit Testing, our approach has several of the benefits of ‘classical TDD’: Tests and code concurrently developed Bugs replicated as tests and fixed Bugfix tests added to test database Simple tests Brief run time for tests Immediate testing feedback New Testing ProcessNew Benefit –Test Driven Development

15. Continual Testing Dev/Test/UAT tested nightly Test & UAT also load tested (Load imposed = 4 x production load) Results emailed to developer list so they see it first thing in the morning (see image on right) Benefits All platforms fully regression and load tested each night Differential testing – identical tests run on different systems each night. When the results differ, bugs are identified by comparing the two systems. In example at right, failures are only on ‘test’ in ‘load’ testing… The load imposed in load testing is updated to reflect growth. Thus, bottlenecks are identified before they hit production. New Testing ProcessNew Benefit – Continual Testing

16. Documentation very important Documentation crucial to getting developer buy-in ‘metatest’ architecture has 35 pages of user documentation (in addition to documentation for the tests themselves) When using open source components, expect bugs Interact with developers of the component Contribute if possible Document problems, link to Bugzilla DBs etc Version control everything to do with testing Test scripts Even test engines binaries Advantages: Branching/ Merging, new developer can be setup in minutes Leverage existing infrastructure Most software can simulate endpoints. For eg: WM C1 OnRamp can create a CommerceOne envelope from XML Get a domain expert to write your tests Implementation Tips

17. Future Steps and Trends • Plans to improve the ‘metatest’ architecture • Implementing other types of testing • Integration with WmDeployer • Integration with CECM Verification Tool [6] • Other Testing Resources in the WM space • Customware’s WmUnit [1] • Wmusers.com article on testing using JUnit [2] • Solstice Software [3]

18. Future Steps and Trends • webMethods Testing Support • WM response to group request for testing support: “Currently no plans exist add unit testing or test service generation services to the developer.” • However, WM’s IDE is moving to the Eclipse platform. This should open up interesting possibilities with Eclipse plug-ins. • Server support: Ideally, on throwing an particular exception, a WM server would save the pipeline in a testcase format for bug replication on development server. (WM tool does this already?)

19. References [1] Customware website: http://www.customware.net/ [2] wmUsers.com testing article: http://www.wmusers.com/ezine/2003feb_parvanitis_1.shtml [3] Solstice software website: http://www.solsticesoftware.com/ [4] JMeter website: http://jakarta.apache.org/jmeter [5] Article on analysis of JMeter logfiles in stress testing: http://wiki.apache.org/jakarta-jmeter/LogAnalysis [6] CECM change verification tool described in: http://www.wmusers.com/software/uploads/2820B2B_Change_Management_Presentation.ppt