SOA Testing – SISE Approach & Scelta Test Automation Framework

1. SOA Testing – SISE Approach & Scelta Test Automation Framework Sathyan S Nair Senior Technical Analyst December 13,2007

2. Agenda • Needs & Solutions • Focus Areas & Challenges • SOA Lifecycle & SISE Approach • Scelta – The Automation Framework • Key – Benefits • Future Research • Open Discussion

3. Needs & Solutions

4. Business & Testing Needs • Business Needs • Evolve IT Solutions • Avoid costly integration and re-engineering efforts • Testing Needs • How do we validate services for functional requirements? • How do we manage the complexity of testing – Integration of multiple services? • How do we validate shared services across multiple applications, as services change and evolve?

5. Address the Needs • How to address the needs? • Various stages of testing - Required to address function as well as interaction complexities • Testers to wear additional hats and follow an evolved approach • Automation Framework for multiple test execution cycles

6. Our Solution SISE Approach & Scelta – The Framework

7. Focus Areas & Challenges

8. How different is SOA Testing • Head-fewer services, without a UI • Services spread across different boundaries • Look beyond FT & PT – Integration, System & Workflow Testing • Requirements change impact – Across multiple application boundaries • Need for multiple end-to-end business process/workflow testing

9. Addressing Challenges

10. SOA Lifecycle – SISE Approach

11. SOA Lifecycle – SISE Approach

12. SISE Approach – Service/Component Testing • What we do? • Validation - Light-weight service Implementation • Scripts Development - Assertions and Boundary Conditions • “Good known data” based validation technique • Test Results • Ensure the proper functioning of individual web-methods/services

13. SISE Approach – Integration Testing • What we do? • Validation – Communication between different systems • Scripts Development – Messaging formats (namespace, SOAP compliance etc), Communication policies (parameters, return type etc), Security policies (Authentication, WS-Security etc) • “Good known data” based validation technique • Test Results • Different types of services across multiple systems seamlessly integrate and communicate with each other

14. SISE Approach – System Testing • What we do? • Validation – Communication between different systems • Scripts Development – Messaging formats (namespace, SOAP compliance etc), Communication policies (parameters, return type etc), Security policies (Authentication, WS-Security etc) • Mock services to handle – Asynchronous Messaging communications • Message Checkpoints – Route messages to appropriate services • Performance Monitors – Study the response times , uptimes etc • Helps the team with known limitations, re-engineer and hardware recommendations • Test Results • Different types of services across multiple systems seamlessly integrate and communicate with each other, under production env • Overall transactions response time calculation

15. SISE Approach – End-To-End Testing • What we do? • Validation – Complete business cases/workflow • Different types of business systems - Types of protocols include HTTP, HTTPS, SMTP, SOAP, XML-RPC, IRC etc. • Mock services , Messages checkpoints & performance monitors deployed for appropriate usage • Service change undergoes a complete end-to-end testing cycle • Study of response time delays and key performance/response issues • Test Results • Complete business workflow completes without any errors • System satisfies the requirements

16. Scelta – The Automation Framework

17. Test Automation Framework

18. Scelta – Input Data & Driver Components • Input Store • XML driven Test Cases • Business and technical users can create test cases • Elements and schema guidelines for file creation • Configuration file for implementation information • Workflow Manager • Test execution - User choice (Business/Integration/security cases etc) • Priority for test workflow/scenario/case • Test execution – Environment configuration • Policy Manager • Assign appropriate policy – Test cases

19. Scelta – Service Validation Components • Service Validation • Validate individual services/web-methods • Assertions for different test-conditions • Xpath query to fetch information • Response Manager • Manages request and directs response • Invokes mock services - For transactions with response delay time • Message Routers – To deliver responses appropriately • Logs & Reports • Centralized logs and reports collector • Customized reports based on customer requirements

20. Scelta – Performance Monitors & Metadata • Distributed Monitoring • Service level monitors and agents • Customized agent for collecting transactional information • Load Manager • Workflow driven load testing • Configurable load – Monitor performance of service • Run periodically - Ensure Web service stability • Service Metadata • Collects Historical information - Measure service reliability and quality • Study service behavior (up/down time, time since last failure, service reliability history etc)

21. Key Benefits – Approach & Framework • Multi-level validation–Defects fixes-Minimal effort & cost • Test Coverage - Different requirements validation • Minimal Defect leakage in production • Asynchronous test script execution • Service response facts , reliability and quality reports • Re-usable across ‘n’ # of test cycles

22. Future Research • Support security related aspects • Finding services – Addressing testing pain-points

23. Open Discussion

24. Thank YouReach Me:sathyan.sethumadhavan@csscorp.com