FLINT 2001 A Framework Approach for B2B Test Automation Danis Yadegar, CEO Arsin Corporation

1. FLINT 2001 A Framework Approach for B2B Test Automation Danis Yadegar, CEO Arsin Corporation

2. Outline • Corporate IT reliability challenges • Test automation framework • Taxonomy of B2B failure point • Practical framework implementation • Questions Company Confidential

3. B2B IT reliability challenges • Application control (ownership) is shared at the Corporate level • Creates need for objective metrics (and tools to generate them) • Drives demand for implementation certification—”points of trust” • Errors can have significant financial and PR effects • Mission-critical application mistakes increasingly make 6:00 PM news • “Chain-reaction” impact across multiple firms • Who caused it? • Who is responsible for fixing it? • Who is legally responsible for the consequences? Company Confidential

4. B2B IT reliability challenges • Complex technology platform • No single point of control—or even monitoring • Process & workflow standards new and immature • Corporate efficiency demands are increasing • Manual testing too slow, too inefficient, too incomplete • Rate of component change is increasing faster than testing efficiency • “Engineering” vs “Testing” dilemma Company Confidential

5. No economic slowdown here … “Software failures are highly visible and can have a direct and profound Impact on a company’s reputation and bottom line. Automated Software Quality (ASQ) tools are important weapons in the IT arsenal.” --IDC Group, 2001 Company Confidential

6. Fortune 500 Organization Application Server eMarketplace SCM OLAP CRM B2B Server Rosettanet PIPs Private Procs Public Procs eMarketplace Application B2B Int. Server Adaptor EAI Events & Services Framework Internet E-Biz Apps Business Portals Customer Services Marketing Buy-side Application ERP Sell-side Application Custom XML Protocols EDI VAN Private Exchange EDI Infrastructure Private Exchange Application B2B Int. Server Adaptor • Back-office packaged applications • Inter-application integration • Procurement specialization (B2B) • Outside the firewall extension (B2B) Inter-Enterprise solution evolution Company Confidential

7. 4.1 Brittle end-2-end reliability Internet RosettaNet Marketplace Document Hub EDI Private Exchange • Participants often flying blind: • Law of Unintended Consequences • And butterflies cause typhoons Upgrade Time ! • Where’s the court? Company Confidential

8. Unwired Office Planning Sales Call Home Purchasing Dept Commuting Testing multiple human interfaces Internet Mobile = Ears + Mouth Web Browsing = Eyes + Hands • Testing complicated by: • UI differences • Nuances in business rules • Automation starting point Company Confidential

9. B2B test challenges • Environment changes daily  “What worked yesterday may not tomorrow” • Automated regression test framework is required • Failures occur in a very complex environment • Transparency before, at, and after point of failure is required • Functionality often machine-to-machine • Focus on component, not GUI, testing is required • B2B creates N-trap scenarios • Centralized test management is required • Functionality may be company-specific • Separate test cases from test data is required • Addition of trading partners must not harm existing players • Entry point certification is required Company Confidential

10. Where the problems come from … • Implementation incompatibilities • from known or unknown sources • Protocol incompatibilities • often locked into 3rd party products • Document-level incompatibilities or errors • Incompatible standards, version levels, element values • Correct component behavior, but broken or inconsistent top-level business behavior • Human errors, outdated policies, procedures not followed • Component upgrades or patches • platform, network, middleware, ERP, finance • Operational process incompatibilities • either in the IT or functional department Company Confidential

11. Test automation framework Certification Services Test Automation B2B / EAI Integration J2EE components Solution Domains Enterprise Test Automation Platform (ETAP) Technologies: Web services ( UDDI, XML, WSDL etc.), J2EE, Data Analytics and Reporting Core Platform Company Confidential

12. Arsin Enterprise Test Automation Platform (ETAP) Company Confidential

13. B2B procurement scenario Buyer’s System Supplier’s System eMarketPlace Buyer Supplier New Demand (MRP/PR or Shortaget) Evaluate Buy Actions 1 Create RFQ Evaluate RFQs Change Demand (MRP/PR or Shortage) Compare Quotes Create Quote Create Order Compare Order 4 XML Integration over Internet XML Integration over Internet 2 Create Sales Order Create P.O. Open Order Open Order 5 Change P.O. 3 Create/Evaluate Order. Create/Evaluate Chg. Change Sales Order Goods Receipt 6 Close PO Ship Items. Delivery 7 Vendor Invoice Verification Shipment 8 Create/Change Item Create/Change Item Invoicing 9 Create/Change Approved Vendors Create/Change Approved Vendors Company Confidential

14. Taxonomy of B2B failure point • Platform level (Hardware, OS) • Network level (Public and Private) • Security (Identify, Access, Non-repudiation) • Document level (Format, Translation, Interpretation) • Application level (Process Integration, Application Response) • Business level (SLA’s, Legal Protection, Industry Perception) Traditional IT Ops ApplicationSupport Business Function &Legal Dept. Automation Frameworks must support needs of each group Company Confidential

15. Log Receiver TestAgent Message Queue Message Queue Receiver Receiver Executor Executor TC TC TC TC Business Transformation L:ogic BusinessRouting Logic eMarketplace Test framework architecture Browser Console • XML based test cases and assertion repository • Browser based test administration utility • Secured HTTP communication with the test Agents • Accepts test cases, parses and executes them • Queues component state impressions before sending them to manager Test Manager Console Request Handler Test Case Sender / Test Dispatcher Assert Manager Reads Writes Log files / Test Results Test Case Repository Send/Receive (HTTP(S)) Send Log file (HTTP(S)) Test Agent Test Agent RNIF Server RNIF Server B2B Server B2B Server Partner System Partner System Company Confidential

16. Component State Impressions (CSI) • A CSI is a description of a component’s state • Represented as a HashMap of name-pair values • Uniquely named, and are explicitly sequenced • Typed to support association with Assertor instances capable of validating • Uniquely associated with 1 Test Agent instance • HashMap elements for a XML validation may be as follows: • <property1>=Audit (Channel Name) • <property2>=8078 (Event ID) • assertor_type=0A1 • xml_dom=<string format of XML DOM object> Company Confidential

17. Test Manager • One instance / framework instance • Used to create and execute test cases • Coordinates distribution of test cases to agents, and directs agent assignments (HTTP-based communication used) • Archives results and events • Assertion Manager • Handles test suite/case registration • Creates TestCaseAssertor instances • Coordinates routing of Component State Impressions (CSI) from Agents to associated Assertors instance • Test Case Assertor • Evaluates CSIs against pre-defined test case assertions • Communicates results to the Test Manager Company Confidential

18. Test Agent • Responsible for test case preparation and execution • Executes test cases by making an appropriate method call(s) on the component/module • Manage message queue of Component State Impressions (CSI) sent to the TestManager for validation. • TestAgent may have an implementation-specific component to perform this function on behalf of the component. (e.g. Channel/Event Listeners, etc.) • Test Agent instances are needed for every component/system under test Company Confidential

19. Framework functional features • Test execution and management is centralized and controlled through a web-based interface • Test results are collected remotely and archived centrally with test execution details • Instrumented application code allows precise component state impression (CSI) capture—even when no exception is thrown • Data is captured at intermediate points between components—can be compared and analyzed • Full test case maintenance via central repository and web-based management utility • Supports both stateless processing & long-lived transactions Company Confidential

20. Deployment models • As a “Certification” platform • Private exchanges, marketplaces, trading hubs • As a B2B Standard / Protocols inter-operatibility and interface verification platform • RosettaNet, JCA Adaptor • As a specification based regression test bed • As a diagnostic aid • View of audit logs for traceability and debug purposes • As an outsourced, hosted regression test automation services • Periodic validation of 3rd party services integrated on a trading website Company Confidential

21. Impact on SQA and Operations • Application code must be instrumented (intrusive testing) • Agent has to be installed and plugged into the target systems to be tested. • Infrastructure specific agent configuration for communication • Setup of different regions ( development/test/ pre prod/ prod ) required • Minor overheads on the communication bandwidth (<5%) • Agent upgrade is required for system / technology upgrades Company Confidential

22. Certification Services Test Automation B2B / EAI Integration J2EE components Solution Domains Enterprise Test Automation Platform (ETAP) Technologies: Web services ( UDDI, XML, WSDL etc.), J2EE, Data Analytics and Reporting Core Platform ETAP evolution Health Level 7 Voice Components SWIFT / FIX Web Service Implementations Wireless Company Confidential

23. Next generation opportunities • Integration with monitoring tools • System monitoring (Platform Level) • Application monitoring (Application Server Level) • Decision support tools • Should this configuration be deployed? • Has the validation process been substantially followed? Company Confidential

24. Questions Company Confidential