From technology to solutions: Customer Centric Testing

1. From technology to solutions: Customer Centric Testing @ Silicon India Software Testing Conference Hyderabad March 26 2011 Ramesh Loganathan VP (Products) & Center head, progress software- Hyderabad

2. Familiar..?

3. In next 30 minutes.. Software consumption patterns are shifting. Enterprises are now increasingly looking for solutions to their biz problems, And not just technology to address their software needs. The biz problems are also undergoing paradigm shifts with problems spanning departments within and outside the organizations. The talk will deliberate on some of these challenges and approaches possible to address them.

4. Enterprise scenarios today Continuously anticipate problematic events and re-prioritize to keep planes in the air and passengers happy Dynamically up-sell based on customer activity, context and location Investigate potentially fraudulent activity while it’s happening and prevent it from spreading Business Strives for Operational Responsiveness…

5. Constantly Increasing Pressures for Responsiveness Source: mail express fax e-mail Document transfer 100 ms 20 ms Algorithmic trading 20 min 30 sec Airline operations Responsiveness is the key to surviving the pace of competition 8 hr 10 sec Call center inquiries 1 day 5 min Track financial position 1 day 15 min Supply chain updates 3 days 1 min Phone activation 1 week 0.5 hour Refresh data warehouse 5 days 2 hrs. Trade settlement 4 weeks 1 day Build-to-order PC Typical Business SLAs weeks days hours minutes seconds

6. At this speeds, one cant operate looking at the rear-view

7. Cannot afford not being prepared or quick What is needed: Operational Responsiveness Visibility Response

8. Enterprises need to be Operationally Responsiveness Real-time Business Visibility Immediate Sense-and-Respond Continuous Business Process Improvement Continuously improve your business’ Operational Responsiveness

9. Challenge: Can’t Rip/Replace Critical Business Applications Business improvement in mission-critical systems Changing the engines on an airliner while in flight =

10. Enterprises massively distributed today:Example- The Transportation & Logistics Value Chain Customer Settlement Event & Information Network Claims Reconciliation Invoicing Engines Carrier Determination Document Mgt Planned Tenders HRMS Contract Services MRO Project Mgt Warehouse Mgt Customer Service Crew Mgt Asset Mgt Domestic Goods Mgt Carrier Optimization International Collections Warehousing & Distribution Billing Trade Services Intermodal Services Routing Planning & Dispatching Sales & Outbound Processing

11. Typical BPM Processes & Integration everywhere Existing Process Shadow Process Shadow Process Non-intrusive monitoring of existing processes Does not require re-automation of process

12. Need to Continuously track & optimize Automate Exception Management Workstep Warehouse status SAP SAP shipping Oracle Siebel Product catalog Logistics ERP CRM

13. Event driven enterprise solutions Examples: Fraud detection Cross-promotion

14. Aggravated by the cloud Customer www Settlement Claims Reconciliation Invoicing Engines Business processes that span people, systems and departments Carrier Determination Document Mgt Planned Tenders HRMS Contract Services MRO Project Mgt Warehouse Mgt Customer Service Crew Mgt Asset Mgt Domestic Goods Mgt Carrier Optimization International Collections Warehousing & Distribution Billing Trade Services Intermodal Services Routing Planning & Dispatching Sales & Outbound Processing Business Process

15. Users access on www enterprise intranet Amazon EC2 cloud S3 s3 ! 1 5 5 4 5 3 5 5 2 hosted SaaS Google App engine events virtualization platform Web based Admin www virtualization SaaS UI UI UI UI UI UI services BAM services services services services services app app app app Event Engine app app PaaS … The SOA enabled cloud- services everywhere! www SOA. Web2.0. Dist Management Managed; Clear SLAs; SOA; Scaleable; Virtualized; Dynamic provisioning; Location transparency Serious challenge testing in this environment…

16. What does this mean for Testing & Quality Assurance?

17. Must be very reliable.. Out of the doorElse, with poor reliability can never ensure Operational Responsiveness Not this kind of reliability though…

18. Customer centric testing • Test, Keeping customer more in focus • Testing biz solutions as opposed to technology • New challenges to the software quality assurance process. • Software QA organizations must be much more aware of these usecases and not just the functionality defined in the functional specifications or the technology standards. • Devise approaches that bring customer environments inhouse • into the automated testing suites & processes. • Must stress and regress functionality. in more complex endues scenarios.

19. Beyond solution stacks.. Distributed solutions’ testing Challenges: • Multiple layers & distributed • Data Driven Testing (No GUI) • Difficulty in isolating bugs • Lack of effective reporting and debugging capabilities • Integration & interoperability • Involvement of 3rd party services • Different authorization / authentication approach • Performance testing problems

20. Stress the new features in an integrated enterprise • Software Systems have to be very robust • Reliable and work as designed • Under stress and varied use conditions

21. Testing in & for the cloud • Testing a solution in the context of a cloud • Integration to apps in the cloud • Testing solutions running in the cloud • Acceptance testing of solutions procured OFF the cloud (SaaS) • Testing solutions in the context of massive integration • Boundary conditions become extreme

22. Release cycles now more frequentTesting for rapid releases (Web2.0 & SaaS style) • Continuous integration is a trend that is rapidly being adopted in testing • Include build automation, auto-deployment of the build into a production-like environment, and ensuring a self-test mechanism such that at the very least, a minimal set of tests are run. • Leveraging virtualised test environments • As soon as code is checked in, a build workflow kicks in that compiles the code - deploys it on to a virtualised test environment, triggers a set of unit and functional tests on the test environment, and reports on the results. • VSTS takes the build workflow one step further, even before checkins • Performs the build before the check-in is finalised, allowing the check-in to be aborted if it would cause a break, or if it fails the tests. • And given historical code coverage data from test runs, the tool can identify which one of the several thousand test cases needs to be run when a new build comes out - significantly reducing the build validation time. • Failed builds and tests are found Immediately • The developer who checked in the offending code is probably still nearby and can quickly fix or roll back the change. • Go Agile! • With Agile getting more mainstream, what are the QA changes needed in integrating into Agile methodologies. In particular, with scrum, how would QA function? If QA is part of a scrum team, how will integration QA work

23. On-demand (virtualized) test infrastructure • Have access to lot of customer scenarios- real or assumed • Easy setup of environment • Accelerate set-up/tear down and restoration of complex virtual environments to a clean state, improving machine utilization • Developers can access • Eliminate no repro bugs by allowing developers to recreate complex environments easily • Better automation of customer environment testing • Improve quality by automating virtual machine provisioning, building deployment, and building verification testing in an integrated manner (details later)

24. Can we be pro-active? We can do better than this.. • Top-down, dev driven • Avert bugs, rather than test-find-fix • From definition & design thru implementation • Interesting research work in ensuring Dev Tools help

25. About Progress Software • A publicly traded (PRGS) $500M enterprise software company with a solid track record of growth with more than 25 years of profitability • More than 140,000 customers and 2,500 partners in more than 175 countries • More than $5 billion of Progress technology, partner applications and services are sold each year to thousands of companies • Invented both the Enterprise Service Bus and commercial Complex Event Processing

26. Progress Responsive Process Management Suite Financial Services Accelerators Communications Accelerators Travel & Logistics Accelerators Real-timeMarket Surveillance Revenue Assurance Smart Airlines Enterprise Integration Data Services

27. Why Progress?Progress Delivers Value for the Banking Industry Enable Business Strategy Trusted Partner Industry Expertise Lowest Cost of Ownership

28. Driving Your Business with Responsive Process Management (RPM)

29. Progress Success in Banking Progress Internal use only

31. Operational Responsiveness • Monitor and share continuous real time information to the appropriate service providers. 2 11 5 8 7 1 13 12 9 10 5 6 8 6 3 4 • Enable invoicing engines to simultaneously manage tariffs, claims, discounts, and equalization. • Customer Request for Outbound Service (via phone, email, EDI, fax) • Appointment for Pickup – Service Provider schedules pickup • Routing Assignment Scheduling (Equipment, crew, intermodal options, delivery windows…) • Intermodal Notification (Anticipate inbound schedule crew, equipment,… ) • Outbound Origination (Shipments in transit) • Location Updates with current location several times en route • Alert Intermodal Carrier - Delayed Inbound • Outbound Delivery to Intermodal Service- Reschedule Equipment and crews • Intermodal Outbound – Shipment in Transit • Location Updates with current location several times en route • Late Arrival- Shipment arrives late at customer. • Drop Off. Proof of Delivery (POD) received. • Reconciliation. Customer reports damaged goods and SLA violation. Continuous Process Improvement – automate, adapt and create new business processes

32. ?? •  For products automation is easy (as it is a necessity to reduce repetitive work). But how would we incentivize automation for services- especially turnkey projects (that may or may not have maintenance phase following dev)  • In multi project/SI environments, what has your experience been building QA/Automation frameworks that could be used across projects? • How do you factor in frequent handle UI changes when automating tests • Challenges in testing multi-channel UI (web/mobile/et al) • Do you think it will be practical to integrate testing & code-coverage on a nightly basis and help accelerate fixing the bugs (by identifying regressive checkins, that caused test failures)? • Cloud challenges? What shifts in QA thinking/test approaches would we need to test in the cloud? If there are multiple solutions in the cloud, how would integration testing work? Your experiences. (all) • 80s manual. 90s rich UI. 2000 web. 2010s- what?

33. The Transportation & Logistics Value Chain Customer Settlement Event & Information Network Claims Reconciliation Invoicing Engines Carrier Determination Document Mgt Planned Tenders HRMS Contract Services MRO Project Mgt Warehouse Mgt Customer Service Crew Mgt Asset Mgt Domestic Goods Mgt Carrier Optimization International Collections Warehousing & Distribution Billing Trade Services Intermodal Services Routing Planning & Dispatching Sales & Outbound Processing Real Time Business Visibility

34. Solution Stacks to be tested • Solutions often present as a stack or suite in a customer environments. • The possible number and complexity of usecases in these combined environments are much higher • The testing complexity thus significantly amplified.