Testing Mobile Apps

1. Testing Mobile Apps Joanne and Xidong

2. Content • Types of Application on Mobile • Browser Application and its testing strategy • Extra: current iOS and Android Automation Testing Software • Client Application and its testing strategy • Common Testing Technology

3. What is the focus of this topic? • To find an proper testing automation that significantly reduces the overall time needed to test each software release.

4. 3 Categorized Applications • Client Application: • Applications are installed on a mobile device and run on that device, may be written to look and feel like a native application for specific phone models. • Messaging Application: • The servers need to receive and respond to the specifications of SMS messages. • Browser Application: • Browser applications are server-based applications that can be accessed through a web browser via a URL from a mobile device.

5. Aspects need to consider for testing (in general) • The human languages (e.g., French, Kanji); • The locales (e.g., UK English, Australian, and US English), which affect things like formatting numbers and currency symbols; • Who pays for updates to be downloaded (users may be unwilling to pay to download updates OTA); • How the software is installed on the device (e.g., in terms of security permissions); and • The number of applications and versions you need to support in parallel.

6. Goals when testing • To ensure we deliver attractive, easy-to-use, working applications for as many users as practical. Lots of happy, frequent users help show our software is successful and useful. • To have justified confidence in the quality of our software. Ideally, most of the bugs would be found and fixed before the software is widely used. • Fast feedback to developers. Fast feedback helps them to fix the code while it is still “warm,” while they are still intimately familiar with it. • To quickly detect issues so they can be addressed. This is particularly relevant when the problem is related to external factors (e.g., an operator’s network configuration or a specific handset model).

7. Markup Language Application on Mobile • Application is written HTML and run by browser in general. • But it has revolution for many years…

8. Markup Language Application on Mobile (cont.) • Flash Lite is a system that enables devices like phones to make use of a subset of the capabilities offered by Adobe Flash. Flash Lite is used on some smartphones and feature phones to enable things like embedded video viewing on sites such as YouTube. • XHTML Mobile Profile (XHTML MP) is a hypertextual computer language standard designed specifically for mobile phones and other resource-constrained devices. • Comparison of XHTML Mobile profile and XHTML basic. http://mobiforge.com/designing/story/comparison-xhtml-mobile-profile-and-xhtml-basic

9. Testing Markup Language Application on Mobile • Strategy: Using a program or script since Markup languages are designed to be processed by programs (such as the web browser) rather than being directly interpreted by a human. • Problems: Cookies are sometimes intercepted by a gateway, or a proxy server, provided by the carrier. Automated tests may need to communicate across multiple distinct connections, in parallel, probably across several devices or computers, in order to trigger cookie sharing issues.

10. AJAX Mobile Application • Ajax (Asynchronous JavaScript and XML) is a group of interrelated web development techniques used on the client-side to create asynchronous web applications. With Ajax, web applications can send data to, and retrieve data from, a server asynchronously (in the background) without interfering with the display and behavior of the existing page.

11. Testing Strategy for AJAX Mobile Application • Be able to test rich UI interactions of the application running within a web browser. (e.g., for an email application new emails may arrive from time to time from the server without any action by the user.) • Testing how well an application copes with intermittent network connectivity can be convoluted and may require an intermediate device. (e.g., to intercept and modify network packets.) • Test specific features that are differentiated from applications running on desktops. (e.g. rotation events)

12. What is Client Application? • Applications are installed onto mobile wireless devices. • They are able to provide users with more functionality, better integration, and better performance than browser applications. • Client applications can be divided into two categories: portable applications and native applications. • Portable applications run in (or on) a virtual machine, such as the Java Virtual Machine. (e.g. Android) • Native applications are written to run directly on particular architecture or platform of the target devices. (e.g. iOS)

13. iOS Automation Testing • ios-driver • Test any IOS native, hybrid, or mobile web application using the Selenium / Webdriver API. IOS automation is as easy as automation for a browser, due to reuse of the well known API. • High level features : • Implements the JSON wire protocol • Runs on emulators and devices • Can be run as a regular node in a Selenium Grid • Can be used in an existing Selenium Grid to manage device / simulator farm • First class support for localized native apps ( 1 test works for all the localized versions ) • No need to change the source • No need to jailbreak the device • No ios-driver app to install on the phone

14. iOS Automation Testing (cont.) • Appiumis an open source test automation framework for use with native and hybrid mobile apps. It drives iOS and Android apps using the WebDriver JSON wire protocol.

15. Android Automation Testing • MonkeyRunner: • Written in Python; • Has APIs for connecting to a device, installing/uninstalling apps, running apps, taking screenshots, comparing images. • Robotium: • A framework provides functional, system and acceptance tests running across multiple activities; S • Supports Activities, Dialogs, Toasts, Menus and Context Menus, and it is integrated with Maven and Ant for running continuous integration tests.

16. Android Automation Testing (cont.) • Robolectric(taken a different path): • Uses shadow objects and runs the tests on a regular workstation/server JVM avoiding dexing, packaging, deploying and running the application tested on an emulator or a real device, reducing the time needed to perform tests. • LessPainful(a step further): • Offering a service for automated testing of applications on multiple real devices. The user uploads the application (*.apk) and a test file written in Cucumber, a business-readable DSL, selects the configuration of devices he wants the tests to run on, and then the tests are automatically executed and a report is generated. The list of devices supported include Garmin Asus, several HTCs, LGs, Samsung Galaxy, Sony Xperia, and Motorola Motodefy.

17. Techniques for Client Application • Automated Unit Tests • iOS application - unit testing framework is available on code.google.com (http://code.google.com/p/google-toolbox-for-mac/) that even supports basic UI testing (http://code.google.com/p/google-toolbox-for-mac/wiki/iPhoneUnitTesting). • Client applications are harder to automate • Ideally the test automation would include interaction with the native system events (e.g., for keyboard and other inputs) and the ability to query the GUI layer (e.g., to read the contents of a text box). Extra features could be provided to enhance our ability to test the application (e.g., where system calls can be intercepted and modified to force certain conditions to be triggered.)

18. Techniques for Client Application • Automated Unit Tests (cont.) • Example of Unit Tests • Running Unit Tests in Emulator • Challenging – capturing output, automating execution of the tests in emulator • Code Injection • Used to obtain information, e.g., statement coverage, timing data, etc., or it may include automated tests protected void testBasicAuthForEmptyStrings() { String authResult = BasicAuth.encode("", ""); assertEquals("EmptyStrings should result in 4 byte result", 4,authResult.length()); }

19. Techniques for Client Application • Code Injection (cont.) • Unique features: • Source code is not required for some forms of code injection, e.g., to add code coverage instrumentation. • The application’s source code does not need modifying • Injecting code requires: • The application, • The code to inject (which may be tightly coupled to the implementation details of the application), • A way to inject the code, and • Instructions on where and how to inject the code.

20. Techniques for Client Application • Test Tools for Client Applications • On-Device debugging • Not really a way to automate the testing • Emulators • Blackberry Emulator: supports automation • Command-line tools are used to enter commands, and to create screenshots • Challenges: • Driving the user-interface. • Interpreting screen-responses, including stitching together images for pages that require scrolling, pattern matching where the content varies from request to requests. • Latency in the device interaction

21. Common Testing Techniques • GUI-level Automation • Challenges (in terms of making the tests relatively reliable): • The need for high contrast, opaque, block colors for backgrounds, etc., to make images • easier to match for menus, etc. • Deciding whether to use image pattern-matching or OCR (Optical Character Recognition) . • Structuring test resources, e.g., you may need a set of images per device. • Image Stitching • Used to assemble a number of screenshots into a composite single image that represents the original page. • Useful when the page is too large to be displayed on a physical phone

22. Common Testing Techniques • Image Stitching (cont.) • Step 1: Image retrieval: • Step 2: Image alignment:

23. Common Testing Techniques • Image Stitching (cont.) • Step 3: Image subdivision and election: • Step 4: Image reconstruction:

24. Common Testing Techniques • Image Stitching (cont.) • Challenges of image matching: • Captured images may vary significantly from device model to model: Content varies across screen sizes and resolutions

25. Common Testing Techniques • Image Stitching (cont.) • Optical Character Recognition (OCR): to extract text from images • Encoding Data in Pixels: • Each character of ASCII: 8 bits =>HEX “0xNN” (N= 0~F) • E.g. “A” = 0x41, “a” = 0x61

26. Common Testing Techniques • “Contact Sheet” • multicolumn format of rows of small images: screenshots of results generated by automated “user” input • Using Transparency Masking:

27. Common Testing Techniques • “Contact Sheet” (cont.) • Using Transparency Masking:

28. Common Testing Techniques • “Contact Sheet” (cont.) • Assisting test automation with human decision making:

29. Common Testing Techniques • Model Based Testing (MBT) • used to test other software domains such as web sites, desktop applications, etc. • helps to automate longer-running automated tests • used MBT techniques to test a number of mobile wireless applications, including a search engine and client applications.

30. When to test manually • UI and rendering problems: • alignment issues, flickering, etc. • When the UI is changing frequently, test scripts tend to break and need repairing, requiring lots of work • When content is dynamic and hard to predict • When the cost of automating is likely to significantly exceed the costs of manual testing

31. How to improve Testability • Generally: • Provide a complete API to enable the test automation code to test “below” (without) the GUI. • In programming languages such as Java consider making methods and data protected. • Browser based application: • Add ID tags to the main elements the automated tests will interact with • Client Applications • Consider using high-contrast colors, visual markers, and even pixel-encoding to make GUI-based automation more reliable • SMS Applications • Provide a scriptable library or interface (e.g., available from Python) to make tests easier to create.

32. Reference • http://mobiforge.com/designing/story/comparison-xhtml-mobile-profile-and-xhtml-basic • http://www.mobileburn.com/definition.jsp?term=Flash+Lite • http://en.wikipedia.org/wiki/AJAX • http://ios-driver.github.io/ios-driver/index.html • http://appium.io • http://appium.io/getting-started.html#hacking-with-appium-ios • http://www.infoq.com/news/2011/09/Android-Testing