CS454 : Software Engineering-3 Mobile Software Engineering Testing Mobile Applications

1. CS454: Software Engineering-3 Mobile Software EngineeringTesting Mobile Applications

2. Introduction This unit provides basic knowledge about mobile applications testing. Differences between mobile applications and traditional applications in terms of testing techniques. Challenges in testing mobile applications. Role of automation in testing mobile applications. Software Engineering in Mobile Computing

3. Software Engineering in Mobile Computing Mobile Applications vs Traditional Ones Limited resources Security and vulnerability Performance and reliability variability Finite energy source Context-Awareness Location-Awareness

4. Software Engineering in Mobile Computing Types of Mobile Applications

5. Software Engineering in Mobile Computing Types of Mobile Applications • App4Mobile is an application that, driven by user inputs, runs on a mobile device with limited resources. • MobileApp is a particular App4Mobile that inputs data from the (potentially evolving) environment surrounding the mobile device and/or from user actions, producing context-based outputs.

6. Software Engineering in Mobile Computing Software Testing Implications

7. Software Engineering in Mobile Computing Mobile Connectivity • Mobile applications typically connect to mobile networks, that may vary in speed, security, and reliability. • Implications on Testing: • Application reliability, performance, security, and correct functioning strongly relies on the connectivity type (Wi-Fi, 3G, bluetooth, etc...) • Functional testing has to be performed in different connectivity scenarios and networks

8. Software Engineering in Mobile Computing Limited Resources • Implications on Testing: • mobile applications resource usage has to be continuously monitored to avoid performance degradation and incorrect system functioning • Actions have to be taken in case of resource shortage

9. Software Engineering in Mobile Computing Autonomy • Mobile devices can never rely on an electrical outlet for functioning. • Different mobile applications may require different energy consumption. • Implications on Testing: • Energy consumption of mobile applications can be evaluated through testing or continuous monitoring.

10. Software Engineering in Mobile Computing User Interface • Implications on Testing: Testing different • screen sizes/resolutions • screen orientations (portrait, landscape) • Usability testing • follow design patterns • difficulty with the input mechanisms • difficulty with scrolling, gestures, etc… • confusing wording of the prompts • less is more: check for useless navigation elements • long download times

11. Software Engineering in Mobile Computing User Interface • Test whether different devices provide an adequate rendering of data • Test whether native applications are correctly displayed on different devices

12. Software Engineering in Mobile Computing Context-Aware Computing An application is aware of the computing environment in which it runs, and adapts/reacts according to its computing, user, physical, or time context Challenges in context-aware computing contextual sensing (e.g. location, time, nearby objects, orientation, social contexts), contextual adaptation/reconfiguration context-triggered actions contextual resource discovery

13. Software Engineering in Mobile Computing Context Awareness • Implications on Testing: • Testing whether the application is going to correctly work on any environment and under any contextual input is a challenge and may lead to combinatorial explosion. • Context-specific test selection techniques and coverage criteria have to be produced

14. Software Engineering in Mobile Computing Adaptation • Mobile apps may adapt and evolve driven by contextual information. • Such an evolution may happen during the application operation and without stopping the service. • Implications on Testing: • approaches for testing the application correct adaptation/evolution have to be developed.

15. Software Engineering in Mobile Computing New Mobile Programming Languages • Implications on Testing: • Traditional structural testing techniques (White box testing and associated coverage criteria) need to be revised in order to be applied to new mobile languages. • Bytecode analysis tools and functional (blackbox) testing techniques may be required to deal with binary code.

16. Software Engineering in Mobile Computing New Mobile Operating Systems • Mobile applications run on top of new operating systems that are still only partially reliable. • Bugs can be generated due to OSs unreliability and variability. • New versions of mobile operating systems are frequently released, not always backward compatible with previous versions. • Implications on Testing: • Testing mobile applications for failures that do not result from the mobile app itself

17. Software Engineering in Mobile Computing Diversity of Phone Manufacturers • Mobile applications may behave differently due to variations in the hardware or O.S. components. • Sensors are typically calibrated differently, so that two (different) phones running the same application, in the same environment, may compute different outputs. • Implications on Testing: • testing techniques for maximizing the diversity coverage while minimizing the devices under test

18. Software Engineering in Mobile Computing Touch Screens • System response time to a touch strongly depends on the device resource utilization, and may become slow in certain circumstances (e.g., busy processor). • Implications on Testing: • test the touch screen functioning under différent circumstances (e.g., resource usages, processor load, memory load) and within different mobile devices.

19. Software Engineering in Mobile Computing Testing Process and Challenges • Testing process (test selection and execution) • Testing scopes (structural and functional) • Testing levels (unit, integration, regression) • Special types of testing

20. Software Engineering in Mobile Computing Testing Process – Test Cases Selection • MobileApps are expected to receive inputs from different context providers (i.e., users, sensors and connectivity devices). • New testing criteria by which mobile test cases are selected to maximize coverage • Models of contextual scenarios may be used to drive the systematic selection and coverage of MobileApps • E.g., testing a tracking application which uses GPS data, the test suite can include parametric inputs associated to the GPS input scenarios, e.g., the entrance/exit to/from a gallery

21. Software Engineering in Mobile Computing Testing Process – Test Cases Execution • How to execute test cases including rich contextual inputs • Simulators are unable to simulate real-world phones with their sensors, GPS, connectivity • New capture-and-replay automated techniques can be realized for executing contextual inputs selected during the test selection phase • Android Robotiumis an open-source tool for automated and black box test execution of third parties applications • Android Robotium resembles Selenium for Android applications

22. Software Engineering in Mobile Computing Testing Artifact – Structural • Structural testing is known as code-based testing • mobile application languages add specific constructs for managing mobility, sensing, and energy consumption which require testing at code level • In case the source code is not available, new bytecode analysis tools need to be realized

23. Software Engineering in Mobile Computing Testing Artifact – Functional • Functional testing is known as model-based testing • Functional testing requires to specify both the mobile application and the environment it may operate in • State-based approaches: • specify the different states a mobile application may reach when different sensed data are received • model different execution modes (e.g., low battery, meeting, and airoplane mode) in which the system can be • Android Robotium, MonkeyRunner run an automated functional test for Android applications

24. Software Engineering in Mobile Computing Testing Level – Unit Testing • Realize automated testing tools which enables the specification of contextual values and context-based coverage in unit testing • Android enables the use of JUnit for unit testing.

25. Software Engineering in Mobile Computing Testing Level – Integration Testing • Most of current testing approaches consider mobile applications in isolation, • Need to consider inter-application communication via intents and content providers

26. Software Engineering in Mobile Computing Special Types of Testing • Performance and reliability • depend on mobile device resources; the device operational mode; the connectivity quality and variability; and other contextual information. • Run-time analysis techniques can be adopted to monitor resources and connectivity state; and prevent performance degradation.

27. Software Engineering in Mobile Computing Special Types of Testing • Memory leaks and energy consumption • Memory leaks may pre-empt the (limited) memory resource • Active processes (of killed applications) may reduce the device (battery) autonomy • Metrics for estimating memory and energy consumption could be devised to automatically predict memory leaks and energy outage

28. Software Engineering in Mobile Computing Special Types of Testing • Stress testing and scalability issues • For the server side portion of the mobile application, tools can be used to simulate connected users. • For the mobileportion of the application, if the mobile application allows a background process, the mobile application should be tested as some other background process is consuming a great portion of the resources on the device. • Test how the application, on both server and the mobile device, reacts when the various parts of the system fail • Make sure that the application recovers, in a very graceful manner, from any failures and hangs

29. Software Engineering in Mobile Computing Special Types of Testing • Interrupt testing • test against interrupts: you can manually trigger lots of notifications to the device while using the app • notifications can be incoming messages, calls, app updates, or push notifications (software interrupts) • pressing the volume up and down buttons or any other kind of hardware button is also an interrupt (hardware interrupt)

30. Software Engineering in Mobile Computing Special Types of Testing • Battery Usage testing • Fully Charged Battery testing scenarios • Low Battery testing scenarios • Standby testing • Installation Testing • Update Testing • Database Testing • Local Storage Testing

31. Software Engineering in Mobile Computing Special Types of Testing • Security testing • Cache storage: Sensitive data such as passwords or tokens is cached on the device. • Unintended local storage: Sensitive data such as passwords, tokens, or credit card details is stored by accident. • Encryption: Sensitive data such as passwords is not encrypted on the device storage. • Client-side validation: Password verification is performed only on the client side. • Unencrypted communication: Communication from the app to the backend systems is not encrypted. • Unnecessary app permissions: Apps use permissions for device features that they do not need or use.

32. Software Engineering in Mobile Computing Special Types of Testing • Accessibility testing • Visual Impairment ,e.g., color identification difficulty • Auditory Impairment, e.g., sound identification difficulty • Physical Impairment, e.g., gestures manipulation difficulty • Cognitive Impairment • problems related to attention, communication, memory, computation, and reading skills

33. Software Engineering in Mobile Computing Cost vs Reliability of Testing • Cost of testing: mobile applications must be cheap, and cost much less than traditional applications • Reliability of testing: testing process must produce reliable products • Automation is certainly among the most important means for keeping the cost of testing low, while guaranteeing an adequate degree of dependability • Cost-effective testing environments include cloud-based testing

34. Software Engineering in Mobile Computing Testing through the layers • Failures do not only depend on buggy applications, but also on operating systems malfunctioning, or sensored data lack of precision • Testing automation towards all the different layers, to clearly separate application-level failures from application framework or OS failures

35. Software Engineering in Mobile Computing Cloud-Based Testing Model

36. Software Engineering in Mobile Computing Cloud-Based Testing Model • Testing on real devices allocated to the customer based on their specification and requirement. Customers are charged based on the usage time of the device • Emulator: Emulator creates a virtual mobile device environment of any platform or any manufacturer’s specific brand. Since it’s an emulator it can be set in no time and can be customized easily • Automated testing tools like HP-Quick Test, IBM’s RQM, and Robotium can be installed in the cloud server

37. Software Engineering in Mobile Computing How to use TaaS • 1. Develop user scenario that need to be tested. • 2. Design test cases. • 3. Select the cloud service provider. • 4. Select subscription based on the duration to access cloud services. • 5. Setup infrastructure and the configuration required to test the result on the device. • 6. Start testing the scenarios on the virtual devices (cloud based). • 7. Monitor testing goals. • 8. Deliver results and redevelop test cases if any. • 9. Deployment

38. Software Engineering in Mobile Computing Filing Mobile Bugs • If you find a bug within a mobile app, you need to report it in order to get it fixed and/or help customer support of the product. • Filing mobile bug reports provides some information to the developers in order to reproduce and fix the bug. • Need to know: • what is important when filing a mobile bug? • what should a bug report look like?

39. Software Engineering in Mobile Computing Filing Mobile Bugs • Bug ID • Description • Steps to Reproduce • Expected Result • Actual Result • Work-around • Reproducible • Operating System, Mobile Platform, and Mobile Device • Mobile-Device-Specific Information

40. Software Engineering in Mobile Computing Filing Mobile Bugs • Browser Version • Software Build Version • Network Condition and Environment • Language • Test Data • Severity • Bug Category • Screenshot or Video • Log Files • Tester Who Found the Bug

41. Software Engineering in Mobile Computing Filing Mobile Bugs - Bug ID • Bad: 123 is a unique ID, but you might have several projects whose IDs are the same. • Good: AppXYZ-123 is good because you’re combining an ID with a project abbreviation and a number.

42. Software Engineering in Mobile Computing Filing Mobile Bugs - Description • Bad: “The app crashed,” “White page,” “Saw an error,” “Bug” • Good: “Error Code 542 on detail message view” or “Time-out when sending a search request”

43. Software Engineering in Mobile Computing Filing Mobile Bugs - Steps to Reproduce • Bad: “I tried to execute a search.” • Good: “Start the app and enter ‘Mobile Testing’ into the search input field. Press the search button and you’ll see the error code 783 on the search result page header.”

44. Software Engineering in Mobile Computing Filing Mobile Bugs - Expected Result • Bad: “It should work” or “I didn’t expect it to crash.” • Good: “I expected to see a search results page with a scrollable list of 20 entries.”

45. Software Engineering in Mobile Computing Filing Mobile Bugs - Actual Result • Bad: “It didn’t work.” • Good: “The search results page was empty” or “I got the error code 567 on the search results page.”

46. Software Engineering in Mobile Computing Filing Mobile Bugs - Work-around • Bad: “I found a work-around.” • Good: “If you put the device into landscape mode, the search button is enabled and the user can search again.” • Work-around steps are important to know since the work-around could cause other problems. • A work-around can also be useful for the customer support team to help customers solve the current problem until the bug gets fixed.

47. Software Engineering in Mobile Computing Filing Mobile Bugs - Reproducible • If you found a reproducible bug does it occur every time? • If the bug happens every time, it should be easy to be fixed. But if the bug occurs only 20% of the time, it is much harder to find a solution. • Bad: “Sometimes” • Good: “The bug occurs two out of ten times.”

48. Software Engineering in Mobile Computing Filing Mobile Bugs – OS/Mobile Device • Bad: “On Android” or “On iOS” • Good: “Android, Version 4.1.2 Google Nexus 4” or “iOS, Version 6.1 iPhone 4S”

49. Software Engineering in Mobile Computing Filing Mobile Bugs – Mobile-Device-Specific Information • Bad: No information • Good: “GPS sensor activated, changed the orientation from landscape to portrait mode” or “Used the device in a sunny place” or “Battery state was 15%” or “Battery state was 100%.”

50. Software Engineering in Mobile Computing Filing Mobile Bugs – Browser Version • Bad: “Google Chrome” or “Mozilla Firefox” • Good: “Google Chrome version 45.35626” or “Mozilla Firefox 27.6”