1 / 56

AMC : Verifying User Interface Properties for Vehicular Applications

AMC : Verifying User Interface Properties for Vehicular Applications. Kyungmin Lee * , Jason Flinn * , T.J. Giuli + , Brian Noble * , and Christopher Peplin + University of Michigan * Ford Motor Company +. Where do we use mobile apps?. Walking. Driving. Everywhere we go

lacklin
Download Presentation

AMC : Verifying User Interface Properties for Vehicular Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AMC: Verifying User Interface Properties for Vehicular Applications Kyungmin Lee*, Jason Flinn*, T.J. Giuli+, Brian Noble*, and Christopher Peplin+ University of Michigan* Ford Motor Company+

  2. Where do we use mobile apps? Walking Driving Everywhere we go (Not-too-distant future) Kyungmin Lee

  3. Where do we use mobile apps? Walking Driving Everywhere we go (Not-too-distant future) Kyungmin Lee

  4. Mobile apps in a vehicle • Pros: • High usability standard • Good form factor • Cons: • Cost of deployment • Limited applications • Pros: • Diverse applications • Easy deployment • Cons: • Distracting apps • Poor form factor Kyungmin Lee

  5. Mobile apps in a vehicle How can we get the best of both worlds? • Pros: • High usability • Good form factor • Pros: • Diverse apps • Easy deployment Kyungmin Lee

  6. Our approach • Create a tool (AMC) that • Automatically explores the application • No source code & app-specific knowledge • Pinpoints UI guideline violations Kyungmin Lee

  7. AMC’s benefits • Developers • Test own apps • Early design feedback • User interface experts • Fewer properties to check • Focus on hard properties Kyungmin Lee

  8. AMC’s benefits • Developers • Test own apps • Early design feedback • User interface experts • Fewer properties to check • Focus on hard properties <AMC’s results> 20X reduction in testing False negative: 2% False positive: 0% Kyungmin Lee

  9. Outline • Motivation • How AMC explores the app • How AMC finds UI violations • How closely do apps follow UI guidelines • Conclusion Kyungmin Lee

  10. How to use AMC App to test Run AMC Get violation report • Violation report • Button size: • 1) Button A at state 1 • 2) … • Contrast ratio: • … • … Application Monkey Modified ApplicationFramework Hierarchy Viewer - State exploration - Find violations Kyungmin Lee

  11. How does AMC explore? State #0 Kyungmin Lee

  12. How does AMC explore? Current Location State #0 State #1 Kyungmin Lee

  13. How does AMC explore? Kyungmin Lee

  14. How do we define a state • Different screens -> Different states vs. Kyungmin Lee

  15. Fine-tuning state definition • Naïve approach: every unique screen -> state • Solution: Structure hash • Same structure, different content -> same state • Avoids state explosion vs. Kyungmin Lee

  16. Fine-tuning state definition • Structure hash is not enough… • Solution: Consider only 0,1,or many items • If there are 2+ items, ignore the rest • Unless they transition to different states vs. Kyungmin Lee

  17. Outline • Motivation • How AMC explores the app • How AMC finds UI violations • How closely do apps follow UI guidelines • Conclusion Kyungmin Lee

  18. Finding UI violations • Checks for 7 UI properties • User actions per task, Text contrast ratio, Word count, Button size, Button distance, Animation, Scrolling • Defined by transportation research groups • Report violation, no violation, or defer to experts Kyungmin Lee

  19. Finding UI violations • Properties of individual state Kyungmin Lee

  20. Finding UI violations • Properties of individual state Kyungmin Lee

  21. Finding UI violations • Properties of individual state Kyungmin Lee

  22. Finding UI violations • Properties of individual state Total of 12 words Kyungmin Lee

  23. Finding UI violations • Properties of individual state Kyungmin Lee

  24. Checking user actions per task Kyungmin Lee

  25. Few complications • Buttons in custom view • Empirically discover buttons • If fails, defer to manual verification • Determining good vs. bad animation vs. Kyungmin Lee

  26. Outline • Motivation • How AMC explores the app • How AMC finds UI violations • How closely do apps follow UI guidelines • Conclusion Kyungmin Lee

  27. Evaluation • Environmental setup Kyungmin Lee

  28. Comparison with expert’s evaluation Tested 12 out of 14 apps & 5 out of 7 properties Kyungmin Lee

  29. Comparison with expert’s evaluation • : Violation • : No violation • : Maybe • 1st column: • AMC’s report • 2nd column: • Expert’s report Kyungmin Lee

  30. Comparison with expert’s evaluation • : Violation • : No violation • : Maybe • 1st column: • AMC’s report • 2nd column: • Expert’s report Kyungmin Lee

  31. Comparison with expert’s evaluation • : Violation • : No violation • : Maybe • 1st column: • AMC’s report • 2nd column: • Expert’s report Kyungmin Lee

  32. Comparison with expert’s evaluation • : Violation • : No violation • : Maybe • 1st column: • AMC’s report • 2nd column: • Expert’s report Agreed 50 / 60 cases ≈ 83% Kyungmin Lee

  33. Comparison with expert’s evaluation • : Violation • : No violation • : Maybe • 1st column: • AMC’s report • 2nd column: • Expert’s report One case of false negative <≈ 2% Kyungmin Lee

  34. Comparison with expert’s evaluation • : Violation • : No violation • : Maybe • 1st column: • AMC’s report • 2nd column: • Expert’s report Kyungmin Lee

  35. Comparison with expert’s evaluation • : Violation • : No violation • : Maybe • 1st column: • AMC’s report • 2nd column: • Expert’s report Animation: 4 out of 381 states Button size & distance: 13 states Reduced by 95%! Kyungmin Lee

  36. Assessment of Android apps Kyungmin Lee

  37. Assessment of Android apps Kyungmin Lee

  38. Assessment of Android apps Kyungmin Lee

  39. Assessment of Android apps 0.75 violations / state Kyungmin Lee

  40. Assessment of Android apps Kyungmin Lee

  41. Assessment of Android apps 0.69 violations / state Kyungmin Lee

  42. Assessment of Android apps Only MPG app had no violations Kyungmin Lee

  43. Conclusion • AMC is a tool that • Explores the application automatically • Pinpoints UI guideline violations • Many vehicular apps with UI violations • Shows the need for a tool like AMC • Can be extended to other UI settings Kyungmin Lee

  44. Questions? Kyungmin Lee

  45. Kyungmin Lee

  46. Backup slides Kyungmin Lee

  47. State coverage Avg. coverage: 80% Kyungmin Lee

  48. States found & runtime Median runtime: ~74 minutes Avg. runtime: 2hr 31min Kyungmin Lee

  49. Non-deterministic transitions • Caused by • Configuration option • Previous paths • Mistakes in state classification • Solution • Avoid using non-deterministic transitions! • Use past exploration information • Combine previous N transitions into one • Attempt to make it deterministic Kyungmin Lee

  50. Non-deterministic transitions • Caused by unaware hidden property BACK BACK Kyungmin Lee

More Related