Lecture 17: Development of and for Devices

1. Lecture 17:Development of and for Devices Jake Wobbrock 05-830Advanced User Interface Software

2. Not a Proper Toolkit Lecture • This lecture might have been about UI toolkits for devices • There’s just one problem • There really aren’t any • At least, not like we think of toolkits for desktop software • Instead we will talk about “things that help you develop for devices.” • And other interesting device issues…

3. Overview • Introduce work in multiple semi-related areas of software development of and for devices • Palm OS • Pocket PC / Windows CE • .NET Compact Framework • J2ME • Waba • BREW • WAP and WML • DirectX

4. Overview cont. • Highlight HCI research projects with and for devices • Phidgets • Rapid prototyping for physical devices (physical widgets) • Pebbles Project • Interoperation of handhelds and PCs • Handheld Web browsing (various efforts) • WebThumb, Power Browser, The Gateway, M-Links

5. Motivation: Handhelds • Cell phones and PDAs have seen huge growth in use over recent years. • “Last year [2002] there was one cell phone sold for every 15 human beings on the planet … Worldwide mobile phone unit sales totaled 423.4M in 2002, a 6% increase from 2001.” • “From 1999 to 2000 the number of PDAs sold nearly doubled, from some 3.6M units to 6.9M… By 2004, the number of units is expected to top 33.5M.” Source: Biz Journals

6. Motivation cont.: Gaming • Nintendo GameBoy • NTT DoCoMo Inc. handhelds running Java • Others by Qualcomm, L.M. Ericsson, Motorola, Siemens … • Convergence: Games for phones and PDAs, not only custom handhelds Source: CNN

7. The Punch Line • PDAs, phones, other handhelds experiencing great growth in use • Development community needs to have good tools and toolkits • What is available today? • Sadly, not much (at least compared to what’s available for the desktop)

8. Palm OS Visit http://www.palmsource.com/products/ • Palm OS by PalmSource • Acer • AlphaSmart • Fossil • Garmin • Handspring • HandEra • Kyocera • Palm • Samsung • Sony • Symbol

9. Palm OS Facts • As of January 2002: • 16M Palm-powered PDAs shipped (to date) • 7000 applications • 150,000 registered developers • Growing at 1000 per month Source: Palm OS Programming, O’Reilly

10. Palm OS Development • Two main ways • CodeWarrior for Palm OS • PRC-Tools (GCC) • Other ways • Sun KVM and J2ME • IBM VisualAge Micro Edition (Windows or Linux) • Jump (use Java, compiles to Motorola 68K) • Waba (subset of Java, stripped-down VM) • PocketStudio (Pascal-based, aimed at Delphi users) • Pocket C (develop on the handheld for the handheld) • Assembler SDK (ASDK) – write directly for Motorola 68K • Pocket Smalltalk (includes garbage-collecting VM)

11. CodeWarrior for Palm OS • Special line of Metrowerks CodeWarrior releases for Palm OS development • Three key components • Editor • Constructor • Palm OS Emulator (POSE) • Write code in C/C++ • Downloaded to Palm via HotSync™

12. Editor

13. Constructor

14. Runs on the desktop Emulates from a “very low level” Designed to simulate speed despite running on a desktop Useful for debugging Not useful for interactions requiring pen-dexterity Emulator (POSE)

15. Technical Details • Palm OS apps center around event cascade • Catch events by defining handlers and handling them • But not nearly as much framework as Visual Basic or MFC… more like Win32 • Often involves large switch statements • Single-threaded, single-process

16. The Event Cascade

17. CodeWarrior Summary • Generally the best option • Originally for Macintosh, then for Windows! • BUT… • Constructor is often not as WYSIWYG or Direct Manipulation as you’d like (e.g., text fields, scroll bars) • It lacks options for all programmatically-settable options that affect visual appearance of controls (e.g., button borders) • Emulator doesn’t work exactly like the device • Sometimes this matters, often this doesn’t

18. PRC-Tools (GCC) • Alternate means for Palm OS development • Developed before CodeWarrior ran on Windows by the Free Software Foundation • Collection of tools for Unix or Windows • Gnu C Compiler (GCC) for Motorola 68K • Build-PRC (combine binaries into *.prc file) • Gnu Debugger (GDB) • PilRC (compiles resource files) • Falch IDE combines these with a UI

19. Pocket PC? Windows CE? • Pocket PC • Not considered (strictly-speaking) a PDA • Or at least not marketed that way • Meant to be a full-fledged PC but pocket-sized • Refers to a device (platform) • Also to a set of standard applications common on all Pocket PC devices, esp. Pocket PC Shell • Windows CE (WinCE) • Refers to just the OS that runs on Pocket PCs • Has large number of capabilities (components) for the Pocket PC developer to choose from • Current version is WinCE .NET

20. Pocket PCs vs. Palm OS Devices • “Basic PDAs allow you to store and retrieve addresses and phone numbers, maintain a calendar, and create to-do lists and notes. More sophisticated PDAs can run word processing, spreadsheet, money manager, games and electronic book reading programs and also provide email and Internet access.” Source: Microsoft

21. Pocket PCs • Toshiba • Compaq iPaq • HP Jornada • Audiovox • T-Mobile • Dell Axim • ViewSonic • Razor Zayo • Cassio Cassiopeia • NEC MobilePro

22. Pocket PC Facts • Projected 2003 sales between 14M – 17M devices • Compare to PC sales of 135M – 138M • Pocket PC projected to catch up by 2008! • Sales hit 10M after just 10 months on the market • Introduced April 2000 • In May 2001, held 26% of market. Rest held by Palm OS devices. Source: C|Net

23. The Shrunken Desktop • Pocket Word • Pocket Excel • Pocket Outlook • Pocket IE • Pocket Windows Media Player • … ??

24. Pocket PC Development • eMbedded Visual Tools 3.0 (VC++, VB) • Largely just like Windows programming • Can even use MFC for VC++ • Compiles for many different processors

25. Challenge • Problem: Portability • Not all Pocket PCs utilize the same WinCE components • So not all Pocket PC apps will run on all Pocket PCs • Despite same underlying OS! • Example: The menu bar at the bottom is specific to Pocket PCs, not WinCE • Created and passed to the Pocket PC shell • Developers often have to change much UI code for different Pocket PC platforms • Example: Handheld PC (H/PC) is wide and short, so tall dialogs have to be re-laid-out

26. Resource Editor

27. Pocket PC Summary • Powerful palmtop computers • More complicated than Palm OS Devices (e.g., can be multi-threaded) • Less integrated vertically than Palm OS devices • Complicates development (e.g., UI components not often supported on all Pocket PC devices) • With more power and capability comes more complexity and complication • Affects the toolkits for development on such devices

28. .NET Compact Framework • Subset of .NET framework for desktops • Not PDA-specific but targeted to any “smart devices”: PDAs, mobile phones, set-top boxes, automobiles, etc. • Delivered as extensions to Visual Studio .NET called “Smart Device Programmability” • Tools and programming model are same as for desktop .NET platform • Easy transition from .NET desktop development to .NET compact framework development

29. .NET Compact Framework cont. • Argument is that there are currently too many mobile CPUs and OSes, so development is difficult • True – but we know the Microsoft answer • .NET would allow integration of devices across multiple platforms, networks, and programming languages • A vision behind .NET in general • Compact Framework is a subset of all .NET capabilities • Reduces footprint on resource-strained devices

30. J2ME – Java 2 Micro Edition • Uses subset of Java Virtual Machine for smaller footprint (i.e., KVM) • Highly optimized runtime environment • Not PDA-specific: smart cards, pagers, mobile phones, set-top boxes, vehicle telematics systems (e.g., OnStar system) • Set of APIs defined by the Java Community Process Program™ • Includes user interface, security, networking protocols, and more

31. J2ME Architecture • J2ME Architecture defines… • Configurations: A virtual machine and a minimal set of class libraries • For Palm OS, configuration is “Connected Limited Device Configuration” (CLDC) • Profiles: Higher-level APIs that further define the application, user interface, and device-specific properties • For Palm OS, profiles are PDA and “Mobile Information Device Profile” (MIDP)

32. J2ME Architecture cont. From http://java.sun.com/j2me/j2me-ds.pdf

33. Waba • Subset of Java • Stripped-down VM: small and fairly fast • No longs, doubles, exceptions, or threads • Open source: http://www.wabasoft.com/ • Waba defines… • A language (strict subset of Java) • A virtual machine (Waba VM) • A class file format (strict subset of Java bytecode) • A set of foundation classes

34. BREW • Qualcomm’s Binary Runtime Environment for Wireless (BREW) • Like J2ME, provides application execution environment • Goes farther! Provides business model for certifying, downloading, and charging premium content • With J2ME, this is left entirely to the marketers and developers to figure out for themselves • Based on C++ (so it is fast, and small binaries)

35. WAP and WML • WAP (Wireless Application Protocol) • Provides “flat” Web content on handhelds • Requires constant connectivity by the device • Lacks a rich interactive experience for end-users • WML (Wireless Markup Language) • Specifies content and user interface for WAP-delivered information • Based on XML: so describes data, not just presentation of data • WMLScript • WAP:WML:WMLScript :: HTTP:HTML:JavaScript • Provides only minimal interactivity

36. WAP and WML cont. • Four things specified by WML • Text and image presentation and layout, including a variety of formatting commands • Deck/card organizational metaphors for information (similar to HyperCard) • Inter-card navigation and linking • Card parameterization and state management • The fate of WAP • Hasn’t been the success its advocates hoped • New push to make possible highly interactive content • J2ME, BREW, .netCF, Waba, but these all send a VM and increase footprint

37. Game Development: DirectX • DirectX: Advanced suite of multimedia APIs • DirectDraw, Direct3D, DirectSound, DirectMusic, DirectInput, DirectPlay, DirectShow… • Allows accessing of special hardware capabilities (e.g., graphics and sound cards) without requiring hardware-specific code • DirectX exists for Pocket PC • Handheld gaming projected to be a big new market opportunity • Again, similar to programming for desktop

38. Transition from (Mainly) Industrial Tools to Research Initiatives for and with Devices

39. Phidgets (Greenberg et al.) • Phidgets are “physical widgets” • Play the same role in physical UIs as widgets do in graphical UIs • Package IO • Hide implementation details, provide abstraction • Have additional characteristics • Require a connection manager • A link between the software and hardware parts • A simulation mode in software only

40. Phidget Architecture • Physical Device • Packaged physical unit to be used by the designer. Most phidgets are built around a circuit board. • Wire Protocol • Communication protocol between device and host computer. Not visible to end-programmers. • Phidget Manager • COM object that has event-based API for end-programmers for connection management. • Phidget-specific COM objects • Created by the Phidget Manager whenever a device is seen. They correspond 1:1 to physical devices. • IGlabPhidget interface • Generic interface which all phidgets have. End-programmers use it for basic identification and capabilities. • Phidget-specific interface • Capabilities specific to a particular phidget. Extends the basic IGlabPhidget interface. • Phidget ActiveX controls • Wraps up the phidget code and provides an on-screen user interface and the ability to simulate the phidget as an control.

41. Pebbles (Myers et al.) • Research project exploring the interoperation of handhelds and PCs • Handhelds can be an aspect of the user, PCs can be a fixture of the environment • Users may move from environment to environment, PCs may stay behind • Handhelds can control other devices too • Personal Universal Controller • Handhelds can be used to control a PC by people with motor impairments

42. A Current Research Issue • Handheld User Interfaces for Web Browsing • Challenges • Constrained screen-space • No mouse cursor (e.g., no link roll-overs) • Pages designed on a desktop for display on a desktop • Scrolling more difficult • Graphic refresh can be slow, scrolling blurry

43. Handheld Web Browsing • The Gateway (MacKay) • Pages displayed as thumbnails expand as certain parts of them are interacted with (CHI 2003). • WebThumb (Wobbrock et al.) • Exploration of interaction techniques for better handheld browsing (UIST 2002). • M-Links (Trevor et al.) • Uses modes to separate action from navigation on mobile phone Web pages (UIST 2001). • Power Browser (Buyukkokten et al.) • Generates summary views of handheld web content for display (CHI 2000).

44. Conclusions • Toolkits for development for devices are far from Amulet-level sophistication • Many processors and devices exist – complicates development and toolkit design • Numerous standards and niche technologies • Minimizing footprint on resource-constrained devices is a key constraint • HCI research can tackle issues beyond the specifics of device hardware and software • e.g., interaction techniques for PC/PDA interoperability • e.g., interaction techniques for browsing