UI & Gaming in Mobile Handsets

1. UI & Gamingin Mobile Handsets Yoshio KuniyoshiVice President & CTO Acrodea Inc.

3. Outline of the Presentation • Background • Recently, many handsets become equipped with GPU. However the purpose of GPU is mainly for Gaming only. On the other hand, user interface is coming to need more and more graphics processing power. • VIVID UI • As a state of the art UI platform for mobile devices, the features of VIVID UI is introduced to describe how today’s UI platform for mobile handsets needs graphics power. • Describe how current VIVID UI uses GPU. • Roadmap for Accelerating User Interface • Describe the ideal way of accelerating user interface. • Describe problems of current GPU and its API implementation. • Gaming • As a state of the art Game platform for mobile devices, the features of X-Forge is described. • Describe programmable shader support for X-Forge.

4. Introduction • UI and Gaming trends in Mobile Handsets • More and More Graphics Processing Power is Required • Display Resolution Gets HigherQCIF → QVGA→ WQVGA→ VGAQCIF : VGA = 1 : 12 • Demands for Richer Media SupportFlash, SVG, PDF, 3D, Movie(MP4 → H.264 quality) • Demands for Advanced 3D Technologies3D Effects makes user interface become more attractive Programmable shaders makes more realistic 3D games • Recently, many handsets become equipped with GPU. • Example • UI Platform for PC • MacOS X Quartz Extreme • Windows Vista Avalon • Handset • 3D Menu (SHARP V602SH) • VGA Display (SHARP 904SH) • Flash Lite 1.1 and 2.0 (Nokia, Sony Ericsson, and Most of Japanese Phones for au/DoCoMo/Vodafone) • GPU ProcessorsNVIDIA GoForce, IMG PowerVR MBX, TOSHIBA T5G, ATI Imageon, …All these GPUs will soon/already support programmable shaders and OpenVG.

5. Introduction Next-generation UI architecture framework: UI architecture based on scripting and OTA differential updating

6. VIVID UI Features • Scripting Based UI Platform --- UI is Data not CodeEasy to create arbitrary UI structure with rich visual expression through scripting. No need to write native codes. This “UI is Data” concept enables easy provisioning of UI content service by OTA. • Development with GUI Authoring ToolsGUI based WYSIWYG operations result in automatic script generation. • Separate Management of UI from LogicLook and feel, and application logic managed separately.Improves collaboration between designers and programmers. • Provides of Standard UI ComponentsProvides buttons, labels, checkboxes, radioboxes, textfields and so on. Other components can be easily added. • Enables Any Kind of Media Composition Designed as a container for any kind of graphics renderer. Various materials including 2D images, Flash, 3D, and Movies can be freely composed by alpha blending.

7. Supported platforms • OS : Brew / Linux / Symbian / RTOS • CPU : ARM7 30MHz or later • TOOL : Windows / Macintosh (future plan) • ROM / RAM size • ROM : between 100KB and 500KB (depends on components) • RAM : between 200KB and more (depends on contents) VIVID UI RequirementsComplete UI solution for today and tomorrows devices

8. UI Concept Planning UI Materials design Authoring with tools VividUI Workshop Packaging Terminal embedding OTA delivery Placed in VividUIdistribution server VIVID UI Contents Development and Distribution Flow Package all data such as XML scripts (compiled form), and resource files into one file Handset with VividUI engine

9. Achieving Diverse UI Instead of embedding a fully designed UI, loading a UI platform environment is a much more desirable direction for handset manufacturers. Then subscribers can select their desired UI from among a variety of options to suit their preferences. By responding to diversified needs, a whole new “UI epoch” will begin.

10. VIVID UI Multi Media Support2D, Flash, 3D, and Movies • VIVID UI can integrates any kind of graphics renderer on a handset including 2D images, Flash, 3D, and Movies. • Various materials including 2D images, Flash, 3D, and Movies can be freely composed by alpha blending. • VIVID UI has built-in 2D renderer, 3D scene graph engine based on OpenGL|ES, and interface to Flash and several movie decoders. 3D Animated Menu Icons Flash Animation 2D Images 2D Menu on top of movie.

11. VIVID UI Workshop • VIVID UI Workshop is an authoring tool to facilitate user interface development with VIVID UI. • This GUI tool enables WYSIWYG based editing of UI structure. UI designers can create UI by just putting components on the design pane of the tool then setting several properties for each component and the tool automatically generates corresponding XML scripts on the fly. • UI designer can verify the behavior and appearance of the UI content with built-in emulator. • Handset profile feature provides easy adoption of different handset models. • Integrates all operations needed for UI content creation. UI Design Tool

12. Plug-ins provide extensibility, and are accessible through scripting Handset menus developed by VUL scripting Skins can be open to the end users, who can freely customize appearances. VUL scripts are interpreted and then drive the VividUI engine. Yellow frame indicates VIVID UI Platform VU API can be used directly for UI app implementation. User Customizable Skin Folder Menu Script Top Menu Script Setting Menu Script Camera Menu Script The abstraction layer for various native functions. VU app VU app VUL interpreter VU plug-in VividUI API Plug-in API VividUI engine VU plug-in Key events and handset UI functions such as screen drawing are accessible through scripting also. Native Abstraction Layer Scripts can launch various embedded programs. Handset Platform API Flash 3D OES 2D SVG-T Video AMC Audio MP3 App launcher Client Architecture

13. UI Scripting Architecture • UI architecture through scripting • VividUI builds a resource file UI composed of XML files, simple scripting language, and images. • Separated UI components • UI components are written asContents: UI structureSkin: UI appearanceLocale: UI localization • Modular structure • Overlaying feature enables UI sharing and componentizing • Presentation script feature • Style sheet definitions enable pixel-level precise control. VUL* script 3 components Contents Structural scripts such as menus Action scripts Skins Style scripts Sound and image resources Locales Localized language data *VividUI Scripting Language.

14. Embedded Advanced Functions • Embedded animation effect • Can be used to impart attractive motion on the UI. • Push and pull communications • Adapts to server “push” communications for timely data delivery to handsets. • Abstraction with rendering engines • 3D/Movie/Flash can be accessed in concert with native rendering engine by scripting. • Abstraction with sound functions • MP3 and other native sound functions can be accessed by scripting. • Abstraction with status displays • Allows acquisition of status through scripting (signal level, battery life) News and real-time stock prices through “push” comm. Innovative UI possible with 3D display Animation generates motion effect VividUI Engine Unique design and placement of status indicators Introduction of realistic sound

15. VIVIDUI Other Features • Support multi-task environment • re-entrant thread safe architecture / passive library • Effective compression for data • LZ and Huffman based compression / optimization for re-use of resource files • Lightweight alpha blending technology • optimized for mobile environment / anti-aliasing feature • Easy to expand UI components • support 2D vector graphics (FLASH) / 3D graphics (OpenGL-ES) • component plug-in API enables to add other UI components • renderer plug-in API enables to support other graphics rendering engines • Support hardware acceleration • can be optimized for CPU – ARM9/ARM11 / HITACHI SH-Mobile • can be optimized for GPU – OpenGL-ES compatible HW • Easy to port • simple porting layer (30 – 70 functions: depends on the requirements) • customizable porting layer • Support OTA differential update • can be optimized for OTA differential technology • can share porting layer with OTA differential technology

16. VIVID UI OES Implementation • Current VIVID UI OpenGL ES Implementation • Supports OpenGL|ES 1.0 and 1.1 • Full OpenGL|ES acceleration for built-in 2D renderer • Full OpenGL|ES acceleration for 3D scene graph engine • Movie is supported by software decoder • Flash Lite is supported (w/ Macromedia Flash Lite library)

17. VIVID UI Acceleration Roadmap • The Ideal way of VIVID UI Accelerated Implementation • Supports OpenGL|ES 1.0 and higher, and/or OpenVG • Full OpenGL|ES and/or OpenVG acceleration for built-in 2D renderer • Full OpenGL|ES acceleration for 3D scene graph engine • Movie is supported by hardware decoder w/ OpenMAX • Accelerated version of Flash Lite is supported • Problems of current GPU and its API implementation • Requirements • Multi context support with independent surface for multi processes. • Multi context support with shared surface for single processes(single/multi threads). • Shared surface support between OpenGL|ES, OpenVG, and OpenMAX. • Resource management • Fine grained power management • Current GPU and API Implementation • Some environments does not support multi context. • Some environments only support multi context with independent surface, but no support for shared surface. • Some environments does not support resource management. E.g.: The upper limit of the texture area is the same as GPU hardware limitation. • No/Rare existing support for OpenMAX and OpenVG w/ shared surface support w/ OpenGL|ES. • No support for fine grained power management from the application.

18. X-Forge - General Presentation

19. The X-Forge™ Tool Chain CODE 1. 2D and 3D Content 2. Particle Effects Executable (EXE) PC Emulator Version Compressed File Library (CFL) CONTENT 3. Music and Sound Effects 4. Scripts Native Device Version

20. The X-Forge™ SDK Additional X-Forge™ Components Your Game Content Graphics Audio Scripts Documentation Stand-alone Tools Your Game Code Programmer Tools Artist Tools 1000+ page API reference Tutorials & sample code Production & tools Optimization guidelines Packager Game Specific Code 3ds Max™ Exporter X-Forge Online Application framework Profile Analyzer Game Graph UI Toolkit X-Forge™ Game Engine Maya™ Exporter Particle FXsystem Physics framework Collisions Particle FX Editor AI Framework 3D Audio OS and Device Abstraction Networking Framework Graphics Audio X-Forge™ Core Resources Controls File I/O UTF8 unicode Debugging tools Localization tools XML SVG Mobile Device OS Device Driver OpenGL ES 1.x/2 Hardware CPU 3D Graphics Accelerator

21. Multi-platform Support Acrodea works in close co-operation with all leading mobile technology companies in order to support a wide variety of different mobile operating systems, processors, 3D graphics accelerators and devices in X-Forge. • Symbian OS™ / Nokia Series 60 • Nokia N-Gage™ • Nokia 3650, 7650, 6600… • Siemens SX1 • Samsung SGH-D700 • Symbian OS™ / UIQ • Sony Ericsson P800 • Palm OS (V5.x) • Tapwave Zodiac™ • Mobile Linux • Sharp Zaurus™ • Microsoft Windows® CE.NET • AlphaCell M5 • Microsoft Smartphone • Orange SPV™ • Microsoft Pocket PC • ARM-based devices X-Forge can also be ported to virtually any additional operating system and hardware. • X-Forge is Nokia OK certified and optimized for the Nokia N-Gage™ game deck • X-Forge has been optimized for leading processor architectures including Intel XScale™ / StrongARM™ and Texas Instruments OMAP™ • X-Forge supports various 2D and 3D graphics accelerators, including products from ATI, nVIDIA and Intel • X-Forge is compatible with standard 3D rendering engines such as OpenGL ES™

22. Existing X-Forge™ Game Content X-Forge™ Delivers Superior Quality Geopod™ Gadarka’s Delusion Snowboarding Extreme Red Faction™ Red Faction™ Super Monkey Ball™ Virtua Tennis™ MotoGP™ Spy Hunter™ Galactic Realms™ Stuntcar Extreme™ Phantom Strike™ Unfinished Business™ Unfinished Business™ Firehammer™ Snowboarding Extreme Gadarka’s Delusion Resistance™ Gadarka’s Delusion Geopod™ Legend of Zord™ Stuntcar Extreme™ Men in Black II™: Alien Pursuit Spy Hunter™ Stuntcar Extreme™ Resistance™

23. Developer Network This is a partial list of developers who have licensed / evaluated X-Forge™.

24. X-Forge Programmable Shader Support • OES Version • Support OpenGL|ES 2.0. • Effects • Supports “Effect” which consists of a shader, and accompanying data and rendering paths.(i.e.: FX file of DirectX) • Built-in Effects • Some effects are built-in (or pre-installed) with X-Forge SDK. Effects for glow, normal map, environment map, bump map will be prepared. • Treat Effect as a part of a scene graph • Assign effects to each model or the entire scene and then export. • These effects will be automatically processed at runtime. • Effect Editing Tool • Effect editor will be provided. • Support High Level API • Provides high level API to manipulate effects from C++ source code.