Design and Implementation of a Wireless Handheld Multimodal Digital Video Library Client System

1. Design and Implementation of a Wireless Handheld Multimodal Digital Video Library Client System Sam K. S. Sze Henry K. P. Choi Feb 27 2002 Group Meeting

2. Abstract We have developed a client system for accessing a multimodal digital video library (DVL) namely, iVIEW. It provides a user interface that meets the challenge of rich multimodal information presentation on wireless handheld device. An XML schema is employed to organize the multimodal metadata. Furthermore, we investigated a context awareness mechanism complementary to the XML schema to facilitate scalable degradation under restricted resources in wireless application environment.

3. Presentation Overview • Introduction • iVIEW Overview – A Practical Perspective • iVIEW Wireless Client • Scalable Degradation • Conclusion

4. Introduction • Evolving DVL technologies • Common challenges for wireless handheld device application development • Challenges for implementing wireless digital video library client system

5. Introduction:Evolving DVL Technologies • Multimodal information extraction and organization. The modals includes: speech, on-screen characters, geographical locations, names, faces, on-screen objects … • Internet access • Wireless access at anytime and anywhere

6. Introduction:Common Challenges of Wireless Handheld Devices • Limitations Inherited from Hardware • Screen Size • Input Device (Pen device) • CPU Power • Memory • Battery Life • Bandwidth Limitations

7. Introduction:Challenges for Implementing a Wireless Digital Video Library Client • Intelligent User Interface • Content Selection • Result Set Refinement • Integrate Handheld Platform Operation Manner • Resources Management

8. iVIEW Overview

9. iVIEW Overview:If Someone Comes to Room 121 and Ask VIEW Technologies …. I want my video content to be searchable over Internet and wireless tomorrow. What kind of stuff can you guys provide? ?

10. iVIEW Overview:What’s your answer? My Answer is … Demonstration

11. iVIEW Client Architecture

12. iVIEW Client:iVIEW Client System Development • The iVIEW Client has been developed in several approaches: • As a Microsoft Windows Native Application using Visual Basic as the frontend with a number of Visual C++ DLLs. (CMU version, Michael Christel) Further extended to support both English and Chinese. (April 2001, Sam) • As a Java Applet for internet access using internal frames (Jacky FYP). Extended to support Real and Quicktime streaming, XML message parsing (August 2001, Sunny, Sam). • As a web page for internet access using Dynamic HTML and ASP. (December 2001, Sam). • Wireless …

13. iVIEW Client:A History of iVIEWWireless Client • April 2001 (1 month) Survey - Sam • June 2001, (3 weeks) Wireless Web Version – Gordon, Sam • August 2001, (2 weeks) Framework of Multi-windows native application using Embedded Visual C++ - Sam • October 2001 (2 weeks) Bluetooth and GSM compliance tested – Sam, Mike • November 2001 (3 weeks) A variant version with HTML viewer control – Mike • December 2001 (1 month) A multilingual search engine using ASP – Sam, Gordon • January 2002 (1 month) A full version of the multi-windows application - Henry • February 2002 (2 weeks) A performance monitor added - Henry

14. iVIEW Wireless Client:Implementation • Platform: iPAQ PocketPC • Wireless connections: • GSM HSCSD • 2.5G CDMA • 802.11b • Bluetooth • Multi-windows User Interface • 3 Phrases of Operation: • Query • Result Set Manipulation • Presentation

15. iVIEW Wireless Client:Query • Chinese and English Query

16. iVIEW Wireless Client:Result Set Manipulations • Segment View, Tree View, Topics View

17. iVIEW Wireless Client:Presentation • <?xml version="1.0" encoding="big5" ?> • <sequence path="/iview/video/"> • <time start="0"> • <script> GOVERNMENT HAS RESTORED FULL </script> • <framefile="frame141_00.jpg" /> • </time> • <time start="2"> • <script> DIPLOMATIC RELATIONS WITH LIBYA • </script> • </time> • <time start="4"> • <script> AFTER A 15 YEAR SUSPENSION. CNN'S </script> • <framefile="frame141_01.jpg" /> • </time> • <time start="6"> • <script> MARGARET LOWRIE HAS THEDETAILS FROMLONDON • </script> • <map>LONDON</map> • </time> • <time start="8"> • <script> BRITISH FOREIGN SECRETARY ROBIN COOK • </script> • </time> • <time start="11"> • <script> ANNOUNCED DIPLOMATIC TIES WITH • LIBYA WOULD BE RESTORED BECAUSE IT • </script> • </time> • <time start="14"> • <script>IT NOW ACCEPTS RESPONSIBILITY • </script> • <framefile="frame141_02.jpg" /> • </time> • . . . . . • </sequence>

18. iVIEW Wireless Client:Gracefully Degradation at Multimodal Presentation

19. iVIEW Wireless Client:Gracefully Degradation at Multimodal Presentation

20. iVIEW Wireless Client:Gracefully Degradation at Multimodal Presentation

21. iVIEW Wireless Client:Embedded Resources Monitor Resources • CPU Usage • Memory Utilization • Bandwidth Consumption

22. iVIEW Wireless Client:Gracefully Degradation at Multimodal Presentation A set modal significance order chain (i.e. S(m1) > S(m2) > S(m3)> …) should also be defined for different video types. If system utilization saturates, the dispatch will be signaled to make a decision to stop dispatching a set of active modal of least significance. Mathematically,

23. iVIEW Wireless Client:Measurement – A Typical Client Operation Profile

24. iVIEW Wireless Client:Measurement – Degrade Transition

25. Conclusion A wireless DVL client system, that meets the challenges of handling complex multimodal presentations in a handheld device, had been developed. We had shown a user interface design that allow flexible content selection, result set refinement in different views that fits the pen computing operation profile, and it also equips the capability of high complexity multimodal presentation.   We define a multimodal presentation XML with its corresponding context aware scalable degrade mechanism that optimizes the presentation content under resources limited and fluctuating environment.

26. Q & A