Exciting Media Limsoon Wong Institute for Infocomm Research

1. Exciting Media Limsoon Wong Institute for Infocomm Research

2. Plan • I will discuss some of the advances on the handling and processing of native media • New things that you can do with texts • New things that you can do with images • New things that you can do with audio • New things that you can do with video

3. New things that you can do with texts

4. Search engines are getting less useful than before too many hits not all relevant not “organized” [Hierarchal clustering from BIGontheNet] take Google to a new level – Yahoo! Finance [p-zoom] may have a leg up on… its competitors – Tech Web News by “I like BigOnTheNet's and Groxis's web search categorization technologies a lot” – Chris Shipley on WashingtonPost.com Take search engines to the next level

5. Extract chemical safety information from Materials Safety Data Sheets (MSDS) Check for conformance to standards Benefits OHD: Check 100% of MSDS (currently < 10%) with same manpower Chemical Suppliers: Savings in distribution of MSDS as it is online End users of chemicals : Better quality MSDS, improved safety MSDS (in variety of formats) searches End-users (English- speaking) Chemical suppliers MSDS (verified) MSDS Knowledge Workbench Knowledge encoding OHD Intelligent information extraction,improve safety

6. How is it done?What’s needed to get it done?

7. New things that you can do with images

8. Abstraction of image content allows interpretation and matching in semantic space Visual query language allows specification of what and where Make computers easier to use Search photos by visual keywords

9. Trained visual keywords for semantic detection and summarisation Automatic indexing using such keywords Faces : Crowd : Buildings : Foliage: How is it done? What’s needed to get it done?

10. Higher PSNR! New perceptual metric perceives correctly. Comparison with MOS (Mean Opinion Score) Pearson Correlation Spearman Correlation PSNR metric 0.66 0.69 New metric 0.83 0.81 Let machines perceive as we do • Perceptual visual quality according to characteristics of human vision • Outperform metrics in ITU-T VQEG test • Adoption in video coding results in efficiency & quality improvement (other systems make compromise betw. the two) Better consistency Better accuracy

11. Third Party Publication: Sign Once, Verify Many Ways Protect authenticity and integrity of data in a robust way

12. New things that you can do with audio

13. Automatic speech recognition Business Logic PSTN Text-to-speech Speech enhancement, Noise reduction Multilingual voice mining, Speech & dialogue processing Text categorization, Natural language processing Semantic outputs voice Let machines listen as we do

14. To build a mobile audio industry… analogous to the graphics industry • Synthesis-directed analysis of sounds • how would you model a lion’s roar? • Algorithms for synthetic sound generation • Tools for sound model automation and support • Cross-platform audio synthesis engine with a small footprint and low compute requirements

15. Mobile and non-invasive health-monitoring devices for home use Potentially large market for densely populated areas with low medical facilities & personnel Aging population in developed countries means more need for automatic continuous monitoring to lower overall medical costs Passive health monitoring devices less of health hazard and allow long-term usage Sound-based automatic detection & classification of medical anomalies, e.g., Long term fetal heart sound monitoring Make health monitoring less hazardous

16. New things that you can do with video

17. Make home videos more fun • Select video frames • Cut to music • Decide on transitions Turn home video into high quality MTV automatically

18. Drowning Early Warning System tracks people in dynamic aquatic conditions intelligently detect water crises situations Prevent drowning, save lives

19. Wireless Network And improve quality at the same time! Watch video any where, any time, on any device!

20. More intelligent CCTV, improve homeland security

21. Tagged to camera h/w • costly • done once at source • “Intrusive” ads • pops out during play! • Enhanced tennis TV • auto-tracking • super-resolution • Software detection: • performed any-time • demographic Ads • cheap • Non-intrusive insertions • detects non-play segment • non-interfering insertion • Tennis TV • manual Improve sophistication of our media industry Existing pains Non-intrusive virtual contents insertion

22. How is it done? What’s needed to get it done? • Robust Scene Modeling and Camera calibration • Given a 2D court model of 3D scene that camera is capturing, identify 3D object positions robustly and accurately • Super-Resolution Image Reconstruction from Video • Given a low resolution image sequence of an object far away from the camera, reconstruct a larger resolution image sequence • This is essentially an ill-posed problem, but we can apply domain info such as motion, pose, etc, to seek a good solution • Robust Object and Landmark Detection • Real-time • Geometric invariance • Deployment and Application Constrains • Real-time

23. Looking even further... Media in 2010 according to Institute for the Future

24. Dimensions of entertainment activities • The event • The process • Popularization of research • Practice & performance level • Entertainment spaces • Entertainment tools • Sharing & social communication • Consuming entertainment • Creating entertainment • The work of entertainment

25. Key shifts shaping new entertainment • Mass to Personal • consumers will appropriate mass media tools for their own personal expression • Packaged to Self-generated • consumers will create the entertainment experiences they engage in • Episodic to Persistent • entertainment experience will be ongoing & will have no clear starting and stopping point • Virtual to Embodied • information, images, and experiences will be embedded into physical objects & physical world

26. Eg: Blogging---how the Web evolves from a publishing medium to a personal creativity tool Blogs are written online, using tools accessed thru a Web browser, combining simplicity and immediacy of instant messaging, & broad accessibility of Web sites Most personal blogs are essentially online diaries, sometimes devoted to specific subjects. They are short, updated frequently, and full of hyperlinks, & crafted to look like snapshots of what their authors are thinking and creating Mass to personal

27. Eg: Fantasy sports leagues In these leagues, individuals act as managers of professional sports teams. They pick players in live drafts & develop rosters for playing their team against other fantasy teams in their league The real experience of Fantasy Leagues is the interaction among the various player managers—the player trading, email, & competition The digital world becomes a vibrant place for all sorts of interactions like trading, creation or production, & spectating Packaged to self-generated

28. Eg: Massive multiplayer role-playing games Sony’s EverQuest has over 400,000 players, 70,000 of which are online at any given time, slaying dragons, traveling to cities, or trading with each other. Individual players acquire property, talents, and social identities over time Players’ actions permanently affect the virtual world: a player’s property will exist even after they’re gone Runs 24/7 & when players log off, game continues to evolve and grow, driven by social behavior of thousands of players that log in each day Episodic to persistent

29. Eg: Geocaching adventure game played worldwide Founder creates a cache which holds a treasure, & posts real world location coordinates on a Web site Seekers retrieve coordinates to cache site on the Web & use a GPS device to get within 20 feet of it Then they use clues, hunting skills, special real-world skills (hiking, scuba diving, etc) to find cache Finally, they sign logbook belonging to cache, recover treasure, & leave a treasure for next seeker Virtual to embodied

30. In summary • The new “entertainment” media in 2012 will be oriented around personal media that • are generated by consumers rather than packaged and distributed by providers • provide persistent experiences that do not disappear with a switch of a button but linger over the course of daily life • are touch points in the physical environment that embody entertainment and set forth a new relationship among consumers, entertainment, and their broader daily life activities

31. Implications • Focus on leveraging distinctive points of view • Co-create continuous experiences with consumers • Think of ways to target players who personalize mass events • Develop tools and processes for fusing physical & virtual campaigns • Focus on promoting user customization • Design for participants to “sell to their friends” • Develop products for persistent experiences • Include digital experience in physical products

32. Thank you

33. title: Exciting Media • abstract: I will discuss some of the advances on the handling • and processing of native media. In particular, I • will look at • - new things that you can do with video • - new things that you can do with audio • - new things that you can do with images • evening of 6th april