Distributed Multimedia Systems 07 The Future of Visual and Iconic Language Applications and Inscription Technology: A N

1. Distributed Multimedia Systems 07The Future of Visual and Iconic Language Applications and Inscription Technology: A New Way to Chat LorRaine Tauchi Duffy, PhD SPAWARSYSCEN San Diego 5361 LorRaine.Duffy@navy.mil Sept 6 2007

2. SSC San Diego TEAM • LorRaine Duffy, PhD. Code 5361, LorRaine.Duffy@navy.mil • Emily Wilson Medina, Code 2734, ewilson@spawar.navy.mil • Sunny Fugate. Code 2725,, fugate@spawar.navy.mil • Omar Amezcua, Code 536204, omar.amezcua@navy.mil • Gary Rogers, Code 536204, rogers@spawar.navy.mil • Nghia Tran, Code 23711, nghia.tran@navy.mil • Hoa Phan, Code 2856, hoa.phan@navy.mil • Vincent Dinh, PhD. Code 2856, dinhvv@spawar.navy.mil • Marion Ceruti, PhD. Code 536, Marion.ceruti@navy.mil • Erin O’Conner, San Diego State Student Foundation SSC SD S&T Grand Challenge: A 5 year commitment to technology development SSC San Diego…on Point and at the Center of C4ISR

3. Tactical Situation Assessment (SA) Background • Based on text chat: Primary and often singularmeans of communicating situation updates on intermittent/ discontinuous networks: • Strikegroups: Establish hundreds of chatrooms, 2-4000 users in each “channel, ”based on functional roles; establish at beginning of mission, continuing for >18 months • Joint/Coalition Forces: Establish chatrooms with joint/coalition service participation, based on mission objectives; allows joint access to service-specific chatrooms to maintain non-intrusive situation awareness of service-specific activities • Functions: • Preserves interaction history • Supports edge users (on low/intermittent b/w) • Immediate COP (common operational picture) change and context updates • General information sharing updates on regular basis (across months) to establish battle rhythm management; • Supports Cross-Domain operations; those without the COP SSC San Diego…on Point and at the Center of C4ISR

4. R xxxxxxZ OCT 05 FM COMCARSTRKGRU TWELVE TO USJFCOM USPACOM ALLIED COMMAND TRANSFORMATION AC2C4ISR LANGLEY VIRGINIA DISA WASHINGTON DC SPAWAR SYSTEMS CENTER SAN DIEGO C NAVAL POSTGRADUATE SCHOOL MONTERY CA USS ENTERPRISE USS ANZIO INFO COMMANDER NAVAL NETWORK WARFARE COMMAND BT MSGID/GENADMIN/COMCARSTKGRU TWELVE/-/OCT// SUBJ/WELL DONE// RMKS/1. I WISH TO SEND NOTE OF PERSONAL THANKS FOR YOUR SUPPORT OF OUR JOINT/COALITION OPEN STANDARDS TACTICAL CHAT DEMONSTRATION ON 19 OCT 05. YOUR ORGANIZATIONS PROVIDED EXCEPTIONAL SUPPORT IN TERMS OF ENGINEERING, EQUIPMENT TO HOST THE TEST, AND PERSONNEL TO PARTICPATE AND PROVIDE FEEDBACK. THE TEST WENT EXCEEDINGLY WELL AND WOULD NOT HAVE BEEN POSSIBLE WITHOUT YOUR ASSISTANCE AND DEDICATION. SPECIAL APPRECIATION GOES TO: What Chat Looks Like Today The 50’s Defense Messaging (and Email): Produced Time-late Difficult to read Not used for real-time ops, but as formality Example Chat Log: Produced in real time Still difficult to read SSC San Diego…on Point and at the Center of C4ISR

5. Challenge: Creating the S&T • Objective: Enable Rapid Situation Assessment Updates through mixture (not engineering integration) of technologies (multi-disciplinary effort) in an area that is relatively new, but ubiquitous (and not very well respected…) • What is chat going to look like in 10 years? A New “’leet speak” • Must not be solely text-driven (too slow; too ambiguous); focus on visual and iconic images and gestures, integrated with text, I-18-N (Internationalization) • Must be able to accommodate edge (tactical) users • Chemical-Biological Warfighters in MOPP (Mission Oriented Protective Posture) gear • Special Operations in Inhospitable Settings: oil spills, fire, tsunamis, oxygen-deprived environments • Must be able to integrate with geographic land-based terrain maps AND non-geographic computer-based “terrain” (global network ops-network topology) • Must be able to incorporate object relationships and time in providing more precise context, mor complex information in smaller “space” • Must be small (in bandwidth) and robust (survive network drops) • Not necessarily keyboard-driven, point and click, nor restricted to PDAs • Embody the challenge of an NBIC (Nanotechnology/Biotechnology/ Information Systems/Cognitive Sciences) approach to development • Need to stay independent of commercial developments SSC San Diego…on Point and at the Center of C4ISR

6. Technical Approach: 3 Development Areas • Linguistic Research & Analysis • Improve the ability to mathematically derive the content of current chat messages, by defining candidate linguistic “themes/words” of military language, for transition to an augmented icon-based language, to improve efficiency of knowledge sharing • Visual and Iconic Language • Develop an iconic/visual (gesture) language to augment text, that can be used to more quickly communicate complex relationships and the evolution of relationships of objects of interest in geographic and non geographic environments; a true situation assessment “leetspeak” • Technology of Inscription • Prototype a keyboard-independenttechnology of inscription for this new language, for use in a Net-Centric environments SSC San Diego…on Point and at the Center of C4ISR

7. Linguistic Research & Analysis • Analyzing chatroom text represents a subset of a larger body of research : unstructured text analysis • Internet documents, blogs, emails, chatrooms • Military usage of chat is widespread, but inadequate for typical information retrieval and management • Statistical analysis of chat--its structure, topical organization, and user trends--must first be applied, few chatroom metrics exist • Acquire sizeable corpus of chat content for chat user database (domain specific/general; millions of lines of chat are required) • Statistically reduce content to significant words, phrases, concepts (within domain and in general) • Produce candidate list for iconic/visual representation SSC San Diego…on Point and at the Center of C4ISR

8. Visual And Iconic Language • Goal is to build a mixed-media situation assessment-specific chat-based lexicon (integrating words, pictures, gestures, with hardware), based on linguistic analysis • In order to: • Increase information carrying capacity • Increase expressiveness of the chat medium • Decrease ambiguity by using a formal iconography and shared context • Increase the ability to reliably share the same context • How can we best augment and disambiguate textual content using visual representations of language, objects, and information? SSC San Diego…on Point and at the Center of C4ISR

9. Visual and Iconic Language (cont) Method: • Gather knowledge of existing iconic language research, development, & design, both military and non-military • Develop software to support visual language communication • Develop icon-set based on linguistics results • Expand visual language dictionary and insert standard icon sets and acronyms • Choose gesture set and develop a visual mapping to items in the dictionary • Test icon suitability • Visual And Icon Language (VAIL) Conference, Aug 2-3, 2007: :  http://cs.unm.edu/~vail SSC San Diego…on Point and at the Center of C4ISR

10. Visual And Iconic Language SSC San Diego…on Point and at the Center of C4ISR

11. Technology of Inscription Prototype a keyboard-independent language-entry technology for net-centric warfare • Phase 1: Wireless data glove • Phase 2 : Visual & Icon-based language integrated/gesture-smart glove with 3-D modeling (requiring knowledge of biomechanics) • Phase 3: Miniaturization via nanotechnology-based circuitry, novel data entry technology • Keeping in mind that Nintendo Wii and Microsoft’s “Table Interface” will dominate commercial market SSC San Diego…on Point and at the Center of C4ISR

12. RONI GALGANO / Union-Tribune Evan Morikawa, a senior at High Tech High School, was one of four winners in the Greater San Diego Science & Engineering Fair for creating a chording keyboard glove that sends wireless signals to his personal digital assistant. http://www.g-speak.com/ Minority Report © 20TH Century Fox Many Parallel Developments to Inscription Technologies Microsoft Surface Table © SSC SD 2006 Nintendo Wii SSC San Diego…on Point and at the Center of C4ISR

13. Summary • 3 Areas of Research Encompassing NBIC principles • Multidisciplinary Team: • computational linguist, software architect, computer programming/science, electrical engineers, physicist, psychologist, 3-D modeler, chemist, iconic artist… • Long Range Applied Goals Drive Integrated Knowledge for unique direction and outcome • Not commercially driven, like business-based research • Not long-range, single-focus science, as in academics • Not a trivial problem, but a unique one… SSC San Diego…on Point and at the Center of C4ISR