1 / 27

Special considerations for Standards in Mobile AR

Special considerations for Standards in Mobile AR. International Workshop on AR Standards – 11. + 12. October 2010, Seoul Timo.Engelke@igd.fraunhofer.de - Institute for Graphics Research IGD. Fraunhofer IGD Institute for Computer Graphics Research.

dani
Download Presentation

Special considerations for Standards in Mobile AR

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Special considerations for Standards in Mobile AR International Workshop on AR Standards – 11. + 12. October 2010, Seoul Timo.Engelke@igd.fraunhofer.de - Institute for Graphics Research IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  2. Fraunhofer IGDInstitute for Computer Graphics Research • Mission: Technologies and Applications of Visualization, Interaction and Communication in the field of ICT • Director: Prof. Dr. techn. Dieter W. Fellner • 4 Locations: Darmstadt, Rostock (Germany), Singapore and Graz (Austria) • 180 Researchers (FTE) • 14,7 Mio. € Turnover • -> Department Virtual and Augmented Reality Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  3. Dep. Virtual & Augmented RealityResearch Topics Computer Vision Computer Graphics Interaction Technologies Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  4. Overview • The aim of the presentation is triggering consciousness about the idea of AR and presenting an open approach practically .. • Where are the problems in standardization of AR? • What are the common structures and where are variations? • What is the role of those structures in the context of AR? • Technological hardware issues • Why and what kind of standards are there and do we need? • Our approach to overcome the mentioned issues and help developing standards • Cannot cover the whole horizon of problems .. But can try … Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  5. Technologies ProblemsGeneral goal of having standard(s) which apply to general needs of AR. • What is the problem? • There are tracking technologies for AR, but they are in an early stage. Every application type needs its custom strategy, technology and specialized content. • What do we want to cover with a standard? • Due to the complexity of many parties involved in the field of AR and the high potential of possibilities. • The terminology AR is not clearly differentiated generally and can lead to confusion. • Which sense is augmented by what and through what? • 2D,3D, training, leading, informing, disinforming, helping, covering etc.. • Diverging hardware and software technologies • Bringing the paradigm to content masses Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  6. Arising potentials of technological developmentin contrast to commonly used technologies in apps Provide tools for everyone Layar, Wikitude etc.. Mass - Solution Common Technologies Customized Mass-Solution Branded… i.e. iLiving Common Technologies Development Common Techno-logies Time Cutting Edge-Solution Complexity Scientific Specific Laboratory-Solutions Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  7. Augmented RealityAbstract Building Blocks Sensors Output Context Information Annotation Engine Context Engine Interaction Device Interaction Engine Reference Model Annotation Model Application Model Content Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  8. Computer Vision Based Augmented Reality Screen Display Image Camera Image+Intrinsics Extrinsics (+ Intrinsics,Occlusions,Lights,Modelsetc..) Vision Engine 3D Annotation Engine Interaction Engine Reference Model Annotation Model Application Model Content Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  9. Role of Reference Model • Retrieval of the camera Pose (6Dof) • Find the absolute coordinates of user and objects • GPS, Compass, Acceleration Data • Use prototype model for registration (i.e. Poster, Marker, Lines…) • Representation • Absolute in latitude, longitude, altitude, roll, pitch, yaw • Relative in Cartesian coordinates and angles (or axis angle, quaternion..) • In relation to some object or object group – real or virtual? • Interaction • Find differences of formulated and real world (POI,AOI,VOI) • This creates potential content for interaction • Track movable and deformable objects • Detect user interaction/behavior using sensors (Hands, gestures etc..) • Allow feedback of generated data Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  10. Role of Visual Annotations in ARBuilding elements in GUIs • Visual Aids • Direct Visual Aids (DVA): Superimposed 3D-models/animations (direct association with object) • Indirect Visual Aids (IVA): “Annotations” related to tracked object (reminder) • Mental Model Building • User’s internal representation of the task • Can be improved by providing context information • Passive Parts • Information about the task in general • Higher-level instructions  global picture • Collaborative Annotations (expert trainer) DVA IVA Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  11. Role of the interaction model • Customizing the Annotation Model • Manipulation Scene Graph • Interactive Content • Customizing the Reference Model • Dynamically change tracking behavior and point of focus (POI) • Allow interchange in between Annotation and Registration Model • Use of declarative formulation as far as possible. • Allow scripting for specialization • Tracking and Events: • 2D-Projected Positions with distance hint • Detection Events (i,e. Object enters/leaves display) • Interaction of user/objects with the scene • Content must be delivered free or encrypted.. Who will do this? Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  12. Hardware technology issues on mobile devices • Low computational resources in relation to modern PCs • Limited memory • OpenGL ES • Different camera modules (Intrinsics) • Sensor support and calibration • Partly supported hardware acceleration • Software Interfaces and Languages • Symbian (C++, Microkernel, Qt) • Android (Java, C++ through NDK) • iPhone (Obj-C, C++ support) • Algorithms need to be optimized for HW • Costly and hard to maintain large set of handsets in one common Code-Base Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  13. Open StandardsSustainability for AR-Content • Applications and generation tool chains should be based on.. • X3D (ISO/IEC 19775:2004) • HTML5 • ECMAScript (ISO/IEC 16262:2002) • JavaScript • CSS • STEP-Format • KML • Benefits • Sustainability • Interoperability • Porting • Rapid development Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  14. X3DOM (x3dom.org)HTML5/X3D Integration Model N900 with miniB Simple car configurator to interactively change color and rims Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  15. X3DOM (x3dom.org)HTML5/X3D Integration Model • X3DOM: experimental open-source framework and runtime • One way for integration of HTML5 and declarative 3D • Ongoing discussion in the Web3D and W3C communities • Declarative content design • Allows embedding 3D data directly into the HTML-DOM • First approaches of CSS integration • Application concepts map well to generic scene-graph • Runtime independent (supports native implementation, plugin, WebGL) • Suitable for desktop and mobile systems alike Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  16. instantReality – mobileAR - Browser Extendable generic AR-Web-Browser for mobile devices based on HTML5 and Javascript Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  17. mobileAR – Open Architecture Proposal for open realization of declarative approaches WEB-View (HTML5 +JS) Individualization of App Abstraction IMA DOM-Declarative Reachability for masses of developers AR-KML ?? Open Annotation Open Context SVG X3Dom A2ML JavaScript Render Sound Vision Sensors Content webGL Native Interface OS Generic Application (arBrowser) Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  18. mobileAR – AR – Browser for mobile devicesAdditionally to Web-Browser a renderer and vision engine is integrated. • Extends the internal Web-Browser with a simple but powerful API • Interaction and visualization can be done declarative in HTML and/or via JS • Can render X3D via native renderer or even X3Dom • Generic Vision Interface • Integrates Vision Processor (VisionLib) for • Fiducials (Markers, Image-Markers) • Poster-Tracking • Object recognition • Panorama and 3D-Model Based scene tracking • Server based Tracking (bidirectional streaming) • Additional Features (about to come…) • Remote Video & Audio Support Scenarios • Encryption and packaging WEB-View HTML5+JS optional Debug View 3D-View X3D Camera Live/Paused Image Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  19. Next steps… • Identification and unification of common technology data • Annotation Data • Reference Data • Create open framework for declarative approaches • Open Annotation Engine • Open Context Engine  Unify Vision and Sensor based approaches • KML in the context of industrial maintenance applications • Evaluate using show-cases • Extend native interfaces • Incooperate Sound  A2ML • Better content management (Offline Usage, Encryption) • Make it free for masses for developing Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  20. Happy Workshopping THANK YOU ! Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  21. Building Block Hierarchy – What data to processContext Engine • Based on available sensors in device Context Engine Localization Engine Other sensors.. Vision Engine Auditive Engine Geolocation Engine Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  22. Building Block Hierarchy - DerivativesContext Engine • Based on capabilities of device Annotation Engine 3D-Annotation Engine Sound-Annotation Engine 2D-Annotation Engine Video Engine Auditive Engine Geolocation Engine Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  23. Parties involved in the game of AR Physical Player …some of them want to • The user of the app • Network Provider • Producer of Cellphones • Application Developer • Content developer/provider • Technology developer • AR Standardization • Learn something • Network • Save costs • Save resources • Save time • Earn money • Have fun • Sustainability Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  24. The user of the app Network Provider Producer of Cellphones Application Developer Content developer/provider Technology developer AR Standardization Identification of needs... Device Creator Communication Provider Content Provider User(s) Techno-logy Devel Content Creator App Devel Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  25. Applications (Fallback Model in Action) • IE8 Plugin (OpenGL and Raytracing) and native iOS app with x3dom layer • Runtime independent – supports (native implementation,) plugin, and WebGL Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  26. Geo Located - Augmented RealityAbstract Building Blocks Localization Sensors Output Position Annotation Engine Localization Engine Interaction Device Interaction Engine Reference Model Annotation Model Application Model Content Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

  27. Geo located and Vision Based Augmented RealityTechnically Abstract Building Blocks Screen Source: Layar Display Image Camera Localization Sensors Intrinsics Extrinsics 2D Annotation Engine Localization Engine Interaction Engine Reference Model Annotation Model Application Model Content Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea

More Related