270 likes | 381 Views
Special considerations for Standards in Mobile AR. International Workshop on AR Standards – 11. + 12. October 2010, Seoul Timo.Engelke@igd.fraunhofer.de - Institute for Graphics Research IGD. Fraunhofer IGD Institute for Computer Graphics Research.
E N D
Special considerations for Standards in Mobile AR International Workshop on AR Standards – 11. + 12. October 2010, Seoul Timo.Engelke@igd.fraunhofer.de - Institute for Graphics Research IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Fraunhofer IGDInstitute for Computer Graphics Research • Mission: Technologies and Applications of Visualization, Interaction and Communication in the field of ICT • Director: Prof. Dr. techn. Dieter W. Fellner • 4 Locations: Darmstadt, Rostock (Germany), Singapore and Graz (Austria) • 180 Researchers (FTE) • 14,7 Mio. € Turnover • -> Department Virtual and Augmented Reality Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Dep. Virtual & Augmented RealityResearch Topics Computer Vision Computer Graphics Interaction Technologies Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Overview • The aim of the presentation is triggering consciousness about the idea of AR and presenting an open approach practically .. • Where are the problems in standardization of AR? • What are the common structures and where are variations? • What is the role of those structures in the context of AR? • Technological hardware issues • Why and what kind of standards are there and do we need? • Our approach to overcome the mentioned issues and help developing standards • Cannot cover the whole horizon of problems .. But can try … Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Technologies ProblemsGeneral goal of having standard(s) which apply to general needs of AR. • What is the problem? • There are tracking technologies for AR, but they are in an early stage. Every application type needs its custom strategy, technology and specialized content. • What do we want to cover with a standard? • Due to the complexity of many parties involved in the field of AR and the high potential of possibilities. • The terminology AR is not clearly differentiated generally and can lead to confusion. • Which sense is augmented by what and through what? • 2D,3D, training, leading, informing, disinforming, helping, covering etc.. • Diverging hardware and software technologies • Bringing the paradigm to content masses Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Arising potentials of technological developmentin contrast to commonly used technologies in apps Provide tools for everyone Layar, Wikitude etc.. Mass - Solution Common Technologies Customized Mass-Solution Branded… i.e. iLiving Common Technologies Development Common Techno-logies Time Cutting Edge-Solution Complexity Scientific Specific Laboratory-Solutions Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Augmented RealityAbstract Building Blocks Sensors Output Context Information Annotation Engine Context Engine Interaction Device Interaction Engine Reference Model Annotation Model Application Model Content Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Computer Vision Based Augmented Reality Screen Display Image Camera Image+Intrinsics Extrinsics (+ Intrinsics,Occlusions,Lights,Modelsetc..) Vision Engine 3D Annotation Engine Interaction Engine Reference Model Annotation Model Application Model Content Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Role of Reference Model • Retrieval of the camera Pose (6Dof) • Find the absolute coordinates of user and objects • GPS, Compass, Acceleration Data • Use prototype model for registration (i.e. Poster, Marker, Lines…) • Representation • Absolute in latitude, longitude, altitude, roll, pitch, yaw • Relative in Cartesian coordinates and angles (or axis angle, quaternion..) • In relation to some object or object group – real or virtual? • Interaction • Find differences of formulated and real world (POI,AOI,VOI) • This creates potential content for interaction • Track movable and deformable objects • Detect user interaction/behavior using sensors (Hands, gestures etc..) • Allow feedback of generated data Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Role of Visual Annotations in ARBuilding elements in GUIs • Visual Aids • Direct Visual Aids (DVA): Superimposed 3D-models/animations (direct association with object) • Indirect Visual Aids (IVA): “Annotations” related to tracked object (reminder) • Mental Model Building • User’s internal representation of the task • Can be improved by providing context information • Passive Parts • Information about the task in general • Higher-level instructions global picture • Collaborative Annotations (expert trainer) DVA IVA Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Role of the interaction model • Customizing the Annotation Model • Manipulation Scene Graph • Interactive Content • Customizing the Reference Model • Dynamically change tracking behavior and point of focus (POI) • Allow interchange in between Annotation and Registration Model • Use of declarative formulation as far as possible. • Allow scripting for specialization • Tracking and Events: • 2D-Projected Positions with distance hint • Detection Events (i,e. Object enters/leaves display) • Interaction of user/objects with the scene • Content must be delivered free or encrypted.. Who will do this? Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Hardware technology issues on mobile devices • Low computational resources in relation to modern PCs • Limited memory • OpenGL ES • Different camera modules (Intrinsics) • Sensor support and calibration • Partly supported hardware acceleration • Software Interfaces and Languages • Symbian (C++, Microkernel, Qt) • Android (Java, C++ through NDK) • iPhone (Obj-C, C++ support) • Algorithms need to be optimized for HW • Costly and hard to maintain large set of handsets in one common Code-Base Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Open StandardsSustainability for AR-Content • Applications and generation tool chains should be based on.. • X3D (ISO/IEC 19775:2004) • HTML5 • ECMAScript (ISO/IEC 16262:2002) • JavaScript • CSS • STEP-Format • KML • Benefits • Sustainability • Interoperability • Porting • Rapid development Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
X3DOM (x3dom.org)HTML5/X3D Integration Model N900 with miniB Simple car configurator to interactively change color and rims Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
X3DOM (x3dom.org)HTML5/X3D Integration Model • X3DOM: experimental open-source framework and runtime • One way for integration of HTML5 and declarative 3D • Ongoing discussion in the Web3D and W3C communities • Declarative content design • Allows embedding 3D data directly into the HTML-DOM • First approaches of CSS integration • Application concepts map well to generic scene-graph • Runtime independent (supports native implementation, plugin, WebGL) • Suitable for desktop and mobile systems alike Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
instantReality – mobileAR - Browser Extendable generic AR-Web-Browser for mobile devices based on HTML5 and Javascript Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
mobileAR – Open Architecture Proposal for open realization of declarative approaches WEB-View (HTML5 +JS) Individualization of App Abstraction IMA DOM-Declarative Reachability for masses of developers AR-KML ?? Open Annotation Open Context SVG X3Dom A2ML JavaScript Render Sound Vision Sensors Content webGL Native Interface OS Generic Application (arBrowser) Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
mobileAR – AR – Browser for mobile devicesAdditionally to Web-Browser a renderer and vision engine is integrated. • Extends the internal Web-Browser with a simple but powerful API • Interaction and visualization can be done declarative in HTML and/or via JS • Can render X3D via native renderer or even X3Dom • Generic Vision Interface • Integrates Vision Processor (VisionLib) for • Fiducials (Markers, Image-Markers) • Poster-Tracking • Object recognition • Panorama and 3D-Model Based scene tracking • Server based Tracking (bidirectional streaming) • Additional Features (about to come…) • Remote Video & Audio Support Scenarios • Encryption and packaging WEB-View HTML5+JS optional Debug View 3D-View X3D Camera Live/Paused Image Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Next steps… • Identification and unification of common technology data • Annotation Data • Reference Data • Create open framework for declarative approaches • Open Annotation Engine • Open Context Engine Unify Vision and Sensor based approaches • KML in the context of industrial maintenance applications • Evaluate using show-cases • Extend native interfaces • Incooperate Sound A2ML • Better content management (Offline Usage, Encryption) • Make it free for masses for developing Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Happy Workshopping THANK YOU ! Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Building Block Hierarchy – What data to processContext Engine • Based on available sensors in device Context Engine Localization Engine Other sensors.. Vision Engine Auditive Engine Geolocation Engine Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Building Block Hierarchy - DerivativesContext Engine • Based on capabilities of device Annotation Engine 3D-Annotation Engine Sound-Annotation Engine 2D-Annotation Engine Video Engine Auditive Engine Geolocation Engine Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Parties involved in the game of AR Physical Player …some of them want to • The user of the app • Network Provider • Producer of Cellphones • Application Developer • Content developer/provider • Technology developer • AR Standardization • Learn something • Network • Save costs • Save resources • Save time • Earn money • Have fun • Sustainability Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
The user of the app Network Provider Producer of Cellphones Application Developer Content developer/provider Technology developer AR Standardization Identification of needs... Device Creator Communication Provider Content Provider User(s) Techno-logy Devel Content Creator App Devel Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Applications (Fallback Model in Action) • IE8 Plugin (OpenGL and Raytracing) and native iOS app with x3dom layer • Runtime independent – supports (native implementation,) plugin, and WebGL Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Geo Located - Augmented RealityAbstract Building Blocks Localization Sensors Output Position Annotation Engine Localization Engine Interaction Device Interaction Engine Reference Model Annotation Model Application Model Content Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Geo located and Vision Based Augmented RealityTechnically Abstract Building Blocks Screen Source: Layar Display Image Camera Localization Sensors Intrinsics Extrinsics 2D Annotation Engine Localization Engine Interaction Engine Reference Model Annotation Model Application Model Content Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea