Augmented-Reality

1. Augmented-Reality Ping Gai HFE 760

2. Augmented-Reality • Augmented- Reality Definition • Augmented Reality vs. Virtual Reality • Visual Display Systems for AR • Video Keying and Image Registration • System Design Issues • Augmented Reality Application

3. Augmented Reality Definition • Augmented Reality is a growing area in virtual reality area. • An Augmented Reality system generates a composite view for the user. It’s a combination of the real scene viewed by the user and a virtual scene generated by the computer that augments the scene generated by the computer that augmented the scene with additional information.

4. Augmented Reality Definition • Typically, the real-world visual scene in an AR display is captured by video or directly viewed. • Most current AR displays are designed using see-through HMDs which allow the observer to view the real world directly with the naked eye. • If video is used to capture the real world, one may use either an opaque HMD or screen-based system to view the scene.

5. AR vs. VR • Virtual Reality: a computer generated, interactive, three-dimensional environment in which a person is immersed.(Aukstakanis and Blatner, 1992) • Virtual Environment is a computer generated three dimensional scene which requires high performance computer graphics to provide an adequate level of realism. • The virtual world is interactive. A user requires real-time response from the system to be able to interact with it in an effective manner. • The user is immersed in this virtual environment.

6. AR vs . VR • VR: the user is completely immersed in an artificial world and becomes divorced from the real environment. The generated world consists entirely of computer graphics.

7. AR vs. AR • VR strives for a totally immersive environment. The visual, and in some systems aural and sense are under control of the system. • In contrast, an AR system is augmenting the real world sense of presence in that world. The virtual images are merged with the real view to create the augmented display.

8. AR vs. VR • For some applications , it may be desirable to use as much as possible real world in the scene rather creating a new scene using computer imagery. For example, in medical applications, the physician must view the patient to perform surgery, in telerobotics the operator must view the remote scene in order to perform tasks.

9. AR vs. VR • A main motivation for the use of AR relates to the computational resources necessary to generate and update computer-generated scene. In VR, The more complex the scene, the more computational resource needed to render the scene. • AR can maintain the high-level of detail and realistic shading that one finds in the real world.

10. AR vs. VR • NO simulator sickness. Vertigo, dizziness introduced by sensory mismatch within display environment can be a problem when one uses an HMD to view a virtual world. • If the task is to show an annotation to the real world.

11. Visual Display System for AR • Hardware for display visual images • A position and orientation sensing system • Hardware for combining the computer graphics and video images into one signal • The associated system software

12. Visual Display System for AR • There are two main ways in which the real world and the computer generated imagery may be combined to form an augmented scene. • Direct viewing of the real world with overlaid computer generated imagery as an enhancement.In this case, the the real world and the CG images are combined optically. • Combining the camera-captured video of the real world with CG imagery viewed using either an opaque HMD, or a screen-based display system.

13. Visual Display System for AR • Two basic types of AR system • Opaque HMD or screen-based AR. These systems can be used to view local or remote video views of real world scenes, combined with overlaid CG.The viewing of a remote scene is an integral component of telepresence applications. • Transparent HMD AR. This system allows the observer to view the real world directly using half-silvered mirrors with CG electronically composited into the image. An advantage id that the real-world can be directly viewed and manipulated.

14. Visual Display System for AR

15. Video Keying • Relevant when an opaque HMD with video input is used to create an AR scene. Video and synthetic image are mixed using a video keyer to form an integrated scene. • Video Keying is a process that is widely used in television, film production and CG. (weather report)

16. Video Keying • When using video keying to design AR scenes, one signal contains the foreground image and the other one contains the background image. The keyer combines the two signal to produce a combined video which is then sent to the display device.

17. Video Keying • Keying can be done using composite or component video signals. • A composite video signal contains information about color, luminance, and synchronization, thus combining three piece of information into one signal. • With component video, luminance synchronization are combined, but chroma information is delivered separately.

18. Video Keying • Chroma keying involves specifying a desired foreground key color. Foreground areas containing the keying color are then electronically replaced with the background image. This results in the background image being replaced with the fore ground image in areas where the background image contains chroma color. • Blue is typically used for chroma keying (Chromakey blue) rarely shows up in human skin tones.

19. Video Keying • If a video image of the real world is chosen as the foreground image, parts of the scene that should show the computer-generated world are rendered blue. • In contrast, if video of the real world is chosen as the background image, the computer generated environment will be located in the foreground.

20. Video Keying

21. Video Keying • A luminance keyer works in a similar manner to a chroma keyer, however, a luminance keyer combines the background image wherever the luminance values are below a certain threshold. • Luminance and chroma keyers both accomplish the same function but usa of a chroma keyer can result in a sharper key and has greater flexibility, whereas a luminance keyer is typically lower resolution and had less flexibility.

22. Z-keying

23. Z-keying • Figure is a schema of the z-key method. The z-key method requires images with both depth information (depth map) as inputs. The z-key switch compares depth information of two images for each pixel, and connects output to the image which is the nearer one to the camera. The result of this is that real and virtual objects can occlude each other correctly. This kind of merging is impossible by the chroma-key method, even if it is accompanied with some other positioning devices such as magnetic or acoustic sensor, since these devices provide only a gross measurement of position.

24. Image Registration • It’s required that the computer generated images accurately register with the surroundings in the real world. In certain applications, image registration is crucial. • In terms of developing scenes for AR displays, the problem of image registration, or positioning of the synthetic objects within the scene in relation to real objects, is both a difficult and important technical problem to solve.

25. Image Registration • With applications that require close registration, accurate depth information has to be retrieved from the real world in order to carry out the calibration of the real and synthetic environments. Without an accurate knowledge of the geometry of the real world and computer-generated scene, exact registration is not possible.

26. System Design Issues • Frame rate, update rate, system delays, and the range and sensitivity of the tracking sensors. • Frame rate is a hardware-controlled variable determining the number of images presented to the eye per second. AR displays which show stereo images alternatively to the left and right eye typically use a scan rate doubler to transmit 120 frames per second so that each eye has an effective frame rate of 60 Hz.

27. System Design Issues • Update rate of the display is the rate at which new images are presented to the viewer. • With a low update rate, if the user using an AR display moves his head, the real and computer-generated images will no longer be registered until the next update. Small errors in registration are easily detectable by the visual system. • What limits the update rate is the relationship between the complexity of the scene and the computational power of the computer system used to generate the scene. This relationship is esp. important for computationally intensive applications such as medical imaging.

28. System Design Issues • The lag in image generation and tracking is noticeable in all HMDs but is dramatically accentuated with see-through HMDs. This is an crucial problem if exact image registration is required. • There are two types of system delays which will affect performance in AR: computational and sensor delays. • As the complexity of the CG image increases, the computational delay is a major factor determining the update of a display. • In addition, sensor delay, the time requires updating the display, is an important variable in determining performance in augmented reality. • Many HMD-based systems have combined latencies over 100ms, which become very noticeable.

29. System Design Issues • Sensor sensitivity • The head-tracking requirements for AR displays. • A tracker must be accurate to a small fraction of a degree in orientation and a few millimeters in position. • Errors in head orientation(pitch, roll, yaw) affect image registration more so than error in position(x, y, z), leading to the more stringent requirements for head-orientation tracking. • Positional tracking errors of no more than 1 to 2 mm are maximum for AR system.

30. System Design Issues • In addition to visual factors, cognitive factors should be considered in the design as well. • Users of systems form mental models of the system they interact with and the mental model they form influence their performance. • With AR displays the designer must take into account two mental models of the environment, the mental model of the synthetic imagery and of the real image. • The challenge will be to integrate the two stimuli in such a way that a single mental model will be formed of the augmented scene.

31. System Design Issues Integrated Mental Model Mental Model of synthetic envoronment Mental Model of real envoronment Virtual world stimuli Auditory, haptic, visual Real world stimuli Auditory, haptic, visual

32. Augmented-Reality Application • Medical • Entertainment • Military Training • Engineering Design • Robotics and Telerobotics • Manufacture, Maintenance and Repair • Consumer Design

33. Augmented-Reality Application • Medical Most of the medical applications deal with image guided surgery. Pre-operative imaging studies, such as CT or MRI scans, of the patient provide the surgeon with the necessary view of the internal anatomy. From these images the surgery is planned. Visualization of the path through the anatomy to the affected area where, for example, a tumor must be removed is done by first creating a 3D model from the multiple views and slices in the preoperative study. AR can be applied so that the surgical team can see the CT or MRI data correctly registered on the patient in the operation theater while the procedure is progressing. Being able to correctly register the images at the point will enhance the performance of the surgical team and eliminate the need for the painful and cumbersome stereotactic frames currently used for registration.

34. Augmented-Reality Application

35. Augmented-Reality Application • Entertainment • Weather report • Virtual studio • Movie special effect • Advertisement

36. Augmented-Reality Application • Military Training The military has been using display in cockpits that present information to the pilot on the windshield of the cockpit or the visor of their flight helmet. This is a form fo AR display.

37. Augmented-Reality Application • Engineering Design Distributed Coollaberation Product visualizatoin The scenario for this application consists of an office manager who is working with an interior designer on the layout of a room. The office manager intends to order furniture for the room. On a computer monitor the pair see a picture of the room from the viewpoint of the camera. By interacting with various manufacturers over a network, they select furniture by querying databases using a graphical paradigm. The system provides descriptions and pictures of furniture that is available from the various manufactures who have made models available in their databases. Pieces or groups of furniture that meet certain requirements such as colour, manufacturer, or price may be requested. The users choose pieces from this "electronic catalogue" and 3D renderings of this furniture appear on the monitor along with the view of the room. The furniture is positioned using a 3D mouse. Furniture can be deleted, added, and rearranged until the users are satisfied with the result; they view these pieces on the monitor as they would appear in the actual room. As they move the camera they can see the furnished room from different points of view.

38. Augmented-Reality Application • Robotics and Telerobotics

39. Augmented-Reality Application • Manufacturing, Maintenance, and Repair • One application area that is currently being explored involves mechanical maintenance and repair. In this scenario a mechanic is assisted by an AR system while examining and repairing a complex engine. The system may present a variety of information to the mechanic. Annotations may identify the name of parts, describe their function, or present other important information like maintenance or manufacturing records. AR may lead the mechanic through a specific task by highlighting parts that must be sequentially removed and showing the path of extraction. The system may also provide safety information. Parts that are hot or electrified can be highlighted to constantly remind the mechanic of the danger of touching them. The mechanic may also be assisted by a remote expert who can control what information is displayed on the mechanic's AR system.

40. Augmented-Reality Application • Consumer Design House Design Fashion, beauty industry ….

41. Reference • http://www.cs.rit.edu/~jrv/research/ar/ • Virtual Environments and Advanced Interface Design, edited by Woodrow Barfield, Thomas A.Furness III

42. Augmented Reality Sites - North America • MIT • Image Guided Surgery home page • Intelligent Room project • J P Mellor's home page • Media Lab Wearable Computer page • CMU • Z-Key project • Magic Eye project • Columbia University • Virtual Worlds research • Architectural Anatomy • University of North Carolina - Chapel Hill • Ultrasound Visualization Research • Hybrid Tracking Research • Latency in Augmented Reality • Ronald Azuma's Augmented Reality page • Telepresence Research Group • Rich Holloway's Home Page • USC Computer Graphics and Immersive Technologies Laboratory • University of Washington Human Interface Technology Lab (HITL) • Colorado School of Mines • Hazardous waste management Bozidar Stojadinovi, Virtual Reality Lab, University of Michigan • Augmented Reality work at the University of Toronto • Argonne National Labs - Michael E. Jebb's Augmented Reality page • NIST description of the Boeing project • Colorado State Univ. - Michael L. Croswell's Augmented Reality page • Ross Whitaker's Augmented Reality page • Mihran Tuceryan's Augmented Reality page • The WorldBoard Project • Vision-based Augmented Reality for Guiding Assembly Rajeev Sharma, Jose Molineros, University of Illinois at Urbana-Champaign • Alexander Chislenko's Intelligent Information Filters and Enhanced Reality Page • Microvision's Virtual Retinal Display