1 / 53

Development of a Tracking Method for Augmented Reality Applied to Nuclear Plant Maintenance Work

Development of a Tracking Method for Augmented Reality Applied to Nuclear Plant Maintenance Work. Presentation by Hi rotake Ishii Research Associate, Kyoto University, Japan Guest Scientist, Institute for Energy Technology, Norway.

mayten
Download Presentation

Development of a Tracking Method for Augmented Reality Applied to Nuclear Plant Maintenance Work

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Development of a Tracking Method for Augmented Reality Applied toNuclear Plant Maintenance Work Presentation by Hirotake Ishii Research Associate, Kyoto University, Japan Guest Scientist, Institute for Energy Technology, Norway

  2. Need further development of hardware and software for NPP operation Background • Serious situation of NPP • Both of improvement of safety and reduction of cost are required. • Introduction of free electricity market. • Difficulties of maintenance for aged NPPs. • Decrease of expert maintenance workers. • Support for maintenance work in plant field • There are some rooms to improve its efficiency and safety by introducing state-of-the-art information technologies. • Augmented Reality (AR) is one of the promising technologies that can improve efficiency and safety.

  3. UP UP UP HMD Paper Manual AR What is the AR? • Augmented Reality(AR) expands the surrounding real world by superimposing computer-generated information on the user’s view. • By using Augmented Reality, it becomes possible for the workers to understand various information of the maintenance work intuitively. Superimposed information Destination

  4. can not be used in NPP Key Technology of the AR • Tracking • Measure the position and rotation of user’s view in real time to superimpose virtual object/information at correct position. • Many kinds of tracking methods are developed • GPS (Differential GPS, Real Time Kinematics GPS) • Ultrasonic/Magnetic/Infrared Sensor • Inertial Sensor • Artificial Marker / Marker-less • Hybrid of above (Combination)

  5. The artificial marker technique has a possibility to be used in a plant. Requirement of Tracking Method applied to NPP • Limitations of NPP field from the viewpoint of tracking • Indoor • Various size of equipment in wide area • Lots of metal objects / obstacles / magnetic source • Requirement • Wide area • Accuracy and stability • Easy / no installation • Less expensive

  6. Artificial Marker Technique • Artificial Marker Technique • Calculates the position and rotation of the camera from the position of markers pasted in the environment by image processing technique. • Is applied to many AR applications. • Strengths • Accurate and stable • Less expensive • High scalability • Weaknesses • Available only when the distance between the marker and the camera is short (this means many markers need to be pasted in the environment) or the size of the markers is large.

  7. It is necessary to make the markers smaller or make it easier to be pasted in a complicated environment Available Distance of Artificial Marker • Maximum distance • ARToolKit (VGA, f=4mm) 1m : 8 cm 3m : 25 cm 5m : 40 cm • Problem • There are many small objects like pipes in a plant. • The surface of the objects is not flat. • It is difficult to paste large markers.

  8. Development of a New Tracking System • Tracking system using barcode marker • Long barcode marker • Easy to paste on pipes • Tracking system using circular marker • Circle shaped marker with code inside • Stable recognition in long distance

  9. Tracking System using Barcode Marker by Hiroshi Shimoda, Hirotake Ishii, Masayuki Maeshima, Toshinori Nakai, Zhiqiang Bian and Hidekazu Yoshikawa, (Kyoto University)

  10. Code”1” Code”0” Gap 40mm 80mm 20mm 40mm Humming code (4 bits) ID of marker (7 bits) Design of Barcode Marker • 11 bits data • 7 bits for ID (128 kinds of markers) • 4 bits for error correction (Humming code) • Length • From 640 mm to 1080 mm

  11. Worker’s view Instruction information by AR Check the crack of the upper pipe. HMD Small Camera Field Worker Barcode Marker Image of Tracking Method with Barcode Marker • Barcode markers are pasted on pipes • 3 position (both edge and middle point) of each marker are registered in advance • Position and rotation of workers are calculated by using 2 barcode markers(Solving P6P problem)

  12. Marker Recognition Algorithm (1) Captured Image:

  13. Marker Recognition Algorithm (2) Binarization: • Binarize the captured image with the camera at preset threshold level.

  14. Marker Recognition Algorithm (3) Labelling: • Collect the connected pixels and mark a unique label on the connected part.

  15. Marker Recognition Algorithm (4) Narrowing search area: • Exclude the parts which have no possibility as the part of the marker by its area and shape.

  16. Marker Recognition Algorithm (5) Extraction parts of marker: • Pick up the 11 parts which are arranged in a line as a candidate of barcode marker.

  17. Marker Recognition Algorithm (6) Decision of code: • Decide the code of barcode marker from the area of each part. 1 0 0 0 1 1 0 0 0 1 0

  18. Marker Recognition Algorithm (7) Comparison with pre-registered barcode marker: • Correct the code of the marker with Humming code part and compare it with pre-registered barcode marker.

  19. Marker Recognition Algorithm (8) Calculate position and rotation of camera • Extract 3 points from 2 barcode markers and solve P6P problem

  20. Basic Evaluation of Marker Recognition • Purpose • Evaluate basic ability of proposed tracking method • Experimental Method Rotation Rotation Camera Distance 120 lux Pipe arrangement: Horizontal / Vertical Rotation: 0 / 20 / 40 / 60 / 80 degree Distance: 1 / 2 / 3 / 4 / 5 / 6 meters Pipe: 60φ×1100 mm

  21. Vertical, 0 degree, 6 meters Vertical, 80 degree, 4 meters Examples of Captured Images Camera Resolution : H320xV240

  22. Recognition Result Camera resolution: 320 x 240 Viewing angle: 53 x 40 degree Recognition rate: 10 – 30 fps (Pentium Mobile 1.4GHz)

  23. Water Purification Facility 10 barcode markers pasted in the area All marker ID and positions were registered in advance. Trial Use in Fugen NPP

  24. Recognized Markers Example

  25. Result • Walked around the area with prototype system. • 1000 frame images were picked up, in which at least one marker was in the view. • Recognition rate: 52.8% (47.2% failed)Erroneous recognition rate: 1.8% • Cases of erroneous recognition • a marker image was captured against the light, • a marker was too far from the camera, and • the angle of a marker against the camera direction was too large.

  26. Example of Erroneous Recognition (Backlight)

  27. Conclusion (Barcode Marker) • Proposed marker-based tracking method for AR support of NPP maintenance work. • Long barcode marker and simple image recognition. • Evaluated basic ability of proposed tracking method in a laboratory. • Long distance, enough fast and feasible. • Trial use in Fugen NPP • Recognition rate: 52.8%, erroneous recognition rate: 1.8%

  28. Tracking System using Circular Marker by Hirotake Ishii, Hidenori Fujino (Kyoto University) Asgeir Droivoldsmo (Institute for Energy Technology)

  29. 1. Detect edges Distance between features 2. Detect 4 lines Before binarize 3. Calculate intersections Max half meters • Easily affected by jaggy shaped edges After binarize Real line • Distance between features is short(Max to size of marker) • Accuracy depends on the distance between the features. • ●● • ●●●●●●● • ●●●●●● • ●● • ●● • ●● • ●●● • ●● • ●● • ●●● • ●●●●●● • ●●●●● Estimated line HIRO Weakness of Square Markers • How to recognize square markers 4. Calculate position and rotation

  30. The center can be recognized accurately in any situation • Triangulation by plural markers can be used. Badly focused Well focused Long distance Distance between features can be long. Distance between features Max several meters In Case of Circular Marker • The center can be recognized accurately in any situation

  31. Outer black circle Middle code area circle which consists of black or white fans Center white circle Design of Circular Marker • Make as simple as possible • Outer black circle and center white circle are used for calculating the threshold to analyze the code area. • Diameter of the marker and the number of division of middle circle can be changed according to the situation.

  32. Marker Recognition Algorithm (1) • Captured Image

  33. Marker Recognition Algorithm (2) • Calculation of logarithm

  34. Marker Recognition Algorithm (3) • 3x3 Sobel Filter

  35. Marker Recognition Algorithm (4) • Labeling

  36. Marker Recognition Algorithm (5) • Eliminate small area

  37. Marker Recognition Algorithm (6) • Recognize ellipse and eliminate non-ellipse area

  38. Marker Recognition Algorithm (7) • Recognize ID of marker and eliminate non-marker area 7 3 21 • Details are in the paper

  39. Marker Recognition Algorithm (8) • Result 7 3 21

  40. Marker position on the image Marker shape on the image PnP solver Single but unaccurate solution Accurate but plural solutions Compare Single and accurate solution Calculate position and rotation of camera • Calculate by using both of the solutions from PnP solver and the rough information of each circular marker’s rotation (Details are in the paper)

  41. Marker Location (XML) Marker Design Marker Maker Printed Markers Marker Visualize Printer Tracking Server Camera Driver PGR Link Link Link DragonFly FireFly Camera Driver CMU Tracking Result (Via TCP/IP) IEEE1394 (IIDC) Display Software OpenGL DirectX Under development Direct Show JAVA3D JNI USB, DV Developed Software Camera Control library (C++) Core library (C++) TCP/IP library for Server TCP/IP DLL for Client

  42. Evaluation of the accuracy and stability (Single marker) • Experimental Method • One circular marker (diameter is 40mm) was pasted on a small box. • The number of the division of middle circle was 8,9,10. • Distance was changed from 300cm to 580cm with 20cm step. • Angle was changed from 0 degree to 75 degrees with 15 degrees step. • For each condition, 100 images were captured and the position and rotation was calculated. The average and the variance were calculated.

  43. Example of the image Camera resolution: H1024 x V768 Focal Length: 8mm (Distance:560cm, Angle:0 degree)

  44. Number of Division Angle (deg.) 0 15 30 45 60 75 Table : Maximum distance (diameter is4cm) 8 520/560 520/560 500/560 440/520 380/440 X/300 9 520/560 520/560 500/560 440/520 380/440 X/X 10 520/560 520/560 500/560 400/500 380/420 X/X Succeeded in all frames(cm) / Failed in some frames(cm) Results (Maximum distance) • Maximum distance of the circular marker is about 2 times of the square marker such as ARToolKit Camera resolution: H1024 x V768 Focal Length: 8mm

  45. Results (single marker, depth) • The accuracy of the position is not good.

  46. Results (single marker, rotation)

  47. Evaluation of the accuracy and stability (plural markers on a helmet) • Experimental Method • 22 circular markers were pasted on a helmet. • The number of the divisionof middle circle was 8. • Distance was changed from 300cm to 560cm with 20cm step. • Angle was changed from 0 degree to 180 degrees with 15 degrees step. • For each condition, 100 images were captured and the position and rotationwas calculated. The average and the variance were calculated.

  48. Example of the image Camera resolution: H1024 x V768 Focal Length: 8mm (Distance:560cm, Angle:120 degree)

  49. Results (Position) • The accuracy is greatly improved Black : plural Markers Red : Single Marker

  50. Results (Rotation) • Tracked in wide angle. In the case of single marker, maximum angle is about 60 degrees.

More Related