By Syed Mohammed Shamsul Islam Lecturer B and PhD student Department of Computer Engineering KFUPM

1. A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System By Syed Mohammed Shamsul Islam Lecturer B and PhD student Department of Computer Engineering KFUPM Co-authors: • Prof. Mayez Al-Mouhamed • Dr. S. M. Buhari • Dr. Talal Al Kharobi

2. Outline • Introduction • Design Issues • Implementation • Performance Evaluation • Conclusions & Recommendations

3. Telerobotics: • A scheme that allows humans to extend their manipulative skills over a network. Visual feedback Visual feedback Force feedback Force feedback

4. Telerobotic Applications • Hazardous situations • e.g. bomb disposal • Scaled-down and scaled-up situations • e.g. micro-surgery • Situations affected by human presence • e.g. closed-chest heart bypass • Teaching, training, maintenance, entertainment.

5. Problem of Conventional Tele-robotics • Time delay: • Processing, copying video data, network delays • Move-n-Wait strategy • Operator has to follow trial and error method to perform a task, which might be dangerous for example in telesurgery.

6. Improving Performance by Using AR • By overlaying virtual image with the real image operator e.g. surgeon can make a plan before going for actual teleoperation  safety. • Rehearsal and correction can be made in the simulation plan. • Saving bandwidth by sending (less frequently) only the planned data.

7. Bi-directional one-to-one mapping between graphic and real co-ordinate spaces Graphic Co-ordinate Space Remote Co-ordinate Space (Video) Challenges of AR • Graphical robot arm must mach with real robot arm in the video.

8. Challenges of AR • Camera Calibration: • Finding accurate camera calibration is challenging. • Viewpoint Registration

9. Design Issues

10. Telerobotic Stereo-Vision System to be Augmented

11. Performance evaluation of the approach Evaluation Superimposition of the virtual object on to the stereo-video and augmented tele-manipulation Augmentation of Stereo-vision Interfacing to the telerobotic stereo-vision system Interface Moving capability of the virtual objects Animation Displaying the virtual objects Virtual object rendering Drawing the virtual objects Virtual object drawing Deriving the robot model equations Mathematical models Design Methodology

12. Mathematical Model of the Robot Manipulator • PUMA 560 Slave Arm: • 6 DOF • 6 links are interconnected serially to each other except first and last. • All the joints are rotational. A rotational joint

13. Each link of the arm is attached with a frame of reference Ri (xi,yi,zi) to specify its position and orientation. Link Li+1 rotates w.r.t Li when frame Ri+1 rotates w.r.t either axes of xi, yi, or zi Link vector OiOi+1,i can be expressed as Where, Mathematical Model of the Robot Manipulator • Now position and orientation of Oi+1 w.r.t Oi-1 can be expressed as

14. Where, Mathematical Model of the Robot Arm • Position and orientation of the end effector is expressed relative to the frame of the base. • So the geometric model of robot arm can be expressed as: This model will give us a skeleton of the graphical arm.

15. Building Body Shapes Around the Skeleton of Graphical Arm • Alternatives: • Taking as is: link 3 and 4 are trapezoidal, others are cylindrical. • Considering all as cylindrical

16. Building Body Shapes Around the Skeleton of Graphical Arm • Cylinder is drawn with finite element representation. • Increasing number of segments will improve quality of view but increase computational complexity. • Alternatives of primitive for drawing a cylinder: • Triangles: • Triangle stripe, Triangle fan, Triangle list • Lines • Line stripe, Line list

17. Data Structure Design • Types of data and nature of manipulation: • A link is represented by its start and end point co-ordinates, radius, axis of rotation, joint angle, orientation matrix etc. • Each link is also associated with the vertices of its body shape. • Links are serially connected and position and orientation of a link depends on position and orientation of its previous link. • There is frequent movement of links in teleoperation and each movement involves matrix calculation for new position and orientation of the links to be changed. • Objectives: • To reduce processing time by reducing computation. • To provide flexibility and generality

18. Data Structure Design • Our strategy: • Each link is defined as an object geometric data of each link is now apart from the other link,. • Configuration data (number of links, number of segments in cylinder etc) kept in separate data file. • World and view transformation matrices are combined to reduce matrix multiplication. • Vertex data of body shape of each link is stored in separate vertex buffer.

19. Displaying the Graphical Arm • Given the robot structure in its model space • Rendering on the display screen: • Scene layout setup: • Transformation matirces, Camera position, target position, viewing angle, lighting. • Viewing Models • Solid Model or Wire-frame Model • Psychological studies and one performance studies in telerobotic system of shows no significant diff. • Wire-frame models have some adv.-reduce occlusion, lesser rendering time. • Rendering: • Scanline rendering, Z-buffering algorithm for HSR.

20. Sample Display of Graphical Arm Wire-frame view Solid view

21. Movement of Graphical Arm in Joint Space

22. Movement of Graphical Arm in Cartesian Space Start Receive the incremental position (X) and orientation (M) matrices Calculate new position, Xnew(t) and orientation, Mnew(t) matrices based on current frame of reference Calculate corresponding angular values Using Inverse Geometric Transformation (new=G-1(Xnew, Mnew) Angle within bound? B

23. Movement of Graphical Arm in Cartesian Space…

24. Acquisition of Real Video Image • The MDTF Approach is used for stereovision. • It is multi-threaded, distributed and provide better performance in video transfer. • Real video image of the slave robot at the server side is captured simultaneously by two video cameras. • Then a reliable client-server connection is established and upon a request from the client a stereo frame comprising of two pictures is sent over LAN through window sockets. • On the client side after detecting and making connection with the server pictures are received and displayed on the screen and on the HMD when connected. • A double buffer, concurrent transfer approach is used to maximize overlapped transfer activities between cameras, processor and the network.

25. 3D Visualization Tools and Techniques • Technique • Alternatives: Sync-Doubling, Page-flipping • Chosen: Page-flipping • Provides higher resolution and avoid ”flashing” problem of 3D imaging. • Display device • Alternatives: HMD, monitor, eye-shuttering glass • Chosen: HMD, monitor • No flickering, easy to work

26. 3D Visualization Example A snap shot of a stereo-image.

27. Camera Calibration • Heikkila’s calibration method which provides very accurate results. It shows accuracy up to 1/50 of the pixel size. • Pinhole camera model with perspective projection and least-square error optimization. • The intrinsic camera parameters: • Focal length, fc • Principal point, cc • Skew co-efficient, alpha_c • Lens distortion co-efficient, kc

28. Camera Calibration… The aspect ratio= fc(2) / fc(1) The field of view angle can be calculated from,

29. Camera Calibration… • Extrinsic Parameters If XX=[X,Y,Z] and XXc are co-ordinate of P in grid and camera ref, then Where, translation vector Tc_1 is the co-ordinate vector of O in camera ref frame and the rotation matrix Rc_1 is the surface normal vector of the grid plane in the camera ref frame.

30. 3D Visualization Algorithm • Step 1: Acquire video image and copy this memory stream to a surface( say FrontSurf). • Step 2: Copy the FrontSurf surface to another temporary surface (say, AugSurf). • Step 3: Draw graphical objects/change in graphical objects on AugSurf. • Step 4: Copy AugSurf to the surface that will be used to display (say, backSurf). • Step 5: Display backSurf with left camera image to left view port/monitor. • Step 6: Display backSurf with right image to right view port/monitor. • Step 7: Observe 3D with HMD.

31. Graphical Tele-manipulation • How Graphic Manipulation Corresponds to the Real • If the base link of the real robot remains fixed relative to the video cameras, the base link of the graphical arm will also remain fixed relative to the graphical cameras. • The end-effector of the graphical arm can be manipulated in the graphical coordinate space, relative to objects in the task space (keeping base link in same location of real robot base).

32. GUI Design • Client Side Input/Output Specification • Connecting to the server PC, master arm and HMD • Receiving and displaying the stereo video. • Taking user's input for movement of the real and graphical robot for simulation • Means of User Interaction: • Master arm • Joystick • Keypad • Mouse

33. Implementation

34. Tools Used • Client-server communication platform • MS .NET with .NET Remoting • Programming Language • Visual C#.NET • Graphics Tool • Microsoft DirectX • 3D Graphics API • Alternatives: • Windows GDI • Java3D by Sun MicroSystems • Open GL (Open Graphics Language) by SGI Silicon Graphics • Direct3D

35. Motivation for Using Direct3D • It increases performance by using hardware acceleration. • It allows applications to run full-screen instead of embedded in a window. As we are using HMD this feature is very helpful in our case to have the 3D effect. • Direct3D perfectly match with Microsoft's various Windows operating systems and with Microsoft .NET framework which are used in our telerobotic client-server framework

36. Direct3D Architecture

37. Graphics Implementation • Virtual Object Modeling Module • Defines the structure of the virtual objects (robot, cubes etc.) • Handles functions like movement of virtual objects. • Display Module: • Synchronization of real and virtual data • Projection on video surface • Augmentation of real video • Page Flipping for HMD stereo visualization

38. Interfacing to Telerobotic Stereovision System • Client Modules:

39. GUI Implementation • Main Form

40. GUI Implementation • Stereo Form

41. Performance Evaluation

42. Evaluation of Graphical System: • Refresh Rate: [Average is taken over 1000 samples, running on Pentium-4, 2GHz machine with 1GB RAM]

43. Time Required for Rendering Graphical Arm [Average is taken over 1000 samples, running on Pentium-4, 2GHz machine with 1GB RAM]

44. Accuracy • Re-projection error with the calibration method: [Pixel Error (0.11689,0.11500)] Pixel Axis in the y direction Pixel Axis in the x direction

45. Comparison to Other Apporaches • Iqbal, A. [1] augmented with only a small red ball at the position of gripper in comparison to our whole graphical arm. • Iqbal, A. [1] used Faugeras [4] calibration with Kuno[5]’s affine frame of reference which led him to noticeable mismatch with real error in the matching shown in his figure. Whereas our computer vision-based calibration reduces error upto 1/50 of pixel size. • J. Vallino reports in his PhD thesis refresh rate of 10fps to be required for AR, when we gets above 11-17fps after overlaying graphical arm with live stereo video. • Graphics rendering of our system is faster than Iqbal, A. [1],[2][3] for our use of Direct3D. • Our system is comparatively cheaper due to the use of commodity hardware (PC) and software.

46. Conclusion

47. Summary of the Work and Contributions • Using hardware accelerated graphics rendition that provides us with excellent refresh rate of the output screen. • Improvement in accuracy of execution by using better calibration method and graphic aids (accuracy up to 1/50 of pixel size). • User-friendly graphical user interface for simple manipulation in the telerobotic AR system.

48. Summary of the Work and Contributions… • Flexible and generalized data structure suitable for telerobotic visualization. • Identifying design strategy for intelligent switching in VR and AR mode for ensuring QoS. • Use of cheap and commercially available hardware and software • Can be used as a cheap and flexible visual tool for showing robot manipulation in the classrooms.

49. Future Research Directions • Providing an intelligent system to switch between VR and AR modes of operation based on network delays to ensure QoS. • Using multi-processor system for processing video and graphics data more efficiently. • Using commercial software available to extract exact 3D model of the workspace objects which will facilitate more accurate task manipulation.

50. References • Iqbal, A. Multistream realtime control of a distributed telerobotic system. M.Sc. Thesis, King Fahd University of Petroeum and Minerals,June 2003. • A Rastogi, P. Milgram, and D. Drascic. Telerobotic control with stereoscopic augmented reality. SPIE, Vol.2653: Stereoscopic Displays and Virtual Reality Systems III:135{146, Feb. 1996. • R. Marin, P.J. Sanz, and J.S. Sanchez. A very high level interface to teleoperate a robot via web including augmented reality. Proc. IEEEInternational Conference on Robotics and Automation, 2002 ICRA '02, Vol.3:2725 - 2730, May 2002. • O.D. Faugeras and G. Toscani. The calibration problem for stereo. Proceedings of Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, Vol. 5, No. 3, June, 15-20 1986.