Transparency Analysis and Haptic Synchronization Scheme for Force-reflecting Teleoperation

1. Lab. Seminar 2009.08.15 Transparency Analysis and Haptic Synchronization Scheme for Force-reflecting Teleoperation Seokhee Lee

2. Contents • Introduction • Requirements of Force-reflecting teleoperation • EBA-based Teleoperation • Delay Jitter Problem and Related Work • Transparency Analysis-based Approach • Transparency Analysis • Haptic Synchronization • Simulation and Experiment Results • Conclusions and Future Work

3. Requirements of Force-reflecting Teleoperation • Stability • The primary requisite for safe system • If the output response are bounded for all bounded inputs, the system is said to be stable. • Instability • Uncontrollable oscillations and chaotic behavior • Sometimes, serious damages to the user • Transparency • Transparency ≈ haptic realism • Mathematically more difficult to analyze since the ultimate goal is to make the user experience a “good feeling” • Optimal transparency • The user cannot distinguish between direct and tele-interaction with a remote environment.

4. EBA-based Teleoperation • EBA (energy bounding algorithm) • Stability algorithm of a haptic simulation system (Kim & Ryu, 2004) • EBA passifies virtual environment and restricts the energy generated in the ZOH (zero order hold) within a consumable energy limit in the haptic device. • Can be applied to teleoperation to ensure robust stability regardless of the amount of time delays and packet losses (Seo et al. 2008).

5. EBA-based Teleoperation • SlaveEBA Control law Control law Bounding law Bounding law MasterEBA

6. Delay Jitter Problem • Delay jitter effect of haptic event • Delayed data transmission, out-of-order arrivals, and empty sampling instances • Teleoperation with delay jitter • Instability • Transparency deterioration • EBA with delay jitter • EBA guarantees stable teleoperation over network delay and packet losses • Limitation • It cannot overcome transparency deterioration according to time-varying network situation.

7. Related Work • Adaptive buffering control using moving-average smoothing technique (Wongwirat & Ohara, 2006) • Buffering time is adjusted to twice the moving average delay. • Problem: the authors mentioned the importance of an optimum buffer size for the transparency but it remained further study. • Adaptive buffering control with interpolation scheme (Berestesky et al. 2004) • Stability is guaranteed by compressing and expanding buffered data. • Problem: although this scheme improved performance of position tracking and stability, it did not focus on transparency of force. • VTR (virtual time rendering) (Ishibashi et al. 2004) • Dynamically adapting the buffering time to improve the interactivity of haptic events in haptic-based NVEs • Problem: little attention has been given to the transparency in the force-reflecting teleoperation.

8. Transparency Analysis-based Approach • Transparency analysis • Quantifies the force feedback distortions caused by network delay and packet loss. • Predicts the maximum allowable delay and loss for the predefined transparency requirements. • Haptic synchronization based transparency analysis • Improves transparency over time-varying delay. • By controlling the playout time of the transmitted haptic event with the transparency-related parameters • By synchronizing the local haptic event with the transmitted event according to the transparency analysis

9. Transparency Analysis • Definition of Transparency • Similarity between the force feedback for a slave robot (FsEBA) and that for a user in master side (FmEBA) • Force feedback increase according to the user's input • Robot keeps in contact with a wall and a user feels the force feedback by moving haptic device facing the wall with constant velocity (vm). • From the control law in master EBA

10. Transparency Analysis • Approximation of force feedback increase • Assumption: c1m≥2c2m • γm,max(n) in bounding laws converges into c2mwhen Fm(n-1) increases monotonically. • Force feedback decrease caused by delay • Network delay reduces Tinteras much as the delayed time • If the network delay increases, the force feedback decreases in proportional to c2m∙vm.

11. Transparency Analysis • Force feedback decrease (Fde,loss) caused by loss • Transparent loss time (Ttr,loss): time period when the force feedback continuously increases even though there exist the packet losses • Maximum allowable delay (Tal,delay) and loss (Tal,loss) • Predefined transparency requirements • Maximum allowable force feedback (Fal) • Maximum allowable force feedback decrease by packet loss (Fal,loss)

12. Haptic Synchronization • Transparency improvement • Output time control of delayed force • Controls playout time of transmitted event. • With transparency-related parameters • Output time control of local position • Synchronizes local event with transmitted event. • To minimize the decrease of interaction time caused by delay

13. Haptic Synchronization • Output time control of delayed force • Ideal target output time xnf • The time at which the event should be output in the case where network delay jitter is always smaller than an estimated maximum network delay jitter Jmax • Target output time tnf • The time when the haptic event should be output in the case where network delay jitters exists

14. Haptic Synchronization Virtual time expansion Virtual time contraction • Output time Dnf • By comparing the arrival time An and the target output time tnf • Virtual time expansion • Delays the target output time • Total delay increase, loss rate decrease • To minimize the transparency degradation caused by delay • Only when packet loss time is larger than allowable packet loss time • Virtual time contraction • Advances the target output time • Total delay decrease, loss rate increase • To minimize the transparency degradation caused by loss • Only when the haptic interactions do not happen

15. Haptic Synchronization • Output time control of local position • In order to minimize the decrease of the interaction time Tinter caused by delay, the playout time of the local position Xmis synchronized with the transmitted haptic event Fds. • Ideal target output time xnP • Output time DnP • If the virtual-time expansion or contraction is executed for transmitted event, the target output time and output time of Xm are also changed.

16. Simulation Results • Matlab/Simulink simulation • Verification of the the transparency analysis • Wall contact motion • Slave robot keeps in contact with the wall. • Haptic device movement • vm=0.05 m/s

17. Simulation Results • Simulated results closely follow predicted values -> the transparency analysis is valid • Delay effect • Transparency analysis • Fde,delay=10∙Tdelay • Delay increase of 100ms -> force decrease of 1N • Simulation results • With delay 0~600ms • Loss effect • Transparency analysis • Ttr,loss=0.002 s • Fde,loss=10 ∙(Tloss-0.002) • Packet losses for 50 ms -> force decrease of 0.48N • Simulation results • With loss time 2~300ms

18. Experimental Results • Verification of the proposed scheme • Transparency improvement over network delay jitter • Assumptions • Maximum allowable force feedback decrease by packet loss (Fal,loss)=1N • Time-varying network delay

19. Experimental Results Transparent force feedback • Wall contact motion • vm=0.2 m/s • Transparency comparison • Each force feedback with different schemes is compared with the transparent force feedback (FsEBA). • RMS force feedback errors • Moving-average adaptive buffering (MAB) = 3.7 N • VTR = 3 N • Skipping = 2.6 N • Proposed scheme = 1 N

20. Remote Calligraphy System • No network delay • Standard shape and force • Transparent force feedback

21. Remote Calligraphy System • The user writes the character as feeling the similar force feedback to the transparent force feedback • Force feedback error is less than 0.2 N • The force errors between the reaction forces and the standard force • 0.6 N (MAB), 0.4 N (VTR), 0.3 N (skipping), and 0.1 N (proposed scheme)

22. Remote Calligraphy System User can write the character most similarly to the standard shape by using the proposed scheme. With the other schemes, although the user thinks that he or she is writing the character well, unintended tick lines are drawn.

23. Conclusions • Transparency analysis and haptic synchronization scheme for EBA-based force-reflecting teleoperation • Transparency analysis • The force feedback distortions according to network variations are quantified. • Haptic synchronization scheme based on transparency analysis • The optimization of the adaptation parameter (buffering time) of the scheme for realistic haptic interactions (transparency) • Simulation and experimental results • Transparency analysis provides an acceptable quantification method • The scheme guarantees more transparent haptic interactions over time-varying network delays

24. Future Work • Accuracy and generality improvement • More simulations and experiments with various haptic interaction scenarios • Paper work • Taylor & Francis, Cybernetics and Systems (regular issue) • Taylor & Francis, International Journal of Human-Computer Interaction (regular issue) • Springer’s Multimedia Tools and Applications (special issue: Multimodal Interaction and Multimodal Content Management) • Manuscript submission deadline: 1 October 2009 • Notification of acceptance: 1 December 2009 • IEEE Transaction on Systems, Man, and Cybernetics (regular issue)

25. END Questions & Comments

26. References J.-P. Kim and J. Ryu, “Stable haptic interaction control using energy bounding algorithm,” in Proc. IEEE/RSJ IROS, 2004. C. Seo, J. Kim, J. Kim, J. Yoon, and J. Ryu, “Stable bilateral teleoperation using the energy-bounding algorithm: Basic idea and feasibility tests,” in Proc. IEEE/ASME AIM, 2008. O. Wongwirat and S. Ohara, “Haptic media synchronization for remote surgery through simulation,” IEEE Multimedia, 2006. P. Berestesky, N. Chopra, and M. W. Spong, “Discrete time passivity in bilateral teleoperation over the internet,” in Proc. IEEE ICRA, 2004. Y. Ishibashi, H. Kasugai, and M. Fujimoto, “An intra-stream synchronization algorithm for haptic media in networked virtual environments,” in Proc. ACM SIGCHI ACE, 2004.