Transparency Improvement of Haptic-based Networked Systems

1. Transparency Improvement of Haptic-based Networked Systems Seokhee Lee

2. Contents Introduction Transparency Analysis for Haptic-based Networked Systems Delay Compensation Scheme for Haptic-based NVEs Transmission and Error Control Scheme for Haptic-based NVEs Haptic Synchronization Scheme for Force-reflecting Teleoperation Conclusions and Future Work

3. Introduction Overview of HNS Motivation and Contributions

4. Overview of HNS • HNS (haptic-based networked system) • System which provides haptic feeling for a user who interacts with remote environments • Haptic-based NVE (networked virtual environment) + force-reflecting teleoperation • Local haptic system + remote haptic systems

5. Requirements of HNS Focused requirement • Main goal ofHNS • Executing a task in a remote environments (real or virtual) • With stability and transparency • Stability • The primary requisite for safe system • Instability • Uncontrollable oscillations and chaotic behavior • Factors causing instabilities • Quantization error, time delay, packet losses, asynchronous switching between continuous- and discrete-time subsystems …… • Transparency • Transparency ≈ haptic realism • Mathematically more difficult to analyze since the ultimate goal is to make the user experience a “good feeling” • Optimal transparency • The user cannot distinguish between direct and tele-interaction with a remote environment.

6. Network Problems Focused network problems • Instability • The delayed and lost haptic data destabilize HNS. • The instability may cause serious damages to the user. • Transparency deterioration • The human haptic sense is more sensitive than the visual and auditory senses. • Because of the sensitivity, the transparency is deteriorated severely with network variations of delay and loss. • High transmission rate • A high update rate (approximately 1 kHz) of haptic rendering leads to high transmission rate of 1000 packets/s. • The available network bandwidth for multiple haptic data may not be sufficient over current Internet.

7. Existing Haptic Data Networking Schemes Focused networking schemes • Network layer solutions • Improve the performance of the network itself for HNS. • QoS provision for haptic data • Class-based weight fair queue (Marshall08), • DiffServ & IntServ (Hirche05) • QoS routing for haptic data • QoS-guaranteed overlay routing (Cen05) • High speed network for haptic data • Optical networks (LaMarche07) • Application and transport layer solutions • Improve the system performance on the assumption that the current network performance is not sufficient. • Delay/jitter compensation • Transmission control • Error control

8. Motivation • Problems of the existing networking schemes (application and transport layer) • There was little consideration to enhance the human haptic perception. • Network adaptation schemes for conventional multimedia were applied to the haptic interaction without careful consideration of the difference between the haptic event and the other data. • Remaining challenge • Minimizing the transparency (haptic realism) degradation • By determining the optimum adaptation parameters for transparent haptic interactions • Transmission rate, error control level, buffering time, spring coefficient for delay compensation,……

9. Contributions • Distinguishing point of the proposed networking schemes • Consideration of human perception characteristics (transparency) for the optimization • Transparency analysis • Deterioration of haptic interaction quality caused by network delay is quantified as distortions of mass, damping, and spring coefficients of a virtual object. • In order to formulate the importance of a haptic event with respect to a packet loss, loss effect of each haptic event is quantified for haptic-based NVEs with a prediction algorithm. • For force-reflecting teleoperations stabilized by a control algorithm, the force feedback distortions caused by network delay and packet loss are quantified when robot keeps in contact with a wall.

10. Contributions • Networking schemes based on transparency analysis • Delay compensation scheme • Allowable delay prediction • Predicts the maximum allowable delay based on the quantified values of mass and damping distortions. • Spring coefficient modification • Modifies spring coefficient according to delay based on the quantified values of spring coefficients’ distortions. • Verification • Experimental results • The proposed scheme improves the haptic interaction quality more efficiently compared with existing delay compensation schemes while avoiding unnecessary trial-and-errors over network delay. • Representative paper • Elsevier Computer Communications, 2009

11. Contributions • Transmission and error control scheme • Haptic event prioritization • Based on the quantified loss effect, all haptic events are classified into several groups with different network QoS requirements. • Priority-based haptic event filtering • Based on the prioritization, select more appropriate data to be transmitted for realistic haptic interaction over bandwidth-limited lossy network. • Verification • Simulation and experiment results • The proposed scheme provides lower packet rate than the existing schemes for a transparent haptic interaction over a bandwidth-limited network. • The proposed scheme guarantees less processing delay and better haptic interaction quality compared with previous error control schemes over a lossy network. • Representative paper • Springer Multimedia Systems, 2009 • Patent • Haptic event transport method for haptic-based collaborative virtual environment and system therefor

12. Contributions • Haptic synchronization (buffering) scheme • Allowable delay and loss time prediction • Predicts maximum allowable delay and loss time based on quantified force feedback distortions and predefined transparency requirements. • Network-adaptive buffering time control • Improves transparency by controlling the playout time of the transmitted haptic event with the transparency-related parameters from transparency analysis. • Verification • Remote calligraphy system • The proposed scheme guarantees less force feedback error compared with the existing haptic synchronization schemes over time-varying network delay. • Representative paper • ACM NetGames, 2009 • Patent application • The method for synchronizing haptic data and the haptic system

13. Transparency Analysis for Haptic-based Networked System Transparency Deterioration Caused by Network Variations Transparency Analysis for Haptic-based NVEs Transparency Analysis for EBA-based Teleoperations

14. Transparency Deterioration Caused by Network Variations • Transparency deterioration caused by delay • If there is network delay while a user manipulates a virtual object, the virtual object does not move immediately. • During that time period, penetration depth increases. • Eventually, the force feedback in spring-damper model also increases.

15. Transparency Deterioration Caused by Network Variations • Transparency deterioration caused by packet loss • Haptic-based NVE without a data prediction scheme • The force feedback also increases in the same manner of the delay case. • Sudden movement • Haptic-based NVE with a data prediction scheme • If the haptic event can be predicted based on past patterns of events, the quality degradation will be small. • Otherwise, the quality degradation is more severe. With data prediction scheme <Original movement of a virtual object> <Unintended movement caused by loss>

16. Transparency Deterioration Caused by Network Variations • Transparency deterioration caused by delay jitter • More severe deterioration of haptic interaction quality • Out-of-order arrivals as well as delayed data transmission and empty sampling instances (Lee06)

17. Transparency Analysis for HNS • Quantification of transparency deterioration according to network delay and packet loss • In order to determine the optimum adaptation parameters of haptic data networking schemes • Focused HNS • Haptic-based NVEs • With CS (client-server) architectures • Force-reflecting teleoperation • With EBA (energy bounding algorithm)

18. Transparency Analysis for Haptic-based NVEs • Position-position interaction • HIP (haptic interaction pointer) position (client -> server) • Virtual object position (server -> client) • Consistency server • Updates movements of all virtual objects. • Distributed force calculation • Each client calculates force feedback by using spring-damper model.

19. Transparency Analysis for Haptic-based NVEs • Transparency • Equality between the human (Zh) and the environment (Ze) impedances • Simplification • The number of clients is not related to transparency. • A virtual object is represented by a mass and damping. • Impedance

20. Transparency Analysis for Haptic-based NVEs • Magnitude responses of impedances with respect to network delay • Confirm that if delay increases, force feedback also increases • It is called transparency (force feedback) distortion • Mass and damping distortion (mh, bh) • Spring coefficient distortion (khm, khb) kh=0.5,ke=0.5,m=0.25, b=0.0025 R=RTT (round trip time)

21. Transparency Analysis for Haptic-based NVEs • Mass and damping distortion • If delay increases, a user perceives a virtual object as having larger mass and damping coefficients than the actual values. • Quantification of mass (mh) and damping (bh) distortion • Low frequency approximation • Human operator usually generates low frequency input (Hirche05) • [10-6~102] • Approximated human impedance • Transparency condition • Distorted mass • Distorted damping

22. Transparency Analysis for Haptic-based NVEs • Distortion of spring coefficient • If delay exists, user is provided with unrealistically large force feedback that seems to be obtained with the larger spring coefficient than the actual value. • Quantification of spring coefficient distortion • Mass-based distortion (khm) • Damping-based distortion (khb) • If virtual object dynamics are affected by mass more than by damping, spring distortion follows mass-based distortion; otherwise it follows damping-based distortion.

23. Transparency Analysis for Haptic-based NVEs • Force feedback distortion according to packet losses • Without a prediction scheme • Quantified in the same manner as the force distortions according to network delay • By using loss time (Tloss) • With a prediction scheme (fdis(n)) • Quantified by using the difference between the actual and the predicted (xpre(n)) positions • Loss effect (LE(n)) of the nth haptic event • In order to formulate the importance of a haptic event with respect to a packet loss

24. Transparency Analysis for EBA-based Teleoperation • EBA (energy bounding algorithm) • Stability algorithm of a haptic simulation system (Kim04) • EBA restricts the energy generated in the ZOH (zero order hold) within a consumable energy limit in the haptic device. • Can be applied to teleoperation to ensure robust stability regardless of the amount of time delays and packet losses (Seo08).

25. Transparency Analysis for EBA-based Teleoperation • SlaveEBA Control law Control law Bounding law Bounding law MasterEBA

26. Transparency Analysis for EBA-based Teleoperation • Definition of Transparency • Similarity between the force feedback for a slave robot (FsEBA) and that for a user in master side (FmEBA) • Assumptions • Robot keeps in contact with a wall and a user feels the force feedback by moving haptic device facing the wall with constant velocity (vm). • Force feedback increase according to the user's input • From the control law in master EBA

27. Transparency Analysis for EBA-based Teleoperation • Approximation of force feedback increase • Assumption: c1m≥2c2m • γm,max(n) in bounding laws converges into c2mwhen Fm(n-1) increases monotonically. • Force feedback decrease caused by delay • Network delay reduces Tinteras much as the delayed time. • If the network delay increases, the force feedback decreases in proportional to c2m∙vm.

28. Transparency Analysis for EBA-based Teleoperation • Force feedback decrease (Fde,loss) caused by loss • Packet losses reduces Tinteras much as the loss period. • Transparent loss time (Ttr,loss): time period when the force feedback continuously increases even though there exist the packet losses

29. Delay Compensation Schemefor Haptic-based NVEs Related Work Delay Compensation Scheme Experimental Results

30. Related Work • Spring coefficient modification scheme (Fujimoto04) • Dynamically changes the reaction force applied to a user by adjusting a spring coefficient of a spring-damper model. • The spring coefficient (Kh) is modified according to the current end-to-end delay (ΔT). • Problems • An accurate spring coefficient for the realistic haptic interaction can only be found by a process of trial and error. • A significant challenge remains in developing an optimum method of determining the spring coefficient.

31. Transparency Analysis-based Approach • Transparency Analysis • Transparency deterioration caused by delay is quantified as distortions of mass, damping, and spring coefficients of virtual objects. • Delay compensation scheme based on transparency analysis • Predicts the maximum allowable delay from the quantified values of mass and damping distortions. • Modifies the spring coefficient according to network delay based on the quantified value of spring coefficient distortion.

32. Delay Compensation Scheme Delay compensation scheme

33. Delay Compensation Scheme • Allowable delay prediction • Quantified mass and damping distortions are proportional to network delay. • Predefined transparency requirements • Maximum allowable mass (mallowable) and damping (ballowable) • Allowable mass and damping gradients (cm, cb) • Mass-based: • Damping-based:

34. Delay Compensation Scheme • Spring coefficient modification • Estimation of the spring coefficient increment perceived by user • By using the spring coefficient distortion in transparency analysis • Modified new spring coefficient • By subtracting the spring coefficient increment from original spring coefficient

35. Experimental Results • Verification of the transparency analysis and delay compensation scheme • Pushing motion of a virtual object • With constant velocity • Haptic devices • PHANToM Omni • Network emulation • NIST Net

36. Experimental Results • Verification of transparency analysis • Force feedback comparison • Force feedback 1 • Original coefficients (m, b, kh) • Delay 300 ms • Force feedback 2 • Original spring coefficient • Distorted mass and damping coefficients when delay=300 ms (m=mh, b=bh) • No delay • Force feedback 3 • Original mass and damping coefficients • Distorted spring coefficient when delay=300 ms (kh=khm) • No delay • All force feedbacks are similar -> quantifications of force feedback distortions are acceptable.

37. Experimental Results • Verification of the proposed scheme • Force feedback comparison • Force feedback 1 • No delay (transparent force feedback) • Force feedback 2 • No delay compensation scheme • Delay of 300 ms • Force feedback 3 and 4 • Existing schemes with different initial settings (Fujimoto04) • Delay of 300 ms • Force feedback 5 • Proposed scheme • Delay 300 ms Transparent force feedback • Force feedback with proposed scheme is most similar to original force feedback -> proposed delay compensation scheme is useful for transparency improvement.

38. Transmission and Error Control Scheme for Haptic-based NVEs Related Work Haptic Prioritization Priority-based Filtering Simulation and Experiment Results

39. Haptic-based NVE over Bandwidth-limited Lossy Network Focused transmission control • Transmission control • Transmission rate adaptation • Transmission rate reduction (traffic reduction)

40. Related Work • Transmission rate reduction • Statistical approach • Focuses on the statistical properties of haptic signal • Applies conventional data compression schemes to haptic data • Haptic compression scheme based on Huffman coding (Hikichi01) • Haptic compression scheme based on DPCM (differential pulse code modulation) (McLaughlin02) • Problem • Even very good compression on the payload itself is useless if a big share of the necessary network bitrate is wasted by packet headers. • Perception-based approach • Packet rate reduction rather than payload compression by using the limitations of human haptic perception

41. Related Work • Perception-based approach • Deadband-based filtering (Hirche05; Zadeh08) • Events are only sent if the change exceeds a given threshold value. • Prediction-based filtering (Kanbara04; Clarke08) • Sender transmits the event only when the difference between predicted and actual events is larger than a threshold value. • Problem • They make the haptic applications very sensitive to packet losses. • Should be used together with a error control scheme over lossy network.

42. Related Work Teleoperation layer • Error control scheme • ARQ (automatic repeat request) • Teleoperation layer (LaMarche07) • TCP for critical haptic data • Smoothed SCTP (Dodeller2004) • Selective ARQ for last update message • FEC (forward error correction) • STRON (Cen05) • To compensate undesirable jitter caused by ARQ • Error-correcting code such as Reed-Solomon code • Problem • The existing researches use conventional error control schemes for video, audio, and events • Large processing delay for haptic interactions • Transparency deterioration e.g. robot arm control control: robot control, feedback1: force, feedback2: video, feedback3: orientation STRON (supermedia transport for teleoperations over overlay networks)

43. Transparency Analysis-based Approach • Transparency analysis • The packet loss effect of each haptic event is formulated based on the difference between the actual and predicted positions. • Transmission and error control scheme based on transparency analysis • Haptic event prioritization • All haptic events are classified into several groups with different network QoS requirements based on the formulated values. • Priority-based haptic event filtering • Based on the prioritization, selects more appropriate data to be transmitted over bandwidth-limited lossy network. • Guarantees less processing delay than existing error control schemes. • By reconstructing lost high-priority event with received low-priority events without any error-correcting code and retransmission

44. Haptic Prioritization • THHLI • Derived empirically • Set to 0.4 mm • A user cannot perceive the deterioration of interaction quality within the range of 0.01 to 0.4 mm (Hikichi01) • Comparison between loss effect (LE) and a threshold value (THHLI) • HLI (haptic event loss index) • HLI=0 (HLI0) • Predictable haptic event (LE(n) < THHLI) • HLI=2 (HLI2) • Unpredictable haptic event (LE(n)≥ THHLI) • Critical haptic event • HLI=1 (HLI1) • Redundant haptic event • Predictable but necessary events for the lost unpredictable events • To compensate the loss effect of HLI2event

45. Haptic Prioritization Gilbert loss model • α • A sufficiently small value (e.g., 10-4), • Indicating that a packet loss rate of less than α is considered negligible. • Decision of HLI1 event • Because haptic events include the position every millisecond, two sequential events have similar values. • In order to compensate the loss effect of HLI2event, several haptic events following a HLI2event can be HLI1event. • Maximum number (NHLI1) of HLI1events is calculated based on Pi • Pi: the probability that a HLI2 event and i HLI1 events are all lost.

46. Priority-based Filtering • Transmission rate reduction • HLI0 events are always filtered. • Filtering HLI1 events • When current transmission rate (r) > allowable bandwidth (R) • Filtering type • Sequential: filtering events sequentially • Alternate: filtering events alternately • Probability increments of packet losses when j HLI1 events are filtered (Pi,jsequential and Pi,jalternate) • Filtering number (j) • Probability (Pi,j) that a HLI2 event and i HLI1 events are lost when j HLI1 events are filtered • Predefined allowable packet loss probability (Pallowable)

47. Priority-based Filtering • Haptic event reconstruction (xrec(n)) • To reduce loss of fidelity caused by lost HLI2event • Reconstruct lost nth HLI2event with HLI1events • Required buffering time (processing time) • NHLI1 (ms) • All haptic events are generated every time when haptic rendering module is updated (i.e., every millisecond). • Worst-case scenario for reconstruction of nth haptic event • Only n+NHLI1th haptic event is received successfully • Tn,n-l : The elapsed time between nth and n-lth events • Assumptions • n-lth event was received successfully • nthHLI2 event is lost but n+mth(1<m<k) HLI1 event is transmitted successfully

48. Simulation Results • Matlab/Simulink Simulation • Verification of priority-based filtering scheme • Virtual object contact motion • A haptic device is fixed at a zero point. • A virtual object moves back and forth from -0.1 to 0. • Penetration depth 0~10 cm • Transparent force feedback • Force feedback when packet loss rate = 0 %

49. Simulation Results • Verification of transparency with low transmission rate • Comparison between priority-based and deadband-based filterings • Loss rate 20% and allowable bandwidth 70 Kbps • Priority-based filtering • Satisfies the transmission rate and transparency requirements (error < 0.1N). • Deadband-based filtering • Threshold = 2.5 mm • Large force feedback error • Threshold = 0.7 mm • Large transmission rate

50. Experimental Results Transparent force feedback • Verification of transparency over lossy network • Transparency comparison (peak force feedback error) • Prediction-based with no error control: 0.11N • Prediction-based with FEC: 0.3N • Prediction-based with ARQ (μ=25ms, ν=4): 0.1N • Prediction-based with ARQ (μ =4ms, ν=1): 0.104N • Priority-based: 0.014N (μ =retransmission timeout, ν=maximum retransmission)