Thesis Defense Olufunke Olaleye

1. Thesis Defense Olufunke Olaleye Symbiotic Audio Communication on Interactive Transport

2. Technology Overview • Today's Internet traffic contains both audio and data packets. • Sensitive traffic (audio) shares bandwidth with other non-sensitive traffic as shown fig 1.1 • VoIP • Audio Chat • Online Radio • Real time internet lecture • Real time Internet conference • Online Music service • Internet News Figure 1.1 Bandwidth Limited Network Symbiotic Audio Communication on Interactive Transport

3. Growth of Audio While, the technology has moved many businesses online, the use of audio traffic over the Internet has growth exponentially. • However, audio perception is highly susceptible to disturbance in temporal quality. • Packet loss, delay and jitter affects quality of service during congestion. • To maximize the audio/voice quality, some algorithms proposed to adapt: • Size of the playout buffer • Coding rate • Packet path diversity [25] • Challenge Receiving feedback from the network about congestion. Source: Infonetics Research, February 2006 Symbiotic Audio Communication on Interactive Transport

4. Research Goal of Thesis • A well-engineered, end-to-end network is necessary to transmit audio over the Internet. • Goal of this thesis: • Reduce the delay and jitter faced by audio traffic in the network during congestion. • An efficient solution: Sender’s end (encoder) ability to sense the state of the network and react accordingly. • R &D: • Developed symbiotic perceptual audio streaming mechanism that is capable of detecting the underlying bandwidth of the network and modify the target bit rate to suit the network condition. • Combines the quantization technique to accurately represent the audio signals without distortion. • Proposed TCP Interactive (iTCP) • --- Operationally state equivalent to the conventional TCP except applications. • --- Optionally subscribe and receive selected local end-point protocol events in real-time. Symbiotic Audio Communication on Interactive Transport

5. The Human Auditory Perception • Human ear perceives signal within the range 20 and 20000 Hz. • High sensitivity is between 2.5 and 5 kHz and decreases beyond and below this frequency band. • Two Principle Perception is based on • Threshold in quiet --- Sensitivity. • Masking threshold --- • Temporal • Simultaneous masking • Human auditory system perceives signal in a non-equal width sub-band called critical bands, it’s unit is barks Figure 2.2 Threshold in quiet and masking threshold Symbiotic Audio Communication on Interactive Transport

6. Impact of Threshold in Quiet Threshold in Quiet Quantized values without adaptation • Signal strength below the threshold in quiet are inaudible to the human ear. Quantized values with 25% adaptation Symbiotic Audio Communication on Interactive Transport

7. Impact of Masking Threshold Quantizes value with a single masker – 37.5% Quantized values with multiple maskers ---50% Final Output Symbiotic Audio Communication on Interactive Transport

8. Effect of Simultaneous & Temporal Masking • Masking threshold --- • Temporal • Simultaneous masking Symbiotic Audio Communication on Interactive Transport

9. Audio Adaptation • Audio signal simultaneously passes though the hybrid filter bank and psychoacoustic model. • The hybrid filter bank • Divides the input signal into chucks of 576 samples called granule and sub bands of frequency. • Provides a specified mapping in time and frequency. • The psychoacoustic model behaves like the human auditory system. • Computes a just noticeable noise level in each subband. • Determines the block (window) type --- (short/long). • Computes the energy in each partitions band (threshold calculation partition). • Convolves the partitioned energy by applying the spreading function (frequency spread of masking). • Apply pre-echo control using some constants (2 or 16). • Compares the threshold with the last threshold and quiet threshold and takes the maximum. • Converts threshold calculation partition to scalefactor bands and calculates the signal to mask ratio. Figure 2.5 Block Diagram of the encoder Symbiotic Audio Communication on Interactive Transport

10. Tables for Threshold calculation partitions Threshold calculation partitions is computed with following parameters: width, minval, threshold in quiet, norm and bval: (44.1kHz sampling rate---long) (44.1kHz sampling rate---short) Parameters for computing the SNR, for short window, it is read from a table. Norm is normaling constant for each sub band Bval is bark value. For low freq, the strength of the masking is limited by minval bark: a non-linear frequency scale. Symbiotic Audio Communication on Interactive Transport

11. Tables for converting threshold calculation partitions to scalefactor bands There are 21 bands at each sampling frequency for long windows and 12 bands each for short windows. (44.1kHz sampling rate---long) (44.1kHz sampling rate---short) • The number of partitions (cbw) converted to one • scalefactor band (excluding the first and the last partition). • The threshold calculation partitions are converted directly to scalefactor bands. The first partition which is added to the scalefactor band is weighted with w1, the last with w2 • bo is the first index value of cbw • bu is the last index value of cbw Symbiotic Audio Communication on Interactive Transport

12. Audio Adaptation The noise allocation block • Uses the output of the psychoacoustic model, “signal to mask” ratio. • Used the noise level in noise allocation to determine the actual quantizers and quantizer levels. • Determines how to allocate the number of code available for quantization of subband signals. The spectrum (frequencies) are broken into "scalefactor bands". Thes bands are determined by the MPEG ISO spec. In the noise shaping/quantization code, we allocate bits among the partition bands to achieve the best possible quality • It uses two nested iteration loops. • Distortion control loop (outer loop) • Rate control loop – (inner loop) • The inner loop quantizes the input signal and increases the quantizer step size until the output can be coded with the available amount of bit. • After completion of the inner loop, an outer loop checks the distortion of each scalefactor band, if the allowed distortion is exceeded, it amplifies the scalefactor band and calls the inner loop again. quantize in an iterative process Symbiotic Audio Communication on Interactive Transport

13. Audio Adaptation • If the overall bit sum is less than the available bits to encode a frame. • The best quantized values are coded by Huffman coding to further reduce their space requirement. • The bitstream formatting block assembles and formats quantized subband signal using Huffman code and other side information into bitstream. • This is then passed into the network for transmission. Symbiotic Audio Communication on Interactive Transport

14. TCP Congestion Control • TCP provides a connection-oriented, reliable delivery of data streams between two applications or hosts • TCP uses two mechanism to detect network congestion • Retransmission timer time out. • Duplicate ACKs. • Congestion Control Algorithms • Slow-start and congestion- avoidance. • Fast-retransmit and fast-recovery. Figure 3.1 Slow Start/Congestion Avoidance (SSCA) mechanism. Figure 3.2 Fast Retransmit/Fast Recovery (FRFR) mechanism Symbiotic Audio Communication on Interactive Transport

15. TCP Congestion Control Internal Events Subscribable Events } Table 3.1. TCP Congestion Control Internal Events • In this thesis, for simplicity, we use event 1, retransmission timer time out. Symbiotic Audio Communication on Interactive Transport

16. iTCP: interactive TCP • An application that has subscribed to the kernel also binds a T-ware to a selected TCP event through the subscription API as shown by line 1 and 2 of figure 4. • When the TCP state changes as a result of congestion in the network, it also causes an event to occur in the TCP kernel. • The event monitor is aware of the changes that occurred; thus, it responds instantly by sending a signal in (3a) to the signal handler and also stores the event information in (3b). • The signal handler catches the signal from the kernel and requests the event type (4a, 4b) from the kernel via the probing API. • The appropriate T- Transientware Modules (or T-ware) (5) is triggered to serve the particular event. User Space 7 Application T-ware [ 1 ] T-ware [ 2 ] T-ware [ n ] 1 5 6a TCP Connection Signal Handler 4a System Socket API Subscription API Probing API 3a 2 6b TCP Kernel 4b 3b Connection State Event Monitor Event Information Figure 4. The iTCP extension and API. Symbiotic Audio Communication on Interactive Transport

17. Symbiosis Throttling Model • During congestion, detected by the time-out event ( ζ = 1 ), the model reduces the target bit rate to considerable lesser or minimum rate. • bmax, target bitrate at a normal state • bmin, the minimum acceptable rate • Reduction ratio = rate retraction ratio = ρ = b min/ bmax (5.1) • “Running generation threshold” function = link between the underlying TCP and the model. • (5.2) Figure 5.1 Symbiosis throttling Model (Input Rate) Figure 5.2 Symbiosis throttling Model (Output Rate) Symbiotic Audio Communication on Interactive Transport

18. Symbiosis Throttling Model • Running control generation function” b(t) = (5.3) • Rs = reservoir size • Estimated Target Buffer Fullness • (5.4) (5.5) • A = Actual number of bits per granule Figure 5.1 Symbiosis throttling Model (Input Rate) Figure 5.2 Symbiosis throttling Model (Output Rate) Symbiotic Audio Communication on Interactive Transport

19. Symbiosis Throttling Model Input Source audio T-ware Event xr xr b Reservoir b (t), ρ Perceptual Model Rs ratio(sb) T Noise Allocation Estimated Target Buffer Fullness Compute Allowed Distortion T (t), xr 0.9 * T , Rs xmin(sb) 0.9 * T (t), Rs Amplify violated masking threshold xfsf(sb) Compute Distortion in Quantization ix Quantization of actual energy xr  ix (xmin – xfsf) of scalefactor band j < 0 violated ok xr (i) = xr (i) * ifqstep xfsf(sb) Quant Compare Compute Distortion in Quantization ix Quantization of amplified energy xr  ix Code Information Rs • The best quantization with allowed distortion to sharp the noise. Output audio stream ifqstep = sqrt(2) ^ ((1 + scalefac_scale) * ifq(scalefactor band)) Symbiotic Audio Communication on Interactive Transport

20. Symbiosis Mechanism: The T-ware • Signal handler • Catch the signal from the kernel • and invoke the appropriate T­ware. • Encoder subscribe with the • “retransmit timer out" event only. • The mechanism use to reduce delay is called T-ware. • It gives the transport layer the ability to communicate with the application layer (encoder). • The key element is the loss event handler that generates a signal. • The signal information is used to probe iTCP service. • Couple with the retraction ratio ρ , the rate is reduced to achieve the objective of the thesis. • T-ware calculates a frugal • state target bitrate base • on the retraction ratio • Stores the reduced rate • in the “rate.par” file. • Recovery-T-ware • The recovery T-ware kicks in at the • Trecovery time • Returns the encoder bit rate to • normal rate. Figure 6a & 6b Pseudo code of the Signal Handler and the Recovery handler Symbiotic Audio Communication on Interactive Transport

21. Experiment Set Up Figure 7.1 Experiment setup Symbiotic Audio Communication on Interactive Transport

22. Experiment (Test Samples) TEST ( 3 types of audio sound quality) • High quality music (HighQmusic), • Low or Poor quality music (LowQmusic) • Speech mixed with music (SpeechMusic) 3 Running Mode • iEXP • iOFF • Classic Table 7.1 Experiment control flags and running modes Symbiotic Audio Communication on Interactive Transport

23. Experiment and Performance Analysis • The parameters used in the experiments: • (i) Predetermined rate retraction ratio ( ρ = 0.50) • (ii) Bit rate • To compute b(t), the rate retraction ratio is multiplied by the current bit rate. (Bit rate * ρ ) • Frames information are collected for the first 800 up to 1200 audio frames of the encoder and the player. Symbiotic Audio Communication on Interactive Transport

24. Performance Analysis • Performance of iEXP is better than the classic TCP. • The delay buildup in Classic and iOFF are much higher than that experienced by the iEXP (iTCP) as indicated by the step jump. • The step jump of iEXP is much smaller as a result of the rate retraction ρ • iEXP was able to recover from delay buildup in few seconds compare to the Classic and iOFF. Figure 7.2. Frame arrival delay on the three audio qualities Symbiotic Audio Communication on Interactive Transport

25. Performance Analysis • Assuming a packet arrives at the receiver end at time tj • Expected arrival time for the packet is ej. • Referential jitter (refJitter(j)) = (tj - ej) • refjitter(j) is negative if packet j arrives early at the receiver end. • It can be buffered and played at the actual time. • refjitter(j) is positive, if packet j has arrived late at the receiver end. • The player will pause and wait for packet that arrive late. • The higher the delay, the higher the jitter experience by the frames. • The step jumps in the iEXP were much smaller than those in TCP-classic and iOFF. • Furthermore, this indicates that the interactive TCP reduces jitter. Figure 8.3. Referential Jitter on the three audio qualities Symbiotic Audio Communication on Interactive Transport

26. Performance Analysis • Symbiotic rate reduction that occurred as a result of the rate modification between the rate controller of the encoder and the symbiosis unit • Target bits and the actual bits generated by the encoder for each frame. • The rate retraction ratio of the symbiosis kicks in when a time out event is triggered and reported by iTCP during congestion. • The effect is observed on the plots as number of bits drop in accordance with the rate retraction ratio. Figure 7.4. Symbiotic Rate Reduction on the three audio qualities Symbiotic Audio Communication on Interactive Transport

27. Performance Analysis • Comparison of frame delay and acceptance ratio • A delay tolerance of d = 2, 4, and 6 seconds are introduced to measure the average frame delay and acceptance ratio. • iEXP mode experience a low delay and high acceptance ratio. • iTCP’s T-ware mechanism allowed the application to use sophisticated techniques to control the temporal qualities of its traffic. Table 7.2 Average Frame Delay and acceptance ratio Low delay & High acceptance ratio. Symbiotic Audio Communication on Interactive Transport

28. Performance Analysis • Overall stream compression • The overall delivery bits in the iEXP mode reduce to 80 – 90% of the original bits • iOFF and Classic cases shows no adaptation. • The file size of iEXP mode reduce significantly compared to the other modes. • The audio quality between the sending end and the receiving end is achieved perceptually by the temporal and spectral resolution. • iTCP provides a means of tradeoff of terrible frame delay for a satisfactory reduction of quality. Table 7.3 Percentage of total bits delivered for each mode Table 7.4 Sizes of the audio files Symbiotic Audio Communication on Interactive Transport

29. Performance Analysis • Aim of the iOFF mode --- Study the overhead introduced by the event notification service. • Increase total transmission time for all modes. • iOFF mode is higher than the classic TCP mode due to event notification service enabled. • iEXP mode is much smaller than the other modes • Indicates the application level performance outweighs the overhead. Figure 7.5 Overhead of the interactivity service Symbiotic Audio Communication on Interactive Transport

30. Conclusions • Aim: Reduce delay and jitter of audio traffic. • Solution: Design an interactive and friendly application that receives the state feedback from the network – Symbiotic encoder. • Achievements: Dynamic reduction of jitter and delay of time sensitive traffic during congestion. • Network congestion reduction by reducing the traffic at the source. • Trade off quality for delay and jitter. • The approach is simple and does not alter any network dynamic to be optimal (e.g fair queuing). • Its effect is entirely on the application layer. • It also further validates interactive transport control protocol (iTCP) usefulness and the efficiency through the idea of the event notification. • Technically, this scheme is not applicable to non-elastic traffic such as simple file transfer. Symbiotic Audio Communication on Interactive Transport

31. Future Work • Optimization: The parameters in the scheme are user defined but can be optimized with the symbiotic throttling model in [13]. The experiment can be performed using the other subsribable events(i.e. duplicate ACK). Symbiotic Audio Communication on Interactive Transport

32. [1] Andersen D., Bansal D., Curtis D., Seshan S., and Balakrishnan H., "System Support for Bandwidth Management and Content Adaptation in Internet Applications," Proc. of OSDI'OO, Oct. 2000, San Diego, CA [2] Balakrishnan H., Rahul H., and Seshan S., "An Integrated Congestion Management Architecture for Internet Hosts," Proc. of ACM SIGCOMM, Cambridge, MA, Sep 1999. pp.115-187. [3] Catherine Boutremans, Jean-Yve L Boudec, “Adaptive Joint Playout Buffer and FEC adjustment for the Internet Telephony,” 2003. [4] Eitan Altman, Chadi Barakat, Victor Ramos, “Queuing Analysis of Simple FEC schemes for IP Telephony,” 2001. [5] Floyd S., Handley M., Padhye J., and Widmer J.,“Equation-Based Congestion Control for Unicast Applications,” August 2000. SIGCOMM 2000 [6] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, “RTP: A transport protocol for real-time applications,” RFC 1889, 1996 [7] Handley M., Floyd S., Pahdye J., and Widmer J., “TCP Friendly Rate Control” (TFRC): Protocol [8] ISO/IEC 11172-3. Information technology -- Coding of moving pictures and associated audio for digital storage media at up to about 1,5 Mbit/s -- Part 3 [9] J. Mahdavi and S. Floyd, “TCP-Friendly unicast rate-based flow control,” in Draft posted on end2end mailing list, January 1997, http://www.psc.edu/networking/papers/tcp%20friendly.html. [10] Jacobson V. and Michael J. Karels, "Congestion Avoidance and Control," Computer Communication Review, vol. 18, no. 4, pp. 314-329, Aug. 1988. [11] Jacobson V., "Modified TCP Congestion Avoidance Algorithm," end2end-interest mailing list, April 1990 [12] Johnny Matta, Christine Pepin, Khosrow Lashkari, Ravi Jain, “A source and channel rate adaptation algorithm for AMR in VoIP using E-model,” June 2003. [13] Khan J. and Zaghal R., "Symbotic Rate adaptation for Time sensitive Elastic Traffic with Interactive Transport," Journal of Computer Networks, Elsevier Science, March. 2006. BIBLOGRPHY Symbiotic Audio Communication on Interactive Transport

33. [14] Khan J., Zaghal R., and Gu Q., "Rate Control in an MPEG-2 Video Rate Transcoder for Transport Feedback based Quality-Rate Tradeoff;" PV2002, Pittsburgh, P A, April 2002. [15] Khan J., Zaghal R., and Gu Q., "Dynamic QoS Adaptation for Time Sensitive Traffic with Transientware," IASTED WOC'03, Banff; Canada, July 2003. [16] Khan J. and Zaghal R., "Event Model and Application Programming Interface of TCP Interactive," Technical Report 'TR2003-02-02', Feb. 2003. [17] N. Shacham and P.M Kenney, “Packet recovery in high-speed networks using coding and buffer management,” in Proc. IEEE Infocom 1999 . [18] Pradhan P., Chiueh T. and Neogi A., “Aggregate TCP Congestion Control Using Multiple Network Probing,” Proc. Of the 20th International Conference on Distributing Systems, ICDCS 2000. [19] Rishi Sinha, Christos Paradopoulos, Chris Kyriakakis, “Loss Concealment for Multi-channel streaming audio,” June 2003. [20] Sisalem D. and Wolisz A., “Towards TCP-Friendly Adaptive Multimedia Applications Based on RTP,” Proc. of the 4th IEEE Symposium on Computers and Communications, 1998. [21] Stevens W. R., “TCP/IP Illustrated, Volume 1: The Protocols,” Addison-Wesley, 1994. [22] Wang R., Yamada K., Sanadidi M. Y., and Gerla M.,“TCP with sender-side intelligence to handle dynamic, large, leaky pipes,” IEEE Journal on Selected Areas in Communications, 23(2):235-248, 2005. [23] Wen-Tsai Liao, Janet J.C.Chen, Ming-Syan Chen Chen, “Adaptive Recovery Techniques for Real Time Audio Stream” [24] Wenyu Jiang, Hening Schulzrinne, “VoIP: Comparision and optimatization of packet loss repair methods in VoIP perceived quality under bursty loss,” May 2002 [25] Yi J. Liang, Eckehard G. Steinbach, Bernd Girod, “Voice over IP: Real-time voice communication over the nternet using packet path diversity,” October 2001. [26] TR2007-02-01 Test Audio Set: Symbiosis Audio Streaming with iTCP, Olufunke Olaleye and Javed I Khan, February 2007. BIBLOGRPHY(cont’d) Symbiotic Audio Communication on Interactive Transport

34. Questions and Comments Symbiotic Audio Communication on Interactive Transport