340 likes | 466 Views
School of Computing Science Simon Fraser University, Canada. Rate-Distortion Optimized Streaming of Fine-Grained Scalable Video Sequences Mohamed Hefeeda & ChengHsin Hsu MMCN 2007 31 January 2007. Motivations. Multimedia streaming over the Internet is becoming very popular
E N D
School of Computing ScienceSimon Fraser University, Canada Rate-Distortion Optimized Streaming of Fine-Grained Scalable Video Sequences Mohamed Hefeeda & ChengHsin Hsu MMCN 2007 31 January 2007
Motivations • Multimedia streaming over the Internet is becoming very popular • More multimedia content is continually created • Users have higher network bandwidth and more powerful computers • Users request more multimedia content • And they look for the best quality that their resources can support
Motivations (cont’d) • Users have quite heterogeneous resources (bandwidth) • Dialup, DSL, cable, wireless, …, high-speed LANs • To accommodate heterogeneity scalable video coding: • Layered coded stream • Few accumulative layers • Partial layers are not decodable • Fine-Grained Scalable (FGS) coded stream • Stream can be truncated at bit level
Motivations (cont’d) • Goal: Optimize quality for heterogeneous receivers • In general setting • FGS-coded streams • Multiple senders with heterogeneous bandwidth and store different portions of the stream • Why multiple senders? • Required in P2P streaming: • Limited peer capacity and Peer unreliability • Desired in distributed streaming environment: • Disjoint network path Better streaming quality
Our Optimization Problem • Assign to each sender a rate and bit range to transmit such that the best quality is achieved at the receiver. • Consider a simple example to illustrate the importance of this problem
Example: Different Streaming Schemes Non-scalable Layered
Optimal FGS Scalable Example: Different Streaming Schemes FGS Scalable 1. FGS enables us to get the best quality from senders 2. However, there too many allocation options, and we need to carefully choose the optimal one
Problem Formulation • First: single-frame case • Optimize quality for individual frames • Then: multiple-frame case • Optimize quality for a block of frames • More room for optimization • Details are presented in the extended version of the paper
Input Parameters • T : fixed frame period • n : number of senders • bi: outgoing bandwidth of sender i • bI: incoming bandwidth of receiver • si: length of (contiguous) bits held by sender i • Assume w.l.g. s1 <= s2 <= …… <= sn
Outputs • Allocation: A = {(Δi , ri) | i = 1, 2, …, n} • Δi: number of bits assigned to i • ri: streaming rate assigned to i • Specifies: • Sender 1 sends range [0,Δ1 -1] at rate r1 • Sender 2 sends range [Δ1 ,Δ1+Δ2 -1] at rate r2 • … • Sender i sends range at rate ri
Integer Programming Problem • Minimize distortion • Subject to: • on-time delivery • assigned range is available • assigned rate is feasible • Aggregate rate not exceeds receiver’s incoming BW
How do we Compute Distortion? • Using Rate-Distortion (R-D) models • Map bit rates to perceived quality • Optimize quality rather than number of bits • Approaches to construct R-D models • Empirical Models: Many empirical samples expensive • Analytic Models: Quality is a non-linear function of bit rate, e.g., log model [Dai 06] and GGF model [Sun 05] • Semi-analytic Models: A few carefully chosen samples, then interpolate, e.g., piecewise linear R-D model [Zhang 03] • Detailed analysis of R-D models in our previous work[Hsu 06]
The Linear R-D Model • Within each bitplane, approximate R-D function by a line segment • Line segments of different bitplanes have different slopes
Visual Validation of Linear R-D Model Mother & Daughter, frame 110 Foreman, frame 100
Rigorous Validation of Linear R-D Model • Average error is less than 2% in most cases
Using the Linear R-D Model • Let yi be number of bits transmitted from bitplane i • Distortion is: • d : base layer only distortion • gi : slope of bitplane i • z : total number of bitplanes
Integer Linear Programming (ILP) Problem • Linear objective function • Additional constraints • number of bits transmitted from bit plane h does not exceed its size lh • bits assigned to senders are divided among bitplanes
Solution of ILP is a Valid FGS Stream • Lemma 1: • An optimal solution for the integer linear program produces a contiguous FGS-encoded bit stream with no bit gaps • Proof sketch • minimizing • Since g1 < g2 < … <gn<0 (line segment slopes), • the ILP will never assign bits to yi+1 if yi is not full
Linear Programming Relaxation • Solving ILP problem is expensive • Solution: Transform it to Linear Programming (LP) problem • Relax variables to take on real values • Objective function and constraints remain the same
Efficient Rounding Scheme • Solve LP • Result is real values • Then, use the following rounding scheme for solution of the ILP
Correctness/Efficiency of Proposed Rounding • Lemma 2 (Correctness) • Rounding of the optimal solution of the relaxed problem produces a feasible solution for the original problem • Lemma 3 (Efficiency: Size of Rounding Gap) • The rounding gap is at most nT + n, where n is the number of senders and T is the frame period • (Extreme) Example: n = 30 senders, T = 30 fps ==> gap is 32 bits • Indeed negligible (frame sizes are in order of KBs)
FGSAssign: Optimal Allocation Algorithm • Solving LP (using Simplex method for example) may still be too much • Need to run in real-time on PCs (not servers) • Our solution: FGSAssign • Simple, yet optimal, allocation algorithm • Greedy: Iteratively allocate bits to sender with smallest si (stored segment) first
Pseudo Code of FGSAssign • Sort senders based on si, s1 ≤ s2 ≤ …… ≤ sn; • x0 = …… = xn = 0; Δ1 = …… = Δn = 0; ragg = 0; • for i = 1 to n do • xi = min(xi−1 + biT, si); • ri = (xi − xi−1)/T ; • if (ragg + ri < bI ) then • ragg = ragg + ri; • Δi = xi − xi−1; • else • ri = bI − ragg; • Δi = T × ri; • return • endfor
Optimality of FGSAssign • Theorem 1 • The FGSAssign algorithm produces an optimal solution in O(n log n) steps, where n is the number of senders. • Proof: see paper • Experimentally validated as well.
Multiple-Frame Optimization • Solve the allocation problem for blocks of m frames each • Objective: minimize total distortion in block • Why consider multiple-frame optimization? • More room for optimization • Solve the problem less often
Multiple-Frame Optimization: Why? Distortion (MSE) Bit rate • More room for optimization: higher quality and less quality fluctuation
Multiple-Frame Optimization • Formulation (in extended version of the paper): • Straightforward extension to single-frame with lager number of variables and constraints • Computationally expensive to solve • Our Solution: mFGSAssign algorithm • Heuristic (close to optimal results) • Achieves two goals: • Minimize total distortion in a block of frames • Reduce quality fluctuations among successive frames • Pseudo code and analysis: see extended version of the paper
Experimental Setup • Software used • MPEG-4 Reference Software ver 2.5 • Augmented to extract R-D model parameters • Algorithms implemented (in Matlab) • LP solutions using Simplex for the single-frame and multiple-frame problems • FGSAssign algorithm • mFGSAssign algorithm • Nonscalable algorithm for baseline comparisons
Experimental Setup (cont’d) • Streaming scenarios • Four typical scenarios for Internet and corporate environments • Testing video sequences • Akiyo, Mother, Foreman, Mobile (CIF) • Sample results shown for Foreman and Mobile
Single Frame: Quality (PSNR) Mobile, Scenario III Foreman, Scenario I • Quality Improvement: 1--8 dB • FGSAssign is optimal
Multiple Frame: Quality (PSNR) Foreman, Scenario II Mobile, Scenario III • Scalable: higher improvement than single frame • mFGSAssign: almost optimal (< 1% gap)
Fluctuation Reduction Foreman, Scenario II Mobile, Scenario III • Small quality fluctuations in successive frames
Conclusions • Formulated and solved the bit allocation problem to optimize quality for receivers in general settings • Multiple heterogeneous senders • Considered single and multiple frame cases • Approach • Nonlinear problem integer linear program • Using linear R-D model • Integer linear program linear program • Using simple rounding scheme • Proposed efficient algorithms • FGSAssign: optimal and efficient • mFGSAssign: close to optimal in terms of average distortion, reduces quality fluctuations, runs in real time • Significant quality improvements shown by our experiments
Thank You! Questions?? • All programs/scripts/videos are available: http://www.cs.sfu.ca/~mhefeeda