Limin Liu, Member, IEEE Zhen Li, Member, IEEE Edward J. Delp , Fellow, IEEE CSVT 2009

1. Limin Liu, Member, IEEE Zhen Li, Member, IEEE Edward J. Delp, Fellow, IEEE CSVT 2009 Efficient and low-complexity Surveillance video compression using backward-channel aware wyner-ziv video coding

2. Outline • Introduction • Low-complexity Video Encoding Approaches for SVC • SVC Using Wyner-Ziv Video Coding • SVC Using Backward-Channel Aware Wyner-Ziv Video Coding • Experimental Results • Conclusion

3. Basic Diagram of Video Surveillance Systems

4. Introduction • Video surveillance system has been widely used to enhance public safety and privacy protection. • Needs efficient transmission and storage. • H.264/AVC is high computational complexity at the encoder. This can increase the cost of a surveillance system.

5. Video Compression Design Considerations for Surveillance Video • General case • Movie studio only needs to encode once to put on DVD but decoded by the consumers many times. • Heavy encoder, light decoder. • Surveillance systems • Encoder is implemented in a simple and low-cost video surveillance camera. • The video is decoded and analyzed at central server. • Light encoder, heavy decoder. • Other requirements • High coding efficiency • Error resilience requirements

6. Surveillance Video Compression(SVC) • Low-complexity encoder based on Wyner-Ziv coding principles to address the tradeoff between computational complexity and coding efficiency. • Backward-Channel Aware Wyner-Ziv (BCAWZ) video coding to improve coding efficiency while maintaining low-complexity at the encoder. • Error resilience scheme for BCAWZ to keep reliable transmission in the backward-channel.

7. Low-complexity Video Encoding Approaches for SVC • Slepian-Wolf Theorem • Construct correlation between X and Y • RX ≥ H(X|Y ) RY ≥ H(Y |X) RX + RY ≥ H(X, Y ) • Regardless of its access to side information Y , encoder A can encode X with high fidelity as long as the decoder A has access to Y. Lower bound

8. Low-complexity Video Encoding Approaches for SVC • Wyner-Ziv Video Coding (WZVC) • Temporal correlation among frames is exploited at the decoder instead of encoder. • Each frame is independently encoded at the encoder. • Computational intensive job of motion estimation is shifted to decoder. • WZVC is an ideal choice for SVC due to the light encoder characteristics.

9. SVC Using Wyner-Ziv Video Coding Initial estimate Turbo code ->parity bits LDPC code ->syndrome bits video

10. SVC Using Wyner-Ziv Video Coding • Video sequence is divided into two groups. • Key frames • Encoded by H.264 INTRA encoder. • Serve to side information at decoder • Wyner-Ziv frames • Encoded by channel coding method. • Turbo code  parity bits • Low-density-parity-check(LDPC) code  syndrome bits

11. SVC Using Wyner-Ziv Video Coding • Key frames • Decoded by the H.264 INTRA decoder. • Supply side information to other frames by find initial estimate. • Initial estimate is derived from previously reconstructed key frames. • Use co-located pixel value in (n-1)th frame. • Take average of the co-located pixel value at (n-1)th and (n+1)th frames. • Wyner-Ziv frames • Quality is low, temporal correlation is not fully exploited. • To obtain higher quality, motion estimation can be done at decoder.

12. SVC Using Wyner-Ziv Video Coding • Higher quality initial estimate • method1 • Search for the motion vector MVn-1 of the co-located block in (n-1)th frame as the predictor of current block. • The reference block in (n-1)th frame is the initial estimate. • method2 • Obtain MVF and MVB • Use to find PF1 Use to find PF2 Use to find PB1 Use to find PB2 • Initial estimate

13. SVC Using Wyner-Ziv Video Coding • The channel decoder uses initial estimate and the incoming parity or syndrome bits to decode the frame. • Assume decoder can communicate with the encoder to request more bits until correctly decoded. • Wyner-Ziv video codec formulates the video decoding problem as an error correction problem. • If the parity or syndrome bits are lost or corrupted during transmission, it will add complexity. • Standard video codec may not be correctly decoded when bits are lost and error can propagate to the following frame.

14. SVC Using Backward-Channel Aware Wyner-Ziv Video Coding • WZVC with INTRA key frames coding efficiency is higher than INTRA but much lower than INTER. • The distance between two neighboring key frames too far or too close will result in lower coding efficiency and degrading the quality of side information. • Use Backward-Channel Aware Motion estimation (BCAME). • The basic idea BCAME is to perform motion estimation at decoder and send the motion information back to encoder through backward-channel. • May increase the latency and cause problem for some SVC.

15. SVC Using Backward-Channel Aware Wyner-Ziv Video Coding Even frames Odd frames(excluding 1, 3) video Mode selection 1, 3 frames  H.264 INTRA encoder  H.264 INTRA decoder

16. SVC Using Backward-Channel Aware Wyner-Ziv Video Coding • Encode first and third frame as INTRA frames. • All the other odd frames are encoded with BCAME called BP frames. • All the even frames are encoded as a Wyner-Ziv frame.

17. SVC Using Backward-Channel Aware Wyner-Ziv Video Coding • BP frames • Mode1Mode2 • Motion vectors are sent back to encoder. • Encoder do mode selection by MSE or SAD. • Encoder use received motion vector with previous reconstructed BP frames to generate MC reference for current BP frame. • Residual between current BP and reference is than transformed and entropy-coded.

18. Error Resilience In Backward-Channel Aware Wyner-Ziv Video Coding • Problem • If there is error or delay at backward-channel. Motion vector is not updated and encoder continues to use it. • Encoder and decoder use different motion vectors in motion compensation for the same frame.

19. Error Resilience In Backward-Channel Aware Wyner-Ziv Video Coding decoder encoder Report error when receive index not match the frame number at the encoder Coding key frames as BP frames Insert the index of the frame as header info before sending channel OK De-synchronization Synchronization Encodes the key frames as INTRA and send frame type to decoder Two-stage error-resilient procedure

20. Experimental Results • H.264/AVC JM8.0 • WZVC with INTRA, INTER or BP key frames • Odd-frames -> key frame • Even-frame ->Wyner-Ziv frame • Block-size -> 8*8 • WZVC with INTER key frames • First-frame -> INTRA • Other key frames -> P frame • H.264/AVC INTER coding • First-frame -> INTRA • Other frames -> IBPBP GOP structure • Quarter-pel motion search • Search range -> 32 • Number of reference frames -> 3

21. Experimental Results • Comparison • WZVC with INTRA key frames • 3-4.5 dB gain for low motion sequences • 1.5-2 dB gain for high motion sequences • H.264 INTRA coding • Discussions • Low motion sequence is more continuous. Hence motion vectors derived from previous reconstructed frame is good. • Many surveillance video are low motion. • WZVC encoder complexity is comparable to H.264 INTRA coding. • WZVC can provide significant rate reduction compared to H.264 INTRA coding.

22. Experimental Results • Comparison • WZVC with INTRA key frames • 7-8 dB loss • 10-13 dB loss • H.264 INTER coding • Discussions • INTER coding is much more efficient than INTRA coding especially in low motion sequences.

23. Experimental Results • Comparison • BCAWZ • 5-7 dB gain • WZVC with INTRA key frames • Comparison • BCAWZ • 1.5-2 dB loss • H.264 INTER coding • Discussions • This gain is achieved with marginal complexity increase at the encoder. • BCAWZ • Two motion vector send from decoder • H.264 INTER coding • 4*2M*2M candidate motion vector for a search range of M. (in this paper is 32)

24. Experimental Results • Comparison • BCAWZ • 2-3 dB loss • WZVC with INTER key frames • Discussions • Use INTER key frames significantly increase the encoder complexity.

25. Experimental Results • Comparison • WZVC with INTRA key frames • BCAWZ (BP key frame) • Discussions • The visual quality of BP frame is significantly higher than WZVC with INTRA key frames.

26. Experimental Results • Backward-channel /forward-channel bandwidth • Lower rate : 10-15% • Higher rate : 5% • This usage can be satisfied in SVC. • Error resilience performance for BCAWZ • Without error resilience • Sharply drops • With error resilience • Effectively recover from backward-channel delay or erasure.

27. Conclusion • Present a low-complexity video encoding framework for surveillance video compression and transmission to address the tradeoff between computational complexity and coding efficiency. • BCAWZ can achieve significantly higher coding efficiency than H.264/AVC INTRA coding as well as existing Wyner–Ziv video coding methods and is close to H.264/AVC INTER coding, while maintaining similar coding complexity with INTRA coding. • Error resilience scheme for BCAWZ to address the concern of reliable transmission in the backward-channel, which is essential to the quality of video data for real-time and reliable activity and event analysis.