880 likes | 1.02k Views
Media Processing in the Network. Wei Tsang Ooi. Research Area. How to build multimedia network applications ?. Outline. Dalí Multimedia Library Problems and related work Research proposal. Question. How to build multimedia network applications ?. Dalí . Current Solutions.
E N D
Media Processing in the Network Wei Tsang Ooi
Research Area • How to build multimedia network applications ?
Outline • Dalí Multimedia Library • Problems and related work • Research proposal
Question • How to build multimedia network applications ?
Current Solutions • black box library code • easy to write but too high level • need finer optimization, resource control • write from scratch • too time consuming
What is Dalí • A small set of simple, fast, reusable primitives and abstractions for image, audio and video processing
Example: Black Box Code • gd GIF programming library 1.3 gdImagePtr im;FILE *in;in = fopen("mygif.gif", "rb");im = gdImageCreateFromGif(in);fclose(in);gdImageRectangle(im,0,0,8,8,red);
f = fopen(argv[1], "r"); BitStreamFileRead(bs,f,0); BitParserWrap(bp, bs); GifSeqHdrParse(bp, sh); rmap = ImageMapNew(); gmap = ImageMapNew(); bmap = ImageMapNew(); n = GifSeqHdrGetCtSize(sh); GifCtParse(bp,n,rmap,gmap, bmap); ih = GifImgHdrNew(); ph = PnmHdrNew(); status = GifImgFind(bp); GifImgHdrParse(bp, ih); h = GifImgHdrGetHeight(ih); w = GifImgHdrGetWidth(ih); r = ByteNew(w, h); g = ByteNew(w, h); b = ByteNew(w, h); i = ByteNew(w, h); GifImgParse(bp,sh,ih,i); ImageMapApply(rmap,i,r); ImageMapApply(gmap,i,g); ImageMapApply(bmap,i,b); Example: Dalí code
Byte Image Audio Buffer Bitstream MPEG Headers JPEG Headers Lookup Table Bit Image DCT Image Vector Image Convolution Kernel Dalí : Abstraction
Byte Image Audio Buffer Bitstream MPEG Headers JPEG Headers Lookup Table Bit Image DCT Image Vector Image ConvolutionKernel Dalí : Abstraction
Byte Image • Two dimensional array of bytes • Can be either physical or virtual physical image virtual image
Composable Abstraction • Byte images can be used to represent: • Gray scale images • RGB images • YUV images • Alpha channels
image smaller newh neww smaller = byte_new (neww, newh); byte_shrink_2x2(image,smaller);
target dx image 2dy smaller newh neww target= byte_clip(image,0,0,dx,2*dy);byte_set(target,0);target = byte_clip(image,3*dx,0,dx,2*dy);byte_set(target,0);
target dx image 2dy smaller newh neww target = byte_clip(image,dx,0,neww,newh);byte_copy(smaller,target);
image smaller newh neww target = byte_clip(image,2*dx,2*dy,neww,newh);byte_copy(smaller,target);target = byte_clip(image,0,2*dy,neww,newh);byte_copy(smaller,target);
Dalí strategies • specialized primitives • byte_shrink_2x2 • memory sharing • byte_clip • explicit memory allocation • byte_new
gop pichdr pic pichdr pic ... Dalí : MPEG headers seqhdr gophdr gop gophdr gop ... seqend
Dalí : MPEG headers seqhdr gophdr gop gophdr gop gop ... seqend pichdr pic pichdr pic ...
Basic Header Primitives • find • parse • skip • dump • encode
Example • Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
Dalí: Bitstream chunk of memory parser 1011001
Mode of Operations • File i/o static memory parser fread/fwrite
Mode of Operations • Network static memory parser socket
Mode of Operations • Memory map entire file parser
Strategies • expose structure • MPEG headers • explicit I/O • Bitstream abstraction
Novel Features of Dalí • explicit I/O • explicit memory management • predictable performance • programs are highly reconfigurable • push semantics to programmer • composable primitives, abstractions
Other Features of Dalí • fast • easily extensible • support wide ranges of formats (GIF, JPEG, MPEG-1, WAV, AVI, ..)
Dalí : Status • Currently in alpha release • C, Tcl and Java binding • Better performance than other tools (e.g. Rivl, mpeg_play)
Back to the question .. • How to build a multimedia network applications ?
capture process compress store source receiver de-compress display capture process compress Breaking it up .. on onemachine on twomachines
source receiver Optimization on onemachine 8 bit colors, 10 frame/s capture process compress store on twomachines de-compress display capture process compress
Optimization on onemachine 8 bit colors, 10 frame/s capture process compress store 8 bit colors, 10 frame/s on twomachines de-compress display capture process compress
Optimization • What if there are more than one receiver, and they have conflicting requirements ?
Problem • Network heterogeneity 100Mbps S R1 36kbps R2 128kbps R3
Suggested Solution • Layered Multicast • streams are hierarchically encoded • each layer is multicast into a group • incrementally combine the layers to refine the quality of the streams
Example Layer 3 Layer 2 Layer 1
Layered Multicast 100Mbps S R1 layer 1,2,3,4,5 36kbps R2 layer 1 128kbps R3 layer 1,2
Layered Multicast • Receivers find out if they can join the next higher layer by experiments • Join the layer, if packet drops/low throughput detected, leave it
Problems • Needs layered encoding • More groups means more states to maintained • Uses a lot of multicast addresses
Alternative Approach • Move computation into the network • where to run ? • when to run ? • how to specify the computation ?
Related Work • QOS Filters (Yeadon 96) • where to run ? • when to run ? • how to specify the computation ?
Filters • transform one or more input streams into an output stream • Examples : • drop frames • decode frames • mix input audio • transcode to lower quality
filters filters flow spec Architecture R1 S R2 session mgr
filters filters Architecture ack R1 S R2 session mgr
filters filters Architecture data R1 S R2 session mgr