560 likes | 566 Views
Explore Dali Virtual Machine for efficient multimedia data processing, video editing, and content conversion. Learn about its design goals, bytecode examples, and strategies for performance optimization.
E N D
Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)
Motivations • We know how to encode, transmit and store multimedia data today • People start looking into ways to process multimedia data
Video Editing • Concat, cross-fade, overlay, cut and paste
Gateway • Convert from one format to another • Change compression rate transcoder output video input video format B (format A)
DBMS “ Find all movies in the database that contain this image “
Set-top Boxes • Subtitling and close captioning • Mixing of incoming movies
Lecture Browser • Match video to slide • Switch between video streams
Current Solutions • Black box C code (mpeg_play) • Hard to reuse/adapt/break apart • Unpredictable performance • Standard library (PPM, IJG) • Least common denominator (RGB for all) • Video frames --> RGB --> gray
History • Rivl - high level scripting language for multimedia processing • Still suffers from “black box” problem
Experience • Real cost of processing • Inherently complex operations • Projective transform • Layered operations • Video decoding • Simple operations but lots of data • Copy
Dali Architecture User Expert User Dali Code DVM Code Dali Virtual Machine Dali Compiler MMX TriMedia C
Today’s Talk DVM Code Dali Virtual Machine
DVM Code : Design Goal • Small number of composable abstractions • Simple, predictable opcodes • High performance • Target for compiler • Easy to extend
Abstractions • Byte image • Bitstream • MPEG • JPEG • Bit masks • PCM
Abstractions • Byte image • Bitstream • MPEG • JPEG • Bit masks • PCM
Byte Image • Two dimensional array of byte • Can be either physical or virtual physical image virtual image
Byte Image Can Represent • Gray scale image • RGB image • YUV image • Alpha channel
image smaller newh neww smaller = byte_new (neww, newh); byte_shrink_2x2(image,smaller);
target dx image 2dy target= byte_clip(image,0,0,dx,2*dy);byte_set(target,0);target = byte_clip(image,3*dx,0,dx,2*dy);byte_set(target,0);
target dx image smaller newh neww target = byte_clip(image, dx, 0, neww, newh);byte_copy(smaller, target);
dx image 2dy smaller newh neww target = byte_clip(image, 2*dx, 2*dy, neww, newh);byte_copy(smaller, target);target = byte_clip(image, 0, 2*dy, neww, newh);byte_copy(smaller, target);
DVM Code Dx = 0.25 * byte_width(image);dy = 0.25 * byte_height(image);neww = 0.5 * byte_width(image);newh = 0.5 * byte_height(image);smaller = byte_new (neww,newh);for (i = 0; i < n; i++) { byte_shrink_2x2(image,smaller); Target = byte_clip(image,0,0,dx,2*dy); byte_set(target,0); target = byte_clip(image,3*dx,0,dx,2*dy); byte_set(target,0); target = byte_clip(image, dx, 0, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 2*dx, 2*dy, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 0, 2*dy, neww, newh); byte_copy(smaller, target);}
General Dali Strategies • Specific instruction • byte_shrink_2x2, byte_shrink_4x4, etc • Explicit memory allocation • byte_new • Reduce data • byte_clip • Composable abstraction • byte image
Performance run fractal with n = 4 on 800x600 gray scale image about 21 frames/sec for 320x240 video
Abstractions • Byte image • Bitstream • MPEG • JPEG • Bit masks • PCM
MPEG • Getting important and pervasive • Complex format • Complex code (mpeg_play) • Most decoders provides GetNextFrame() interface • Random access, direct transfer difficult
Abstraction for MPEG Video seq hdr gop hdr gop hdr seq end . . . gop gop
Abstraction for MPEG Video seq hdr gop hdr gop hdr seq end . . . gop gop pic hdr pic hdr . . . pic pic
DVM Code Examples • Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
DVM Code Examples • Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
DVM Code Examples • Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
DVM Code Examples • Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
Concat 2 Video Sequences inbs1 inbs2 hdr gop hdr gop outbs hdr gop hdr gop
DVM Code • Concat two sequences While not end_of(inbs1) { gop_hdr_dump(inbs1, outbs); gop_dump(inbs1, outbs); } While not end_of(inbs2) { gop_hdr_dump(inbs2, outbs); gop_dump(inbs2, outbs); }
Performance Analysis • Decode MPEG sequences to PPM • No output • Run on 3 different sequences
MPEG Strategies • Expose structure • Gop header, gop instead of just frames • Break up layer • Parse, decode, color space conversions
Show Time ! • Decode a MPEG sequence • For each frame scale it to “stamp” size • Create a contact sheet
Abstractions • Byte image • Bitstream • MPEG • JPEG • Bit masks • PCM
What Is Bitstream ? • Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
Bitstream chunk of memory parser 1011001
Mode of Operations • File i/o static memory parser fread/fwrite
Mode of Operations • Network static memory parser socket
Mode of Operations • Memory map entire file parser
mpeg system stream inbs audio video audio video audio video
mpeg system stream inbs audio video audio video audio video filter videobs
mpeg system stream inbs audio video audio video audio video filter videobs parser
Bitstream Strategies • Predictable performance • Explicit I/O