Multimedia Synchronization

1. Multimedia Synchronization Brian P. BaileySpring 2006

2. Announcements

3. MM Synchronization • Applications composed of more than one media (at least one continuous) • Express desired relationships • content, spatial, temporal, and interaction • combinations of each

4. Content and Spatial Relationships • Content • define how views relate to data sources • e.g., a graph linked to a table of data • Spatial • define relative positions of media objects • subdivide the space; express relationships • e.g., pack command in Tcl, layout managers in Java, tables in HTML, etc.

5. Temporal Relationships • Define how media are coordinated in time • audio should not drift from video by > 80ms • voice narration should accompany a slide and end when user navigates elsewhere • display different caption for each video scene; and update it in response to user interaction • Intra-media and inter-media relationships • Time-independent and dependent media

6. Lip Synchronization Left: audio after video; Right: audio before video

7. Lip Synchronization Nottolerable Notdetectable Nottolerable Tolerable Tolerable

8. Tele-pointer Synchronization Left: pointer before audio; Right: pointer after audio

9. Synchronization Guidelines • Lip synchronization within 80ms • video before audio is more tolerable • Other fine-grained synchronization should typically be within range of 500ms

10. Interaction Relationships • Define how interaction affects playback • e.g., if user transitions to next slide in narrated slide show, narration should change as well • Classes of interaction • navigation, participation, and control • asynchronous and synchronous

11. Synchronization Model • Enables expression of media and synchronization relationships • An effective model should support: • spatial and temporal relations (fine & coarse) • rich interaction (beyond VCR control) • efficient runtime (interaction monitoring) • be usable and comprehensible

12. Timeline Hierarchical Petri net Interval Event-based Common threads provide language to express relationships runtime system to monitor relationships policies to enforce relationships Models

13. Timeline Model • Uses a single global timeline • Actions triggered when the time marker reaches a specific point along timeline

14. Example • Define a timed sequence of images, each image has a caption that goes with it I1 I2 I3 C1 C2 C3 t1 t2 t3

15. Rule language At (t1), show (I1, C1) At (t2), show (I2, C2) At (t3), show (I3, C3) Visual environment Example (Cont.)

16. Hierarchical Model (SMIL) • Based on sequential and parallel • Apply operators to only the start/end points of each media object I1 I2 I3 I1 T1

17. Example • Narrated slide show • image, text, audio on each slide • select link to move to the next slide … S2 A2 T2 I2 S1 A1 T1 I1

18. Timed Petri Nets • tokens, places, transitions, and arcs

19. Example Specify audio video synchronization 11ms 11ms 11ms 11ms 11ms 33ms 33ms

20. Interval Model • 13 relationships between two intervals B A Before A Starts B A B Meets A Ends Equal A B B A During Overlaps A B B

21. Event Model (Nsync) • Associate actions with expressions • Expressions may contain scalars, clocks, variables, relations, and connectives • When the expression becomes TRUE, invoke associated action When “Time > Q.end + 5 && !Response” Answer=WRONG

22. Background and Time Model • Each media object attached to a clock • Clock implements logical time • Value = Rate * System + Offset • Express temporal behavior as relationships among clocks • Interactive events tied to variables

23. Example: Delayed Transition Overview More Info? No Yes More Info More Info Detailed Narration

24. Model Specification When “Narration >= Overview &&!MoreInfo” NextSlide When “Narration >= Overview && MoreInfo” PlayDetails When “Narration >= Overview + Details” NextSlide Narration: narration’s logical timeline Overview: normal transition point Details: additional narrative details MoreInfo: records kitchen info status

25. Reactive Interface

26. Model Specification When “Video >= 0 && Video < T1” Select Kitchen When “Video >= T1 && Video < T2” Select Deck When “Video >= T2 && Video <= T3” Select Yard

27. Expression Evaluation • Propositional logic breaks down • returns logic value only at present time • requires polling to catch future transitions • Predictive logic • returns logic value at present time along with a prediction of any future transition • eliminates need for intermittent polling/timers

28. Predictive Logic States • WBT(t) False now, but Will Become True at future time t • WBF(t) True now, but Will Become False at future time t

29. Prediction Example When “Video > 10” Action (then - now)t = ----------------- rate 10 Rate = 1 t = (10 - 0) / 1 = WBT(10) Video Time System Time 0

30. Prediction Example When “Video > 10” Action (then - now)? = ----------------- rate 10 Rate = 1 Rate = 2 ? = (10 - 3) / 2 = WBT(3.5) Video Time ? System Time 0 3

31. Evaluation Rules for “AND” WBT(x) && WBT(y) = WBT( max(x, y) ) WBF(x) && WBF(y) = WBF( min(x, y) ) WBF(x) && WBT(y) = WBT(infinity) if (x < y) WBT(y) then WBF(x) otherwise

32. Take Home Exercise WBT(x) | | WBT(y) = ? WBF(x) | | WBF(y) = ? WBF(x) | | WBT(y) = ?

33. Pros • Complements current languages • adds ability to express combinations of interactive and temporal behavior • syntax can easily be translated into mark up • Predictive logic useful in run-time engines • eliminates need for polling/timers • enables look-ahead pre-fetching

34. Cons • Difficult to visualize rule propagation • makes system difficult to debug • Rules are not groups into hierarchies • enable divide and conquer strategy • Lack of scope • all rules always active • guard actions with complex expressions

35. Take Home Exercise • Be able to model relationships within relatively simple applications • Weigh tradeoffs between models