Media Processing Workflow

1. Media Processing Workflow Alex Zambelli Media Technology Evangelist Developer & Platform Evangelism alexzam@microsoft.com

2. Workshop Overview • Silverlight media capabilities • Smooth Streaming 101 • 15 min break • Live Smooth Streaming • Workflow and architecture • 15 min break • Encoding for Smooth Streaming

3. A Brave New World • Encoding 320x240 video on a laptop is “so 2002” • HTTP adaptive streaming has entirely changed the game in the past 2 years • The future for Silverlight is all Smooth Streaming, all HD • Video quality is better than ever, but at the price of complexity and bandwidth • Don’t worry, it only gets easier from here

4. Silverlight Media Formats • Windows Media • ASF file format (*.asf, *.wmv, *.wma) • VC-1 (WMV9), WMV8, WMV7 video codecs • WMA Pro, WMA Standard, MP3 audio codecs • MPEG-4 • MP4 file format (*.mp4, *.3gp, *.mov) • H.264 video codec (Baseline, Main, High profiles) • AAC audio codec (AAC-LC profile)

5. Silverlight Media Extensibility • Smooth Streaming • VC-1 and H.264 video • WMA Pro, WMA Standard and AAC-LC audio • Supported via Smooth Streaming Media Element extension • MediaStreamSource lets developers add support for 3rd party codecs, formats and delivery methods

6. Silverlight Media Protection • Silverlight DRM, powered by PlayReady • PlayReady AES-256 encryption • Windows Media DRM encryption • Silverlight 3 • VC-1 and WMA encryption only • Live DRM (Direct License Acquisition) only • Silverlight 4 • H.264 and AAC encryption • Offline DRM

7. Silverlight Media Delivery • Windows Media Services streaming • Windows Server 2008 and 2003 • Unicast (WMS HTTP) • Multicast • HTTP progressive download • Any HTTP 1.0/1.1 web server • IIS Smooth Streaming • Windows Server 2008 (IIS7)

8. Silverlight Media Framework • No need to reinvent the wheel every time a Silverlight video player is needed • Silverlight Media Framework:http://smf.codeplex.com • All essential playback features plus advanced Smooth Streaming features • DVR, Fast Forward, Rewind, Replay, Slow Motion • Markers, captions, dynamic ad insertion • Rich analytics and logging

9. Media Delivery Methods • Unicast • Traditional Streaming • Progressive Download • HTTP-based Adaptive Streaming • Multicast • IP Multicast

10. Unicast Many one-to-one streams Requires more bandwidth per user May require more servers For private and public networks Origin Server

11. Multicast A single one-to-many stream Uses bandwidth of only one stream Requires multicast-enabled networks Typically requires fewer servers Origin Server

12. Unicast Methods Movie HTTP Adaptive Streaming Video @ 01:04? Video @ 01:06? Video @ 01:08? Traditional Streaming Play Seek Pause Progressive Download Movie

13. Internet Streaming Challenge • Traditional streaming designed for efficient media delivery • Works well in small networks but on the Internet suffers from serious drawbacks: • Doesn’t scale to today’s Internet audience • Doesn’t take into account today’s Internet structure and organization • Designed for network conditions less volatile than Internet traffic

14. HTTP Adaptive Streaming • Hybrid media delivery method • Acts like streaming but is in fact a series of short progressive downloads • Video and audio delivered as series of small files over HTTP • Built for the Web • Leverages existing HTTP caches • Scales exceptionally well to meet high demand • Resilient to network unpredictability • Client can seamlessly switch video quality and bit rate based on perceived network bandwidth and CPU resources • Applicable to both on-demand and live delivery

15. Windows Server Media Delivery IIS Media Services • Traditional Streaming • Unicast • WMS RTSP • WMS HTTP • Multicast • WMS Multicast • Progressive Download • Bit Rate Throttling • HTTP Adaptive Streaming • Smooth Streaming

16. Smooth Streaming • Smooth Streaming • Microsoft implementation of HTTP-based adaptive streaming • Best of both worlds • Responsive and efficient like streaming • Cheap and scalable like progressive download • Superior user experience • No buffering, no stutter • Instant start, instant seek • Seamless bit rate switching based on bandwidth and CPU

17. Multi Bit Rate Revisited 00:27.24 03:47.16 05:05.04 07:33.10 2.4 Mbps 1.5 Mbps 700 kbps 300 kbps

18. Adapting Bit Rate in Real-Time 300K @ 00:00? 700K @ 00:02? 2.4M @ 00:04? 1.5M @ 00:06? 2.4M @ 00:08? 300K (start quickly) 00:00 00:02 00:04 00:06 00:08 700K (good network) 2.4M (great network) 2.4M 1.5M (glitch) 2.4M (play on…) … Bit Rate Heuristics 300K

19. Smooth Streaming Design • Smooth Streaming File Format based on MP4 (ISO Base Media File Format) • Video is encoded and stored on disk as one contiguous MP4 file • One file per bit rate • Each video Group of Pictures (GOP) is stored in a separate Movie Fragment box • This allows accurate fragmentation at key frames • Contiguous file is virtually split up into chunks when responding to a client request

20. Smooth Streaming File Format File Type (ftyp)

21. On-Demand Presentations • MP4 files containing video/audio/data fragments • New file extensions instead of *.mp4 • *.ismvfor files containing video and audio tracks, or only video tracks • *.ismafor files containing only audio tracks • On-demand server manifest file • File extension: *.ism • Describes the relationships between media tracks, bitrates and files on disk • Uses SMIL 2.0 XML format • Client manifest file • File extension: *.ismc • Describes available streams, codecs, bitrates, resolutions, metadata • Uses XML format

22. On Demand Server Manifest

23. On Demand Client Manifest

24. Client-Server Communication • All requests to the server are formed as RESTful URLs • Client always first requests the client manifest:http://iis.net/video.ism/Manifest • After parsing the manifest the client begins making requests for fragments by indicating bit rates and offset times it learned from the manifest:http://iis.net/video.ism/QualityLevels(350000)/Fragments(video=40040000)http://iis.net/video.ism/QualityLevels(48000)/Fragments(audio=62293333) • Every valid HTTP request returns a file of MIME type “video/mp4”

25. Serving Fragments On Demand • IIS7 Smooth Streaming handler interprets the RESTful URL request • It maps the quality level to a physical ISMV file based on the server manifest (*.ism) and calculates the fragment location within the contiguous ISMV file based on the time offset • Because the wire format (fragment) is just a subset of the disk format (MP4), the extraction process is simple • No re-muxing necessary • Dynamic stream switching logic is fully implemented in Silverlight application code – no server-side detection

26. Demo • DEMO: • Setting up On Demand publishing points • Using Fiddler to monitor Smooth Streaming traffic

27. Live Smooth Streaming • Live publishing point identified by *.isml extension • Client-server communication very similar to on-demand • Server continually appends MP4 file with new fragments from encoder and makes them available to client • This allows DVR-like seeking within the existing archive

28. Encoder-Server Communication • Encoder pushes fragmented MP4 stream to server in body of a long-running HTTP POST requesthttp://iis.net/live.isml/stream(720p) • Encoder inserts Live Server Manifest Box into start of MP4 stream • Contains a SMIL-formatted Live Server Manifest very similar to On-Demand Manifest • Describes all tracks in that encoder’s stream

29. Encoder-Server Communication • Each sent fragment contains a box with absolute time and duration • Timestamps are typically derived from the input source Linear Time Code (LTC) • If multiple encoders are used to encode the same video source, it’s essential that a time code generator is used to keep their LTC in sync

30. Serving Live Fragments • Server parses the encoder manifests and starts collecting MP4 fragments • Server builds a cumulative runtime index in memory for all incoming fragments • Server stores fragments into a temporary DVR ISMV or a permanent ISMV archive • When the pub point is shut down, server generates ‘mfra’ index box for ISMV archive

31. Live Client Workflow • Client requests the latest manifesthttp://iis.net/live.isml/Manifest • Server adds up all the encoder manifests and the runtime fragment index and sends the latest version to the client • Every fragment contains information about the next fragment • No need to repeatedly download manifest • Client builds its own cumulative runtime index based on the lookahead info in the fragments

32. Demo • DEMO: • Basic Live Smooth Streaming workflow

33. Live Smooth Publishing Points • ISML files are SMIL 2.0 formatted XML configuration files • Live publishing points can be created via the IIS Manager UI or by manually creating ISML files on server • Both push and pull publishing models are supported • Live publishing points can also push to and pull from other live publishing points

34. Live Publishing Point – IIS Mgr

35. Live Publishing Point - ISML

36. ISML Syntax • sourceType • Push: source streams will be pushed to pub point • Pull: pub point will pull streams from listed URLs • publishing • Streams: pub point can serve streams to other live pub points • Fragments: clients can request fragments • Archives: archiving is enabled • lookaheadChunks: number of fragments server will buffer before making them available to clients • manifestWindowLength: length of DVR moving window • archivePath: local path where archives will be written

37. Ingests and Origins Origin Server: Delayed Stream Origin Server: Delayed Stream Encoder Ingest Server Origin Server: Realtime Stream Origin Server: Realtime Stream 8 Mbps Origin Server: VOD Archive Origin Server: VOD Archive

38. DEMO • DEMO: • Advanced Live Smooth Streaming workflow

39. Network Design Considerations • Bandwidth, bandwidth, bandwidth Encoder egress bit rate = (All video and audio bit rates together) x (Number of ingest publishing points) • Leave enough headroom for bit rate spikes and bursts • Play it: leave 50%-100% headroom

40. Building Smooth Streaming Origins • Network • Make sure NICs can handle the cumulative ingress and egress bandwidth • CPU and memory • Follow typical server configuration guidelines • Disk • Live Smooth Streaming can require high read/write rates • 10,000 RPM disks are a good idea

41. Redundancy and Failover • Encoders • Active-active redundant encoder configuration not supported • Encoder hot swap is possible as long as encoder doesn’t signal end-of-stream to server • Configurable time-out periods can help survive temporary loss of network connectivity • Some encoders support pushing to multiple publishing points concurrently, while others do pub point roll over in case of failure

42. Redundancy and Failover • Servers • IIS is designed to discard duplicate fragments • This allows ingest and origin servers to be set in active-active configuration Ingest server: Primary Encoder Origin server Ingest server: Secondary

43. Redundancy and Failover • Origins • CDNs can dynamically change DNS to redirect traffic to secondary origin when primary fails Ingest server: Primary Origin server: Primary DNS Encoder Ingest server: Secondary Origin server: Secondary

44. Video Encoding

45. Smooth Streaming Encoders • Many vendors to choose from: • Anystream (Grab Networks) • Digital Rapids • Envivio • Inlet Technologies • Microsoft • Rhozet (Harmonic) • Telestream • VBrick • ViewCast • Winnov

46. H.264 or VC-1? • Customers should choose whichever codec best fits their encoding workflow and meets their quality requirements • However, keep in mind: • H.264 decoding is typically more CPU intensive than VC-1 decoding for the same resolution and frame rate • In Silverlight 3: H.264 decoding requires about 15-25% more CPU time than VC-1 decoding with similar content properties • Good rule of thumb (for now): • For HD video, use VC-1 to reach largest audience • For SD video and smaller, use H.264 if quality gains are noticeable

47. Encoding for Smooth Streaming • Use VC-1 Advanced Profile or H.264 Main/High Profile • 2 second key frame distance works very well • Must be closed GOP • Client heuristics work best with fixed length (non-adaptive) GOP • Set video buffer size to 2x-3x key frame distance • Use WMA Professional audio codec for best quality at low bitrates • Comparable to HE-AAC – excellent stereo quality even at 48 kbps • Client heuristics are currently tuned for CBR content, so if using VBR set the peak bit rate within 15% of average bit rate for best playback

48. Multi Bit Rate Encoding Revisited 00:27.24 03:47.16 05:05.04 07:33.10 2.5 Mbps 1.5 Mbps 750 kbps 350 kbps

49. Variable Resolution • Why do we vary encoded resolution together with bit rate? • Encoding the same resolution at inappropriately low bit rates introduces objectionable compression side effects into the video: blockiness, twirling details, color smearing, etc. • By lowering the resolution proportionally to the bit rate we maintain a consistent level of compression quality in exchange for giving up some visual detail • It’s a compromise: between two evils, customers prefer a blurry picture over a blocky picture

50. VC-1: 640x360 at 1250 kbps