Inside the Olympics An Architecture and Development Review

1. PC39 Inside the OlympicsAn Architecture and Development Review  Jason Suess Senior Technical Evangelist Microsoft  Eric Schmidt Director Microsoft

2. Player Demo

3. Agenda • Business imperative • What we built • Where do videos come from? • How we managed the ship… • Lessons learned Limelight & Level3 Live Event (Beijing) International Broadcast Center (Beijing) 30 Rock (New York) DMZs (New Jersey, Burbank) CDN Distribution Video Player Video Production Workflow

4. The Business Perspective • Deliver every minute of every sport live from Beijing to the web • Where possible deliver “HD” quality • Break the mold on user experience • Make it feel “solid-state” • Reach the broadest audience as possible • Leverage existing assets as much as possible • High dwell time • Quality impressions for their advertisers • Innovate

5. The Result • 1.3 billion page views • 50 million unique visitors • 70 million videos watched • 27 minutes of viewing per session • 600 million minutes of video delivery • 5,000 unique clips viewed per day during the final week • 35 million mobile views • 130,000 peak streams • 3.4 petabytes of video delivered

6. Video Production Workflow Limelight & Level3 Live Event (Beijing) International Broadcast Center (Beijing) 30 Rock (New York) DMZs (New Jersey, Burbank) CDN Distribution Video Player Video Production Workflow

7. The Workflow & Delivery Perspective • 2,200 events covering 25 sports with up to 35 simultaneous live feeds • 19 days of live production • 5,000+ videos to be produced • 2000 new clips • 3000 highlights/encores • Redundant, scalable content delivery

8. Content Packages • Live • 2,200 hours and up to 35 concurrent streams • Synchronized with live commentary and play-by-play • Rewind • VOD archive of live streams • Available within an hour of the completion of the event • Highlights (light touch and craft edit) • 3-5min highlights of individual events • Varying degrees of graphics and voice commentary • X hours • Encores • Broadcast replays • Cut down to individual sports and broadcast commercials removed • Full broadcast graphics and voice commentary • X hours

9. The Planning Complexity • Anticipate Content • When • Where • Size • Determined CPU, power, cooling, storage and ingress/egress needs • Production schedule drove advertising schedule

10. Encoding Profiles Encore/Highlights • Built for 1024x768 • Needed to meet IOC advertising requirements • Restricted to a 40~meg (DS3) line of out Beijing • We didn’t know how hard to really push… • http://alexzambelli.com/blog/2008/08/21/an-inside-look-at-nbc-olympics-video-player/ Ads Live/Rewind

11. Where does video come from?(live) • Multiple cameras at each event • Routed to International Broadcast Center • HD PAL Source • Linked to live encoder farm • 40 encoders – Digital Rapids • Linked 2 Windows Media Services boxes • Streams reflected back to New York • Pushed through DMZ • Streams then reflected to CDNs Live Event (Beijing) International Broadcast Center (Beijing) DMZs (New Jersey, Burbank) Limelight & Level3 30 Rock (New York)

12. Live & Rewind

13. Beyond live… • Rewind encoding on live • Highlights factory • Encore publishing • Scheduling • Ad booking SharePoint Server

14. Metadata was the key • Common Message Schema • Video Identification • Scheduling • Command/Control • Delivery

15. Highlights Factory

16. Publish • CMS

17. Content Packages • Live • 2,200 hours and up to 35 concurrent streams • Synchronized with live commentary and play-by-play • Rewind • VOD archive of live streams • Available within an hour of the completion of the event • Highlights (light touch and craft edit) • 3-5min highlights of individual events • Varying degrees of graphics and voice commentary • X hours • Encores • Broadcast replays • Cut down to individual sports and broadcast commercials removed • Full broadcast graphics and voice commentary • X hours

18. Video Players • Motivations for WMP player • Silverlight install base concerns • Concerns about corporatedeployment restrictions • Proven track record/WMS pedigree • Motivations for Silverlight • Next generation user experience • PIP • Overlays • Better navigation • etc • Adaptive streaming • Cross-platform/cross-browser

19. http://ad.doubleclick.net/adi/nbcu.olympics/sport_sw_general_sl;area=sport;section=general;video=highlights;videoid=249000;type=sl;player=popup;sport=sw;athlete=2;tile=2;sz=300x250;ord=964055555.7812571http://ad.doubleclick.net/adi/nbcu.olympics/sport_sw_general_sl;area=sport;section=general;video=highlights;videoid=249000;type=sl;player=popup;sport=sw;athlete=2;tile=2;sz=300x250;ord=964055555.7812571 Player Operation http://manifest.olympics.video.msn.com/d1/0822_HD_MUL_AU_CE523_MBR.elive http://chunks-54.olympics.video.msn.com/nbcu/chunks/0822_HD_MUL_AU_CE523_MBR/629f379d85d767625601deb2052776ec9f54d3db3220b1c3592b6b3cc1d5.vid index.xml Commentary File JSON File Each of these meta-data files are written to origin by the CMS when the asset is published

20. The Life of a Video • Load the player

21. JSON Schema

22. Content Protection • Geo-fencing • Geo based IP restriction • Tokenization • URL in JSON is tokenized with a hash based on time (ttl) and secret key "http://manifest.olympics.video.msn.com/d1/0824_HD_MUL_AU_CE529_MBR.elive?e=1225296772&p=42&h=21f9ee892f5093c624a0f95f846c7a7b&token=c3RhcnRfdGltZT0yMDA4MTAyOTE2MDI1MiZlbmRfdGltZT0yMDA4MTAyOTE2MTI1MiZkaWdlc3Q9MGQ1Y2M3NTFkYTE4YmY1OGM1MTEzOTQ0MTA2ZGNmZDM=

23. The Life of a Video • Load the player • Load the JSON • Edge side include calculates and inserts tokens • Player is handed video URL • Request “video URL” • Geo check • Token check • CDN responds with actual URL • Live -> WMS • Non-live -> Manifest file • Unzip, decrypt • Silverlight plays URL

24. CDN Distribution Limelight & Level3 Live Event (Beijing) International Broadcast Center (Beijing) 30 Rock (New York) DMZs (New Jersey, Burbank) CDN Distribution Video Player Video Production Workflow

25. Two CDNs Limelight • Provisioned identically • URL structure • CName attributes (token authentication, geo-restriction, etc.) • Content • Dynamic traffic routing with Internet Traffic Manager (ITM) • Granular control • Content type • Region • Straight percentages • Traffic split (active-active) • Limelight 95% (primary) • Level3 5% (warm backup) Level3’s ITM Video Requests http://manifest.olympics.video.msn.com/... Level3

26. CDN Provisioning Details • live.olympics.video.msn.com • Live WMS streams • Token authentication and geo-restriction • wmp.olympics.video.msn.com • On-demand WMS delivery • Token authentication and geo-restriction • intl-wmp.olympics.video.msn.com • On-demand WMS delivery • Token authentication but no geo-restriction • manifest.olympics.video.msn.com • HTTP delivery of adaptive streaming manifests • Token authentication and geo-restriction • intl-man.olympics.video.msn.com • HTTP delivery of adaptive streaming manifests • Token authentication but no geo-restriction • chunks-xx.olympics.video.msn.com • HTTP delivery of adaptive streaming “chunks” • No token authentication or geo-restriction

27. Video Player Development Limelight & Level3 Live Event (Beijing) International Broadcast Center (Beijing) 30 Rock (New York) DMZs (New Jersey, Burbank) CDN Distribution Video Player Video Production Workflow

28. Working back from 08-08-08 • 06/07 – site vision and scoping meetings • 08/07 – IBC and data center design • 11/07 – player vision and scoping meetings • 01/08 - user experience design • 02/08 - traffic forecasting • 03/08 - development hit full stride • 04/08 - functional testing and spin up of CDN workflow • 05/08 – Beta 1 • 06/08 – Beta 2 • 07/08 - final build out in Beijing & stabilization • 08/08 - go live

29. The Ultimate Team Effort • NBC (New York and Beijing) • Video workflow • Vendor coordination • Overall monitoring • Microsoft (Redmond) • CDN coordination • Adaptive streaming delivery • Silverlight delivery • Demand generation (MSN) • Monitoring • DeltaTre (Italy) • Content management system • Web site design and development • Client HTML and Javascript • Schematic (New York, Atlanta, Costa Rica) • Video player and user experience design • Implementation of Silverlight and WMP player • Ad logic • Limelight Networks (Lead) & Level3 (Back-Up) • WMS and HTTP capacity build out • Geo fencing & Tokenization • Digital Rapids & Anystream • Encoding/ Meta data • Workflow synchronization • Signiant • Content distribution to CDN’s and publishing points • Workflow synchronization • DoubleClick • In-Stream and companion ad delivery • Silverlight 2 integration • Omniture • Site and player metering • Analytics • Keynote • WMS and adaptive streaming monitoring • Sliq • Live stream and publishing point monitoring Pick the right partners and make the investment up front Constant scrum like communication to keep everyone moving the same direction

30. Player Design Interaction Model > Wireframes > Comps > Motion Tests

31. Player Architecture WMP Player Silverlight Player Silverlight XAP Logging Presentation LayerHTML, CSS, WMP Presentation LayerXAML, animation, UI logic Ad Call Preparation Dart.dll Elivelibrary.dll Ad Model Logic Ad Model Logic Bridge Player instantiation, translations layer Bridge Player instantiation, translations layer Business Abstraction LayerServer communication, user interaction HTML Host ComponentServer communication, user interaction HTML Javascript/HTML C#/XAML

32. Monitoring & Game Time Operations

33. Monitoring • NBC implemented an all-up view for all critical systems (storage, CPU health, power management, etc.) • But that was only part of the story… • Delivery and CDN health • Concurrent streams • 404 errors • Outbound traffic • Cache efficiency • … It was really hard to monitor the health of chunked delivery and understand what was happening at the client level, so….

34. Inline Monitoring

35. Real-time Telemetry • Less than 5 min. delay • Aggregate view, with drill down to source log

36. Drill-down

37. Targetting Core Issue

38. Physical Architecture – ORCSWeb.com • Silverlight 2.0 Clients • 60% IE, 40% other • IP Load Balancer (Redundant Pair) • 5.4 million hits during peak day • 40 million unique users • Web Tier (15 Web Servers) • WS2K8, IIS 7.0, ASP.NET, WCF • SQL2K8 SSB Initiator Queues • Data Tier (Write Only) • WS2K8, SQL2K8 • SQL2K8 SSB Target Queue, RawLog Table • Data Warehouse (Read Only) • WS2K8, SQL2K8, SSIS, SSRS, SSAS • RawLog Dimensional Model

39. 24 Hour "War Rooms" • New York • Beijing • Phoenix • Redmond • Torino • All connected by NBC’s virtual meeting space/call bridge system • Real-time system to tell us when we had media playback problems (many different potential causes) • Use data to diagnose • Multiple people with different domain expertise to fix

40. Lessons Learned • Scrumming builds better teams, meet face to face • Everyone should know all roles and all architectural touch points • Reduce complexity via common schema • Even when traffic volume decreased, the number of unique video’s viewed grew, providing the right UI is essential to make the long-tail work • Long-tail delivery hides issues • The industry needs better telemetry and monitoring solutions • “Chunked” workflow presented new challenges • We left some on the table in terms of bandwidth and video profile estimates • IIS Smooth Streaming will help content publishers more efficiently tweak their workflows

41. Why were we successful? • Windows Media pedigree • Strong partners • The quality of MSN’s audience • Silverlight was ready for primetime • The competency, passion and resources to get it done

42. Q&A

43. Q&A Please use the microphones provided

44. Evals & Recordings Please fill out your evaluation for this session at: This session will be available as a recording at: www.microsoftpdc.com

45. © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.