1 / 44

LOLA ( LOw LAtency Audio Visual Streaming System)

LOLA ( LOw LAtency Audio Visual Streaming System). A LOW LATENCY, HIGH QUALITY AUDIO/VIDEO TRANSMISSION SYSTEM FOR NETWORK MUSICAL PERFORMANCES AND INTERACTION Conservatorio G. Tartini Consortium GARR. The Dream…. ...at the GARR Conference 2005, in Pisa...

zanna
Download Presentation

LOLA ( LOw LAtency Audio Visual Streaming System)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. LOLA(LOw LAtency Audio Visual Streaming System) A LOW LATENCY, HIGH QUALITY AUDIO/VIDEO TRANSMISSION SYSTEM FOR NETWORK MUSICAL PERFORMANCES AND INTERACTION Conservatorio G. Tartini Consortium GARR

  2. The Dream… ...at the GARR Conference 2005, in Pisa... To perform together, from distant locations... as if distance has vanished in a click of a computer mouse.

  3. The LOLA Team Production Conservatorio di musica G. Tartini - Trieste Implementation Paolo Pachini: general coordination Carlo Drioli: programming Nicola Buso: testing and musical advice Claudio Allocchio (Consortium GARR): testing and networking advice Massimo Parovel: conception and supervision

  4. A/V before LOLA • H.323/SIP • High Video Compression • Low Quality Audio Codecs • Video over Audio Priority • Significant roundtrip latency • Optimised for Low Bandwidth • Proprietary Polycom “Music Mode”

  5. A/V before LOLA • DVTS • No Video Compression • DV Audio Codecs • Audio/Video in DV frames • High roundtrip latency due to IEEE1394 • High Bandwidth (30Mbps) • Good Quality • OK for music lessons

  6. A/V before LOLA • ConferenceXP • Indipendent Audio and Video Handling (compression, codecs) • Low to High Quality Audio/Video Codecs, HD 1080p • Multiple Audio/Video streams • High roundtrip latency • Variable Bandiwidth (2Mbps to ... inifinity)

  7. To play together… • Audio Latency below ~75ms (depends on music gender) • Eyes Contact to synchronize • Spatial Sound immersion for expression • Room Reverbering • Continuous Presence of all musicians • Non distracting environment

  8. What do we need? • A very fast CAPTURING and RENDERING device (audio & video) • A very fast ENCODING and DECODING (audio & video) • A network service which is • Very FAST 1 Gbps and above • Very STABLE < 3 ms packets Jitter • ERROR FREE < 0.3 % packets loss • no commodity Internet can deliver these features •  it’s a big challenge ALSO for National Research and Education Networks

  9. LOLA design concept • LOLA has been conceived and developed for dedicated end-to-end connections (circuits, optical lambdas) because there is no protection against network problems in it and in full mode it requires up to 500M for Standard Resolution (color). • LOLA has been also adapted to run in minimal configuration over very reliable NRENs… in the middle of the highway traffic at 92M (Standard Resolution, 30fps, B/W)

  10. Events: November 2010 • NPAPWS 2010: • Paris Ircam: • two pianos (Teresa Trevisan, Flavio Zaccaria)… • Bach Brandeburg Concerts, Reger Suites for 2 pianos • Trieste-Paris: 1200km

  11. Events: June 2011 • NPAPWS 2011: Barcelona, Grand Teatre del Liceu: two violins (Laura Agostinelli, Sebastiano Frattini)… • Trieste-Barcelona (via Madrid): more than 2700km

  12. Events: June 2011 • NPAPWS 2011: Barcelona, Grand Teatre del Liceu: two violins (Laura Agostinelli, Sebastiano Frattini)… • Bartok: suites for 2 violins • Trieste-Barcelona (via Madrid): more than 2700km

  13. Events: October 2011 • Internet2 Fall Meeting: NIU School of Music, DeKalb IL – Raleigh NC

  14. The Dream… • Let’s see and here it! • G.F. Handel/J. Halvorsen: • Passacaglia for violin and cello • Marjorie Bagley, violin • School of Music, Theatre and Dance • University of North Carolina – Greensboro, NC • Cheng-Hou Lee, cello • School of Music • Northern Illinois University – DeKalb, IL

  15. Motivations/Goals • Target applications: • interactive musical tasks • Network musical performances • Network music education and training • ... but also... • interactive on stage performances • Network Dance performances • Network Drama performances • Network performing arts education and training • ...

  16. Motivations/Goals • ... and beyond: • remote real time surgery • virtual space immersion • ... you name it... • Let the user invent new possibilities...

  17. Challenges • Management of synchronous audio video streams over packet networks • Minimal delay requirements for interactive task and music performances • Optimal balance between AV presentation delay and quality • The speed of computer/networking equipments • ... and beyond: the speed of light...

  18. Targets • A system suitable for musical performances relying on both audio and visual communication (but also other applications) • A Transparent and Natural end-user interaction • Providing lowest possible delay using available technology • Low cost and portable equipment • Oriented to dedicated high performance networks (LightNet Project, GARR, GÈANT, Internet2,...)

  19. The LOLA System a very basic schema (well, what’s the difference with DVTS, CXP?) Research & Education Networks

  20. LOLA internals Connection negotiation and management • Send/receive and manage connection requests • Negotiation of audio/video formats and compatibility check • Video: 640*480, 30-60 fps, BW/Color Audio: 44100, 16/24 bit, mono/stereo, multichannel... • Transmission of raw audio and video streams, monitoring

  21. LOLA internals Audio/Video data grabbing and transmission • Grabbing and Tx with very low latency devices • Video: Grabber BitFlow, BW progressive scan camera • Audio: RME HDSP9632 Network: 1GBps Ethernet Card / 1 GBps Network Connection (Circuit?) • CODEC latency: < 5 ms (audio and video) • Network latency: <1 ms (LAN); ~1 ms per 100Km (WAN); stay below 60ms RTT (~4000Km) ? • a few jitter buffers are required (delay vs quality balance)

  22. LOLA internals Audio/Video data grabbing and transmission: the different solutions explored • 1) DirectShow based A/V acquisition and presentation, networking, A/V processing • “easy” solution but inefficient, due to lack of control over buffering solutions and other low-level parameters

  23. LOLA internals Audio/Video data grabbing and transmission: the different solutions explored • 2) C++ programming of A/V processing, through hardware specific video libraries, ASIO audiolibraries, and Winsocket networking libraries. • better solution with good efficiency, still some delays in network send/recv (needs accurate multithreading programming, Winsocket is a black box)

  24. LOLA internals Audio/Video data grabbing and transmission: the different solutions explored • 3) as in 2), but with use of high performance networking libraries (WinPCap) • lowest round-trip delay, and full control of threads processing

  25. B/W vs Color issues Audio/Video data grabbing and transmission: color video • Color video with low latency industrial grabbers can be obtained by using standard consumer-class cameras with analog RGB output or industrial digital cameras supporting the “Camera Link” standard. • The preferred choice for Lola was to adopt a digital solution by mounting a Camera Link grabber (Neon BitFlow) and color camera (Sentech C33A). Advantages over the analog color video grabbing solution include superior signal quality, control of the camera parameters through the software application, and cost effectiveness. • Switching to uncompressed color video implies higher bandwidth requirements (bits per pixel increase from 8 to 24/32) and higher computational load if software bayer decoding is required, as it is often the case for industrial color cameras. Experimentation with color video stream is ongoing work. • At 30fps, B/W is 96Mbps… Color is 300Mbps minimum… and UP!

  26. LOLA GUI • Controls for connection negotiation and management • Audio/Video buffering controls • Visualization options, audio mixer, etc • Recording

  27. The Dream…

  28. First tests: 110km June 4th 2009 - Trieste fiber optics metropolitan network “Lightnet”

  29. Yes, we can play! First test with music: September 21st 2009 • Two Pianos, in two studio rooms at Tartini, linked over the loop with LOLA • Round Trip Latency ~90ms • monitor on the music score, “as if the other pianist was in his canonical ‘duo position’, e.g. in front” • Tests performed: • One piano plays alone, with the return audio channel open; sound was coming back, but no echo cancelling needed; • Two pianos play together some scales and easy execises; • Two pianos play together some canone by J.S. Bach; • The latency is artificially increased to test interaction limits; • We can try with a full setup and a real Piano Duo to get feedback

  30. Bach Brandeburg Concerts Trevisan-Zaccaria Piano Duo: November 5th 2009 • Two Pianos, in two Concert Hall at Tartini, linked over the loop with LOLA • Round Trip Latency ~80ms, mostly due to CODECs • Sound Rendering, Room Environment, musician interaction with LOLA environment. • Tests performed: • Play Bach Branderburgh Concerts • Roundtrip Latency tests • Remote sound in (insulating) earphone vs audio monitors • Adaptation techniques to delay • They can play together, but too much attention is payed to handle the delay. No confortable environment for artistic performance. • We need to go further down with CODECs delays

  31. The Triangle Test Experiments without WinSock: December 21st 2009 • Just a Triangle, to make accurate latency measurments • New Round Trip Latency ~20ms ! • Tests performed: • Various buffers configuration to identify optimal results • Sound Quality check (sending recorded audio tracks) • We can give Bach another chance to meet technology!

  32. Bach Brandeburg Concerts (bis) Trevisan-Zaccaria Piano Duo: February 4th 2010 • Two Pianos, in two Concert Halls at Tartini, linked over the metropolitan loop with LOLA • Round Trip Latency 20ms (artifically up to 70ms) • Tests performed: • Play Bach Branderburgh Concerts • They started to play... and played them all! • Natural interaction • Many tests to refine the Sound Rendering of the remote piano.

  33. Let’s go Real! Trevisan-Zaccaria Piano Duo: October 16th 2010 Trieste-Milan-Trieste loop; ~40ms • Sound equalisation and spatialisation tests for the “remote” piano; • it seems to run safely in the middle of the highway:

  34. Beware of Traffic Jams NPAPWS 2010: Too many CERN LHC data!

  35. …and Switches… NPAPWS 2011: Too many PPS!

  36. …and Caterpillars! • DANTE TICKET: 15368 • Type: Dashboard Alarm • Status: Open • Description: [mad - gen] GEN-MAD-IP (interface so-7/2/0) is down • Location A: Madrid, ES • Location B: Geneva, CH • Incident Start: 14/06/2011 10:30:34 (UTC) • Incident Resolved: (UTC) • Ticket Open: 14/06/2011 10:37:29 (UTC) • Ticket Close: (UTC) • Translation: the fiber was CUT! • The re-routing went via Geneva-Paris-Madrid • Another 1000km  3700 !!! • … it was fixed just in time (4 hours before)

  37. LOLA v1.x Summer 2011: v1.0, v1.1, v1.1.1/1.1.2 • s/w code cleanup from old debugging functions • a new control channel set of functions: • internal remote latency evaluation; • “remote ready” function; • a new simpler user interface • easy install/remove procedures • enable user/password, license procedure • new “beta-level” functions: • 60fps • self-bounce-back function

  38. Work in Progress • color impact (on h/w and network bandwidth); • QoS implementation through DiffServ; • further optimize graphic card rendering; • improving video latency further! • spatial audio rendering (multichannels); • Linux porting

  39. The Speed Race We won the CODECs challenge ... the (academic) Network Challenge now! • 95 to 450 Mbps (GigaEthernet circuits) • Dedicated Ligthpaths • jitter < 3ms • ~1ms per 100Km • when the Atlantic will be large 30ms... we will call Captain Kirk and Scotty: • UltraLight Speed, Scotty!

  40. Yes Sir! Done!

  41. LOLA: How do I get it? Freely available for non-commercial academic use! • you build your own hardware following the specs: • 1 PC with PCI bus, min CPU duoCore 2.0Ghz • 1 RME HDSP audio card; • 1 BitFlow Grabber video card; • 1 fast industrial video process control camera • (you build your own camera cables) • you sign the free non-commercial academic use license (soon on-line, now via e-mail) and get the s/w • … and you’re a member of the LOLA community to start experimenting with it!

  42. LOLA: the community Building/testing together to improve it and gain experience! • your feedback is fundamental to further development; • more different h/w configurations • a lot to study about audio and video rendering • a collaboration in progress (with CESNET) for “LOLA in a Box” appliance; • multi-point experiments • what will happen with multicast LOLA? • give ideas… and let’s try to make that happen!

  43. A big thank you to… What we did today could not have been done without the LOLA community help: • Paul Bauer, Director, NIU School of Music • Dan Nichols, audio/video engineer, NIU School of Music • Dan Pritts, Sr. System engineer, Internet2 • Jeff Hagley, System engineer, Internet2 • Justin Trieger, Internet2 System Manager, New World Symphony • Marjorie Bagley & Cheng-Hou Lee, our duo • Ann Doyle, Global & Cultural collaborations, Internet2

  44. Questions? http://www.conservatorio.trieste.it Paolo.Pachini@conts.it Nicola.Buso@conts.it http://www.garr.it Claudio.Allocchio@garr.it

More Related