1 / 23

Positional Audio and Window Highlighting for the Access Grid

Positional Audio and Window Highlighting for the Access Grid. Robert Putnam (putnam@bu.edu) Boston University Scientific Computing and Visualization. Agenda. The problem Prototype positional audio Prototype window highlighting Distribution/installation Discussion. Who's talking now?.

erwin
Download Presentation

Positional Audio and Window Highlighting for the Access Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Positional Audio and Window Highlighting for the Access Grid Robert Putnam (putnam@bu.edu) Boston University Scientific Computing and Visualization

  2. Agenda • The problem • Prototype positional audio • Prototype window highlighting • Distribution/installation • Discussion Boston University Scientific Computing and Visualization Group

  3. Who's talking now? • In some AG contexts, it is not always clear which video window contains the current speaker. The problem is exacerbated by: • Multiple, concurrent speakers • Users unfamiliar with the AG • Users unfamiliar with each other • Remedy: look for familiar face or moving lips. Boston University Scientific Computing and Visualization Group

  4. What’s that noise? • In some AG contexts, it is not always clear who is transmitting echo or noise. • Remedies: • Broadcast appeal • Operator mutes RAT streams one at a time until offender is found. Boston University Scientific Computing and Visualization Group

  5. Solution #1: Positional audio • Use location of site's video window(s) to determine its audio position in stereo field. Boston University Scientific Computing and Visualization Group

  6. Solution #1: Positional audio • Preliminary system design • Components • (Minimally modified) vic and RAT • “Agent” which communicates with vic and RAT • Sequence of events • Operator moves window • Agent gets report of window movement from vic • Agent sends audio panning command to RAT • Need: communications & audio localization layers. vic agent ? RAT ? ? Boston University Scientific Computing and Visualization Group

  7. Positional audio • RAT has built-in support for “3D audio”* • RAT’s localization methods: • Interaural Intensity Difference (IID) • Difference in sound intensity between near and far ear. • Interaural Time Difference (ITD) • Difference in time of arrival between near and far ear • Head Related Transfer Function (HRTF) • 3D-location-specific filtering effect of ears, head, and torso. • *Actually “1D,” since there’s only a single degree of freedom. Boston University Scientific Computing and Visualization Group

  8. Positional audio • Revised system design: • What about communications? Next slide… vic agent ? RAT 3D audio ? Boston University Scientific Computing and Visualization Group

  9. Message Bus • RAT’s control, user interface and engine already communicate via the Message Bus (“mbus*”): • So… rat-ui rat-engine mbus rat-control *see www.mbus.org Boston University Scientific Computing and Visualization Group

  10. Positional audio • Revised system design: • Next: what mbus message is used to enable 3D audio? vic agent RAT 3D audio mbus mbus Boston University Scientific Computing and Visualization Group

  11. 3D audio mbus message • 3D audio mode is enabled for a given RAT source via an audio.3d.user.settings message: audio.3d.user.settings rat-ui rat-engine tcl mbus Boston University Scientific Computing and Visualization Group *see www.mbus.org

  12. Talking to RAT • So, agent sends audio.3d.user.settings message to rat-engine: • ssrc = “Synchronization source” = RTCP stream id • azimuth = horizontal angular distance of listener from sound source. (90 = panned left; -90 = panned right; 0 = panned center) audio.3d.user.settings ssrc azimuth vic ? agent rat-engine mbus mbus Boston University Scientific Computing and Visualization Group

  13. Listening to vic • Next, create new mbus message • window.xy • Normalized xpos (0 – 1.0) • Normalized ypos (0 – 1.0) • Name of video window (e.g., “LANL(Video)”) • Whenever video window is moved, resized, or renamed, send window.xy message to agent. vic window.xy agent audio.3d.user.settings rat-engine mbus mbus Boston University Scientific Computing and Visualization Group

  14. Associating vic and RAT streams • Agent responsible for determining which video windows correspond to which RAT streams (ssrc’s): • As of AGtk 2.4, it is assumed that vic and RAT streams have the same name • I.e., RAT “LANL” corresponds to vic “LANL(Video)” vic window.xy agent rtp.source.name rat-engine “LANL(Video)” 04b6910a “LANL” Boston University Scientific Computing and Visualization Group

  15. Solution #2: Window highlighting • System shown at May 2003 AG retreat • Feature request: visual indication of current speaker • But, danger of distraction/fatigue (i.e. too much visual clutter or “flashing”). Boston University Scientific Computing and Visualization Group

  16. Window highlighting • Change border color of site’s video windows when its audio rises above a threshold. • Two new mbus messages: • RAT: rat.source.amplitude(source_id amplitude) • Report each stream’s peak amplitude 2-3X / second • vic: highlight.window(window_id on_off) window.xy audio.3d.user.settings vic agent rat-engine highlight.window rat.source.amplitude Boston University Scientific Computing and Visualization Group

  17. Window highlighting Boston University Scientific Computing and Visualization Group

  18. Window highlighting • Window stays highlighted during brief pauses in audio unless another site begins transmitting. • Each site’s local video windows also highlight when RAT transmitting (“On Air” alert). • Window border highlight color and width is configurable in AG service. Boston University Scientific Computing and Visualization Group

  19. Setting up the system • Install stereo speakers/amplifiers • Install stereo wiring to and from echo canceller (aka “the Gentner”): RAT pc line out l r Gentner amplifier l r l r l r Boston University Scientific Computing and Visualization Group

  20. Setting up the system • Create left+right echo cancellation reference: Boston University Scientific Computing and Visualization Group

  21. Setting up the system • Remove or disable original video consumer and audio services and add two new ones: • HighlightVideoConsumerService.zip • PositionalAudioService.zip Boston University Scientific Computing and Visualization Group

  22. Setting up the system • Configure HighlightVideoConsumerService: Venue Client Windows Desktop Boston University Scientific Computing and Visualization Group

  23. What next? • Ready for beta testers! Get it here: • http://scv.bu.edu/~putnam/ag-audio/services-2.4.zip • Zip file contains services for Windows / Linux, PIG / multi-computer configurations. • Request features / report bugs here: • putnam@bu.edu Boston University Scientific Computing and Visualization Group

More Related