Progressive Perceptual Audio Rendering of Complex Scenes

1. Progressive Perceptual Audio Rendering of Complex Scenes Thomas Moeck - Nicolas Bonneel - Nicolas Tsingos - George Drettakis - Isabelle Viaud-Delmon - David Alloza 1 4 1,2 1 3 1 1- REVES/INRIA Sophia-Antipolis 2- Computer Graphics Group, University of Erlangen-Nuremberg 3- CNRS-UPMC UMR 7593 4- EdenGames

2. Objectives • Efficient audio rendering of very complex scenes with moving sources • Without audible impairment of the quality • Verify results by user tests

3. Previous Work • Rendering complex auditory scenes • Clustering [Tsingos et al. 2004]: replace many sources with a representative • Still can only treat ~200 sound sources (cost of clustering itself) • Scalable audio processing • Importance-guided processing of few frequency/time bins [Fouad et al. 1997, Wand & Straßer 2004, Gallo et al. 2005, Tsingos 2005]. • Audio processing (e.g., HRTF, spatialization) is expensive • Crossmodal effects • Neuroscience Literature: “Ventriloquism affects 3D audio perception” • Ventriloquism spatial window can vary from a few up to 15 degree • Few papers on ecological experiments

4. Methodology • Recursive approach to clustering • Reduce cost of clustering • Scalable perceptual premixing • Faster premixing without audible loss of quality • Taking perceptual and cross-modal information into account • Improve audio clustering algorithm • User experiments to detect improvement possibilities • Improving quality with results of tests • Validation of resulting algorithms

5. Overview of the algorithms • Masking of inaudible sources (with energy) • Clustering of remaining sources • Progressive premixing within each cluster • Spatial audio processing (HRTF) recursive

6. Our Work • Optimized recursive approach of clustering • Clustering performance evaluation • Improved scalable perceptual premixing • Quality evaluation study • Study of cross-modal effects by user experiments • Using results of cross-modal studies to develop audio-visual clustering algorithm

7. Optimized Recursive Clustering • Recursive splitting of clusters • Fixed-budget approach • Using a fixed number of clusters • Variable-budget approach • Splitting clusters until break condition is reached • Break condition: Average angle error • Optimal number of clusters • Variant used by EdenGames • 8 cluster budget • Local clustering when necessary

8. Eden Games’ implementation Test Drive Unlimited

9. Clustering Performance Evaluation • Performance of recursive algorithms are clearly better

10. Improved progressive scalable perceptual premixing (1) • After clustering: Premixing in each cluster • Why? Effects can be done afterwards - less cost because viewer signals • Only premixing necessary data • Assigning frequency bins to sound sources (iterative importance sampling) by using pinnacle value

11. Improved progressive scalable perceptual premixing (2) premixing clustering

12. Improved progressive scalable perceptual premixing (3) • Iterative importance sampling • Calculation of importance value from energy, loudness or audio saliency map • Assignment of frequency proportional to importance • until pinnacle value is reached • Reassignment of remaining frequencies to sounds relative to importance values

13. Varying budget

14. Quality Evaluation Study (1) • MUSHRA (“Multiple Stimuli with Hidden Reference and Anchors”) test of perceptual premixing • 7 subjects, aged from 23 – 40 • Ambient, music and speech • Various budgets (2% – 25 %) • With and without pinnacle value • Using loudness or saliency as importance value

15. Quality Evaluation Study (2) • Results: • Approach is capable of generating high quality using 25% of the original data • Acceptable results with 10% (2% in case of speech) • Significant Effects: • Budget • Importance value • Pinnacle value

16. Study of Cross-Modal Influences – Questions • Do we need more or fewer clusters in the viewing frustum? • We move spatial position of sound sources to representative in cluster • How tolerant are we to this error ? • Do visuals influence the perceived quality?

17. Study of Cross-Modal Influences – Setup (1)

18. Study of Cross-Modal Influences – Setup (2)

19. Study of Cross-Modal Effects – Setup (3)

20. Uniform distribution [1/4]

21. [2/3] condition

22. [3/2] condition

23. [4/1] condition

24. Study of Cross-Modal Influences – Results • Statistical analysis of the results shows: • We need more clusters in the viewing frustum • No significant difference of visuals/no-visuals but possible cross-modal effect

25. Modifying the algorithm • Introducing weighting term in clustering:  Increasing number of clusters in the viewing frustum

26. Cross-Modal illustration

27. Video: Putting it all together

28. Conclusions • Up to nearly 3000 sound sources possible in good quality • Main limitation are graphics (!) • Better quality because more clusters in viewing frustum • Future work • experiment with auditory saliency measurements • handle procedurally synthesized sounds?

29. Questions?