Position Calibration of Audio Sensors and Actuators in a Distributed Computing Platform Vikas C. Raykar | Igor Kozintse

1. PositionCalibration of Audio Sensors and Actuators • in a Distributed ComputingPlatform • Vikas C. Raykar | Igor Kozintsev | Rainer Lienhart • University of Maryland, CollegePark | Intel Labs, Intel Corporation

2. Motivation • Many multimedia applications are emerging which use multiple audio/video sensors and actuators. Speakers Microphones Distributed Capture Current Work DistributedRendering Cameras Number Crunching Displays Other Applications

3. What can you do with multiple microphones… • Speaker localization and tracking. • Beamforming or Spatial filtering. X

4. Some Applications… Speech Recognition Hands free voice communication Novel Interactive audio Visual Interfaces Multichannel speech Enhancement Smart Conference Rooms Audio/Image Based Rendering Audio/Video Surveillance Speaker Localization and tracking MultiChannel echo Cancellation Source separation and Dereverberation Meeting Recording

5. More Motivation… • Current work has focused on setting up all the sensors and actuators on a single dedicated computing platform. • Dedicated infrastructure required in terms of the sensors, multi-channel interface cards and computing power. On the other hand • Computing devices such as laptops, PDAs, tablets, cellular phones,and camcorders have become pervasive. • Audio/video sensors on different laptops can be used to form a distributed network of sensors. Internal microphone

8. Common TIME and SPACE • Put all the distributed audio/visual input/output capabilities of all the laptops into a common TIME and SPACE. • For the common TIME see our poster. Universal Synchronization Scheme for Distributed Audio-Video Capture on Heterogenous Computing Platforms R. Lienhart, I. Kozintsev and S. Wehr • In this paper we deal with common SPACE i.e estimate the 3D positions of the sensors and actuators. Why common SPACE • Most array processing algorithms require that precise positions of microphones be known. • Painful and Imprecise to do a manual measurement.

9. This paper is about.. Z Y X

10. If we know the positions of speakers…. Y If distances are not exact If we have more speakers Solve in the least square sense ? X

11. If positions of speakers unknown… • Consider M Microphones and S speakers. • What can we measure? Distance between each speaker and all microphones. Or Time Of Flight (TOF) MxS TOF matrix Assume TOF corrupted by Gaussian noise. Can derive the ML estimate. Calibration signal

12. Nonlinear Least Squares.. More formally can derive the ML estimate using a Gaussian Noise model Find the coordinates which minimizes this

13. If noise is Gaussian and independent ML is same as Least squares Maximum Likelihood (ML) Estimate.. we can define a noise model and derive the ML estimate i.e. maximize the likelihood ratio Gaussian noise

14. Reference Coordinate system | Multiple Global minima Reference Coordinate System Positive Y axis Similarly in 3D 1.Fix origin (0,0,0) 2.Fix X axis (x1,0,0) 3.Fix Y axis (x2,y2,0) 4.Fix positive Z axis x1,x2,y2>0 Origin X axis Which to choose? Later…

15. On a synchronized platform all is well..

16. However On a Distributed system..

17. Multimedia/multistream applications Operating system I/O bus Audio/video I/O devices The journey of an audio sample.. Network This laptop wants to play a calibration signal on the other laptop. Play comand in software. When will the sound be actually played out from The loudspeaker.

18. On a Distributed system.. Time Origin Signal Emitted by source j t Playback Started Signal Received by microphone i Capture Started t

19. MS TOF Measurements Joint Estimation.. Microphone and speaker Coordinates 3(M+S)-6 Microphone Capture Start Times M -1 Assume tm_1=0 Totally 4M+4S-7 parameters to estimates MS observations Can reduce the number of parameters Speaker Emission Start Times S

20. Nonlinear least squares.. Levenberg Marquadrat method Function of a large number of parameters Unless we have a good initial guess may not converge to the minima. Approximate initial guess required.

21. Closed form Solution.. Say if we are given all pairwise distances between N points can we get the coordinates.

22. Classical Metric Multi Dimensional Scaling dot product matrix Symmetric positive definite rank 3 Given B can you get X ?....Singular Value Decomposition Same as Principal component Analysis But we can measure Only the pairwise distance matrix

23. How to get dot product from the pairwise distance matrix… k i j

24. Example of MDS…

25. Can we use MDS..Two problems 1. We do not have the complete pairwise distances 2. Measured distances Include the effect of lack of synchronization UNKNOWN UNKNOWN

26. Clustering approximation…

27. i j j i j j Clustering approximation… i i

28. Finally the complete algorithm… Approximation TOF matrix Clustering Approx ts Approx Distance matrix between GPCs Dot product matrix Approx tm Dimension and coordinate system MDS to get approx GPC locations TDOA based Nonlinear minimization perturb Microphone and speaker locations tm Approx. microphone and speaker locations

29. Sample result in 2D…

30. Algorithm Performance… • The performance of our algorithm depends on • Noise Variance in the estimated distances. • Number of microphones and speakers. • Microphone and speaker geometry • One way to study the dependence is to do a lot of monte carlo simulations. • Or given a noise model can derive bounds on how worst can our algortihm perform. • The Cramer Rao bound.

31. Gives the lower bound on the variance of any unbiased estimator. • Does not depends on the estimator. Just the data and the noise model. • Basically tells us to what extent the noise limits our performance i.e. you cannot get a variance lesser than the CR bound. Rank Deficit..remove the Known parameters Jacobian

32. Number of sensors matter…

33. Number of sensors matter…

34. Geometry also matters…

35. Geometry also matters…

36. X Room Height = 2.03 m Speaker 3 Mic 3 Mic 4 Room Length = 4.22 m Speaker 2 Speaker 4 Mic 2 Mic 1 Speaker 1 Z Room Width = 2.55 m Synchronized setup | bias 0.08 cm sigma 3.8 cm

37. Experimental results using real data

38. Summary • General purpose computers can be used for distributed array processing • It is possible to define common time and space for a network of distributed sensors and actuators. • For more information please see our two papers or contact igor.v.kozintsev@intel.com • rainer.lienhart@intel.com • Let us know if you will be interested in testing/using out time and space synchronization software for developing distributed algorithms on GPCs (available in January 2004)

39. Thank You ! | Questions ?