1 / 76

Cross-Modal (Visual-Auditory) Denoising

1. Cross-Modal (Visual-Auditory) Denoising. Dana Segev Yoav Y. Schechner Michael Elad. Technion – Israel Institute of Technology. Motivation. Noisy digits sequence. Digits sequence. Denoised by state of the art algorithm of Cohen & Berdugo. Segev, Schechner, Elad, Cross-Modal Denoising.

marsden-harvey
Download Presentation

Cross-Modal (Visual-Auditory) Denoising

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 1 Cross-Modal (Visual-Auditory) Denoising Dana Segev Yoav Y. Schechner Michael Elad Technion – Israel Institute of Technology

  2. Motivation Noisy digits sequence Digits sequence Denoised by state of the art algorithm of Cohen & Berdugo Segev, Schechner, Elad, Cross-Modal Denoising

  3. Motivation • Use one modality to denoise another? • Use video to denoise • a soundtrack? Segev, Schechner, Elad, Cross-Modal Denoising

  4. a Noise • Very intense • Non-stationary • Unknown • Unseen source. Single microphone Segev, Schechner, Elad, Cross-Modal Denoising

  5. denoised audio Cross-modal Example-Based very noisy audio Input time (sec) video Algorithm Output For human and machine hearing Segev, Schechner, Elad, Cross-Modal Denoising

  6. Intuition Segev, Schechner, Elad, Cross-Modal Denoising

  7. Intuition Segev, Schechner, Elad, Cross-Modal Denoising

  8. Intuition I E Training xample set nput test set Segev, Schechner, Elad, Cross-Modal Denoising

  9. Speech Examples Extraction Segev, Schechner, Elad, Cross-Modal Denoising

  10. Speech Examples Extraction ~syllable (0.25 sec) Segev, Schechner, Elad, Cross-Modal Denoising

  11. Music Segments Extraction lophone Xylophone Segev, Schechner, Elad, Cross-Modal Denoising

  12. Music Segments Extraction lophone Xylophone Sound Segev, Schechner, Elad, Cross-Modal Denoising

  13. Principle ... ... Examples Segev, Schechner, Elad, Cross-Modal Denoising

  14. Principle ... ... Examples Segev, Schechner, Elad, Cross-Modal Denoising

  15. Audio Only ... ... Examples Segev, Schechner, Elad, Cross-Modal Denoising

  16. Audio Only ... ... Examples Segev, Schechner, Elad, Cross-Modal Denoising

  17. Cross-Modal Denoising • Cross-modal representation. • Generating multimodal features. • Learning feature statistics. • Cross-modal pattern recognition. • Rendering a denoised signal. Segev, Schechner, Elad, Cross-Modal Denoising

  18. Feature-space Creation time (sec) Input video Video feature-space Input audio Audio feature-space Segev, Schechner, Elad, Cross-Modal Denoising

  19. Feature-space Creation time (sec) Audio-video feature-space Input audio-video Segev, Schechner, Elad, Cross-Modal Denoising

  20. Feature-space Creation Audio-video examples feature-space Training audio-video time (sec) Segev, Schechner, Elad, Cross-Modal Denoising

  21. Distance-measure Feature-space Segev, Schechner, Elad, Cross-Modal Denoising

  22. Distance-measure Feature-space Segev, Schechner, Elad, Cross-Modal Denoising

  23. Distance-measure Feature-space Segev, Schechner, Elad, Cross-Modal Denoising

  24. Distance-measure Nearest Neighbor Feature-space Segev, Schechner, Elad, Cross-Modal Denoising

  25. Distance-measure Nearest Neighbor Feature-space Segev, Schechner, Elad, Cross-Modal Denoising

  26. Distance-measure ... ... Examples Segev, Schechner, Elad, Cross-Modal Denoising

  27. Distance-measure ... ... Examples Segev, Schechner, Elad, Cross-Modal Denoising

  28. Rendering a denoised signal Noisy audio Clean segment Clean segment Clean segment Segev, Schechner, Elad, Cross-Modal Denoising

  29. Rendering a denoised signal Noisy audio Clean segment Clean segment Clean segment Denoised Segev, Schechner, Elad, Cross-Modal Denoising

  30. Distance-measure ... ... Examples Segev, Schechner, Elad, Cross-Modal Denoising

  31. Cross-Modal Association Examples ... ... ... ... Input Segev, Schechner, Elad, Cross-Modal Denoising

  32. Cross-Modal Association Examples ... ... ... ... Input Segev, Schechner, Elad, Cross-Modal Denoising

  33. Cross-Modal Association Examples ... ... ... ... ... ... ... ... ... ... Input Segev, Schechner, Elad, Cross-Modal Denoising

  34. Cross-Modal Association Examples ... ... ... ... ... ... ... ... ... ... Input Segev, Schechner, Elad, Cross-Modal Denoising

  35. Bartender experiment Segev, Schechner, Elad, Cross-Modal Denoising

  36. Cross-Modal Association Examples ... ... ... ... ... ... ... ... ... ... Input Segev, Schechner, Elad, Cross-Modal Denoising

  37. Cross-Modal Denoising • Cross-modal representation. • Generating multimodal features. • Learning feature statistics. • Cross-modal pattern recognition (NN). • Rendering a denoised signal. Segev, Schechner, Elad, Cross-Modal Denoising

  38. Feature Statistics as a Prior Feature-space Segev, Schechner, Elad, Cross-Modal Denoising

  39. Feature Statistics as a Prior Feature-space For the k-th example segment: Segev, Schechner, Elad, Cross-Modal Denoising

  40. Feature Statistics as a Prior bi - fif - ty- two Feature-space For the k-th example segment: bi ty ar fif two Segev, Schechner, Elad, Cross-Modal Denoising

  41. Feature Statistics as a Prior Next cluster bi ty fif two ar 1 bi 1 1 1 ty 1 fif 1 Feature-space 1 2 1 two bi 1 ar Current cluster ty ar fif two Segev, Schechner, Elad, Cross-Modal Denoising

  42. Feature Statistics as a Prior Syllable consecutive probability Next cluster bi ty fif two ar 53 23 bi 26 5 1 12 60 43 17 6 ty 22 4 1 fif 5 3 6 2 13 12 21 two 9 7 2 7 11 ar = Current cluster Number of examples in training set The probability for transition between clusters Segev, Schechner, Elad, Cross-Modal Denoising

  43. Feature Statistics as a Prior Hidden Markov Model fif fif Time delay two two bi ty ty bi P Segev, Schechner, Elad, Cross-Modal Denoising

  44. Feature Statistics as a Prior Audio noise fif fif Time delay two two bi ty ty bi P Segev, Schechner, Elad, Cross-Modal Denoising

  45. Feature Statistics as a Prior Hidden Markov Model Audio noise fif fif + Time delay two two bi ty ty bi P Segev, Schechner, Elad, Cross-Modal Denoising

  46. Cross-Modal Association Examples ... ... ... ... Input Segev, Schechner, Elad, Cross-Modal Denoising

  47. Cross-Modal Association Examples ... ... ... ... ... ... ... ... ... ... Input Segev, Schechner, Elad, Cross-Modal Denoising

  48. Cross-Modal Association Examples ... ... ... ... ... ... ... ... ... ... Input Segev, Schechner, Elad, Cross-Modal Denoising

  49. Cross-Modal Association Input video Segev, Schechner, Elad, Cross-Modal Denoising

  50. Cross-Modal Association Input video Segev, Schechner, Elad, Cross-Modal Denoising

More Related