1 / 22

NlogN Entropy Optimization

NlogN Entropy Optimization. 1. Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky. Sponsors: ISF, Dvorah Foundation. 49. Kernel Estimators: Parzen Windows. Estimated PDF. Data. True PDF. Shwartz, Schechner & Zibulevsky, NlogN entropy optimization. Previous Work.

saad
Download Presentation

NlogN Entropy Optimization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NlogN Entropy Optimization 1 Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation

  2. 49 Kernel Estimators: Parzen Windows Estimated PDF Data True PDF Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  3. Previous Work Parametric PDF:Hyvärinen 98, Bell; Sejnowski 95, Pham; Garrat 97. Cumulants:Cardoso ; Souloumiac 93. Not accurate

  4. Order statistics:Vasicek 76, Learned-Miller; Fisher 03. KD trees:Gray; Moore 03. Previous Work Not differentiable

  5. Entropy Estimation Kernel Estimators: reduced complexityPham, 03, .Erdogmus; Principe; Hild, 03, Morejon; Principe 04,Schraudolph 04, (Stochastic gradient).

  6. Source Range: Continuous

  7. 50 Parzen Windows Estimator Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  8. 51 Minimization of Mutual Information Differentiable Computationally efficient - Currently O(K N) online code (see website) 2 2 Independent Component Analysis Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  9. Parzen Windows as a Convolution 52 Wish it was … Discrete convolution Convolution Sampling Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  10. 53 Efficient Kernel Estimator Samples of estimated sources PDF estimation Fan; Marron 94, Silverman 82. A Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  11. 53 Samples of estimated sources Interpolation to uniform grid (histogram) PDF estimation Fan; Marron 94, Silverman 82. Efficient Kernel Estimator A B Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  12. 53 Samples of estimated sources Interpolation to uniform grid(histogram) Discrete convolution with Parzen window C PDF estimation Fan; Marron 94, Silverman 82. Efficient Kernel Estimator A B Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  13. 54 Efficient Entropy Estimator C D A • Interpolation to original values Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  14. 55 Can it be Used for Optimization? • Iterations exploiting derivatives of . W separate Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  15. 56 Can it be Used for Optimization? W separate • Binning fluctuations of . • Fluctuations amplified by differentiation. • Fluctuations slow convergence, false minima. Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  16. 57 Quantization and Optimization Function Quantized function Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  17. 57 Quantization and Optimization Function Quantized function Function with a quantized derivative Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  18. Analytic Entropy Gradient Accurate derivative Efficient calculation Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  19. Analytic Entropy Gradient Complexity K- number of sources, N-data length. Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  20. 58 Entropy Gradient by Convolutions Convolution Convolution Convolution Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  21. 59 Entropy Gradient by Convolutions • Calculation of using convolutions. • Approximation of convolutions with complexity. • Distinct quantization of the derivative. Not differentiation of a quantized function. Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

  22. 60 Super ICA performance Non-parametric algorithms. Parametric algorithms. K=6 random sources, N= 3000 samples. Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

More Related