Frame, Reproducing Kernel and Learning

1. Frame, Reproducing Kernel and Learning Alain Rakotomamonjy Stéphane Canu Perception, Systèmes et Information Insa de Rouen, 76801 St Etienne du Rouvray France Alain.Rakoto,Stephane.Canu@insa-rouen.fr http://asi.insa-rouen.fr/~arakotom NIPS 2000 Workshop on Kernel methods

2. Motivations • Wavelet-based approximation (wavelet or ridgelet networks) are regularization networks? • Construction of multiresolution scheme of approximation • kernel adapted to the structures of function to be learned NIPS 2000 Workshop on Kernel methods

3. Motivations Ctd. • Frame based framework for learning Approximating highly oscillating structure Without losing regularity in smooth region NIPS 2000 Workshop on Kernel methods

4. Road Map • Introduction on Frame • From Frame to Kernels • From Frame kernels to learning • Conclusions and perspectives NIPS 2000 Workshop on Kernel methods

5. Frame : A definition • H : Hilbert Space dot product A sequence of elements of H is a frame of H if there exists A,B > O s.t A,B are the frame bounds NIPS 2000 Workshop on Kernel methods

6. Frame : definition Ctd. • Frame intepretation • Frame allows stable representation • as for all f in H Frame = "Basis" + linear dependency + redundancy being a dual frame of Fn in H NIPS 2000 Workshop on Kernel methods

7. Particular cases of Frame • Tight Frame • Frame with bounds s.t A=B • Orthonormal Basis • A=B=1 • Riesz Basis • Frame elements are linearly independent NIPS 2000 Workshop on Kernel methods

8. Examples of Frame • Tight Frame of IR2 • Frame of L2(IR) F2 F1 F3 Y is an admissible wavelet NIPS 2000 Workshop on Kernel methods

9. Road Map • Introduction on Frame • From Frame to Kernels • From Frame kernels to learning • Conclusions and perspectives NIPS 2000 Workshop on Kernel methods

10. Frameable RKHS • Condition for having a RKHS Suppose H is a Hilbert space of function and a frame of H H is a RKHS if On a frameable Hilbert Space, this is equivalent to The Reproducing Kernel is NIPS 2000 Workshop on Kernel methods

11. Construction of Frameable RKHS • A Practical way to build a RKHS • F is a Hilbert Space of function A finite set of F elements such that   is a RKHS with {Fn} as frame elements NIPS 2000 Workshop on Kernel methods

12. Example of Frameable RKHS • frameable RKHS included in L2(IR) Fi : L2 function (e.g Fi is a wavelet) span {Fi}i=1…N is a RKHS Example 3 wavelets at same scale j span a RKHS with kernel NIPS 2000 Workshop on Kernel methods

13. Road Map • Introduction on Frame • From Frame to Kernels • From Frame kernels to learning • Conclusions and perspectives NIPS 2000 Workshop on Kernel methods

14. Semiparametric Estimation • Context Learning from training set (xi,yi)i=1..N Semiparametric framework One looks for the minimizer of the risk functional in a space H + span{Yi}i=1…m H being a RKHS Under general conditions, span{Yi}i=1…m : parametric hypothesis space NIPS 2000 Workshop on Kernel methods

15. Semiparametric Estimation • Parametric hyp. space is a frameable RKHS P is a frameable RKHS spanned by {Fn}, with P  H, H RHKS Semiparametric estimation on H with P as a parametric hyp. space One looks for the minimizer in H of As spaces are orthogonal, backfitting is sufficient for estimating f* NIPS 2000 Workshop on Kernel methods

16. Semiparametric Estimation • Frame view point • H frameable • H defined by kernel K H= P + N P : Frameable RKHS, N : Frameable RKHS H N: "unknown component" to be regularized P  N : due to linear dependency of frame P : "known component" not to regularized KN=KH-KP P : Frameable RKHS NIPS 2000 Workshop on Kernel methods

17. Multiscale approximation • H a frameable RKHS H is splitted in different spaces {Fi}i=1…m-1 and H0 And any space Hi or Fi is a RKHS Hi : Trend Spaces Fi : Details Spaces NIPS 2000 Workshop on Kernel methods

18. Resid. Resid. Resid. f* Multiscale Approximation Ctd. At each step j, trend obtained at step j-1 is decomposed in trend and details H H2 F2 H1 F1 H0 F0 NIPS 2000 Workshop on Kernel methods

19. Multiscale Approximation Ctd. • Validity • At each step, representer Theorem Hypothesis must be verified • Solution NIPS 2000 Workshop on Kernel methods

20. Illustration on toy problem Function to be learned Data xi : N points from the random sampling of [0, 10] Algorithm - SVM Regression - Multiscale Regularisation networks on Frameable RKHS Sin/Sinc based kernel Wavelet based kernel NIPS 2000 Workshop on Kernel methods

21. Results • N=902 • Results are averagerad over 300 experiments and normalized with regards to SVM performance Wavelet Kernel Sinc Kernel SVM 1 ± 0.096 0.9297 ± 0.312 0.5115 ± 0.098 L2 error 0.7252± 8.022 0.8280± 0.025 1 ± 0.028 NIPS 2000 Workshop on Kernel methods

22. Plots of typical results NIPS 2000 Workshop on Kernel methods

23. Road Map • Introduction on Frame • From Frame to Kernels • From Frame kernels to learning • Conclusions and perspectives NIPS 2000 Workshop on Kernel methods

24. Summary • new design of kernel based on frame elements • algorithm for multiscale learning • But • no explicit definition of kernel • Time-consuming NIPS 2000 Workshop on Kernel methods

25. Future work • Multidimensional extension • Tight Frame of multidimensional wavelet • Using a priori knowledge on the learning problem • How to choose the frame elements? • Theoretical justification and analysis of multiscale approximation NIPS 2000 Workshop on Kernel methods

26. NIPS 2000 Workshop on Kernel methods