Informational Complexity Notion of Reduction for Concept Classes

1. Informational Complexity Notion of Reduction for Concept Classes Shai Ben-David Cornell University, and Technion Joint work with Ami Litman Technion

2. Measures of the Informational Complexity of a class The VC-dimension of the class. The sample complexity for learning the class from random examples. The optimal mistake bound for learning the class online (or the query complexity of learning this class using membership and equivalence queries). The size of the minimal compression scheme for the class.

3. Outline of the talk Defining our reductions, and the induced notion of complete concept classes. Introducing a specific family of classes that contains many natural concept classes. Prove that the class of half-spaces is complete w.r.t. that family. Demonstrate some non-reducability results. Corollaries concerning the existence of compression schemes.

4. Defining Reductions We consider pair of sets (X,Y) where X is a domain and Y is a set of concepts. A concept class is arelations R overXxY (so eachyeY can be viewed as the subset{x: (x,y)eR}ofX ). An embedding of C=(X,Y,R) into C’=(X’,Y’,R’) is a pair of functions r:X X’, t:Y Y’, so that (x,y)eR iff(r (x), t (y))eR’ . Creduces toC’, denoted C C,’ if such an embedding exits.

5. Relationship to Info Complexity If C C’then, for each of the complexity parameters mentioned above, C’ is at least as complex asC. E.g., if C C,’ then, for every e and d, the sample complexity of (e, d) learning C is at most that needed for learning C’. (This is in the agnostic prediction model)

6. Immediate observations If we take into account the computational complexity of the embedding functions, then we can also bound the computational complexity of learning C by that of learning C’ For every k, the class of all binary functions on a k-size domain is minimal w.r.t. the family of all classes having VC-dimensionk.

7. Universal Classes We say that a concept class C is universal for a family of classesFif every member of F reduces to C . Universal classes play a role analogous to that of, say, NP-hard decision problems – they are as complex as any member of the family F

8. Some important classes For an integer k, let HSk denote the class of half spaces over Rk. That is HSk=(Rk, Rk+1, H) where ((x1,….xk),(a1,…ak+1))eH iff Saixi +ak+10 Let PHSk denote the class of positive half spaces, that is, half spaces in which a1=1. Finally, let HSk0denote the class of homogenous half spaces (I.e., those having ak+1=0), and PHSk0 the class of poditive and homogenous half spaces.

9. Half Spaces and Completeness The first family of classes that comes to mind is the family VCn- the family of all concept classes having VC-dimensions n. Theorem: For anyn>2, no classHSk is universal forVCn (This holds even if we consider only finite classes)

10. Dudley Classes (1) Next, we define a rich subfamily of VCn for which classes of half spaces are universal. LetF be a family of real valued functions over some domain set X. For any function g , let h be any real valued functionover X and definea concept class DF,h = (X, F, RF,h ) where RF,h = {(x,f) : f(x)+h(x)0}. (Note that all the PPD’s defined by Adam yesterday were of this form)

11. Dudley Classes (2) Classes of the form DF,h = (X, F, RF,h ) are called Dudley Classesif the family of functions F is a vector space over the reals (with respect to point-wise addition and scalar multiplication). Examples of Dudley classes: HSk , PHSk ,HSk0 ,PHSk0 , and the class of all balls in any Euclidean space Rk

12. Dudley’s Theorem Theorem: If the a family of functionsF is a vector space, then, for everyh, the VC dimension ofDF,h equals the (linear) dimension of the vector spaceF . Corollary: Easy calculations for the VC dimension of the classes HSk , PHSk ,HSk0 ,PHSk0 , k-dimensional balls.

13. A Completeness Theorem Theorem: For every k, PHSk+10isuniversal, (and therefore, complete) for the family of all k -dimensional Dudley classes. Proof: Let f1 , …fk be a basis for the vector space F, define r:X Rk+1, t:F Rk+1, be r(x) = (f1(x), …. fk(x), h(x)) and for f= Saifi t(f)=(a1, …ak, 1, 0)

14. Corollaries k-size compression schemes for any k-dimensional Dudley class. Learning algorithms for all Dudley classes. An easy proof to Dudley’s theorem. (show that for anyk–dimensionalF, the classHSk0 is embeddable into DF,h ,for h=0)