740 likes | 847 Views
Similarity-based Classifiers: Problems and Solutions. Classifying based on similarities :. Van Gogh. Monet. Van Gogh Or Monet ?. the Similarity-based Classification Problem. (paintings). (painter). the Similarity-based Classification Problem.
E N D
Classifying based on similarities: Van Gogh Monet Van Gogh Or Monet ?
the Similarity-based Classification Problem (paintings) (painter)
Examples of Similarity Functions Computational Biology • Smith-Waterman algorithm (Smith & Waterman, 1981) • FASTA algorithm (Lipman & Pearson, 1985) • BLAST algorithm (Altschul et al., 1990) Computer Vision • Tangent distance (Duda et al., 2001) • Earth mover’s distance (Rubner et al., 2000) • Shape matching distance (Belongie et al., 2002) • Pyramid match kernel (Grauman & Darrell, 2007) Information Retrieval • Levenshtein distance (Levenshtein, 1966) • Cosine similarity between tf-idf vectors (Manning & Schütze, 1999)
Example: Amazon similarity 96 books 96 books
Example: Amazon similarity 96 books 96 books
Example: Amazon similarity 96 books Eigenvalues Rank 96 books
Well, let’s just make S be a kernel matrix Flip, Clip or Shift? Best bet is Clip. 0 0
Well, let’s just make S be a kernel matrix Learn the best kernel matrix for the SVM: (Luss NIPS 2007, Chen et al. ICML 2009)
Let the similarities to the training samples be features • SVM (Graepel et al., 1998; Liao & Noble, 2003) • Linear programming (LP) machine (Graepel et al., 1999) • Linear discriminant analysis (LDA) (Pekalska et al., 2001) • Quadratic discriminant analysis (QDA) (Pekalska & Duin, 2002) • Potential support vector machine (P-SVM) (Hochreiter & Obermayer, 2006; Knebel et al., 2008)
Weighted Nearest-Neighbors Take a weighted vote of the k-nearest-neighbors: Algorithmic parallel of the exemplar model of human learning. ?
Weighted Nearest-Neighbors Take a weighted vote of the k-nearest-neighbors: Algorithmic parallel of the exemplar model of human learning.
Design Goals for the Weights ? Design Goal 1 (Affinity):wi should be an increasing function of ψ(x, xi).
Design Goals for the Weights (Chen et al. JMLR 2009) ? Design Goal 2 (Diversity):wi should be a decreasing function of ψ(xi, xj).
Linear Interpolation Weights Linear interpolation weights will meet these goals:
Linear Interpolation Weights Linear interpolation weights will meet these goals:
LIME weights Linear interpolation weights will meet these goals: Linear interpolation with maximum entropy (LIME) weights (Gupta et al., IEEE PAMI 2006):
LIME weights Linear interpolation weights will meet these goals: Linear interpolation with maximum entropy (LIME) weights (Gupta et al., IEEE PAMI 2006):
LIME weights Linear interpolation weights will meet these goals: Linear interpolation with maximum entropy (LIME) weights (Gupta et al., IEEE PAMI 2006):
LIME weights Linear interpolation weights will meet these goals: Linear interpolation with maximum entropy (LIME) weights (Gupta et al., IEEE PAMI 2006):
Kernelize Linear Interpolation regularizes the variance of the weights
Kernelize Linear Interpolation only need inner products – can replace with kernel or similarities!
KRI Weights Satisfy Design Goals Kernel ridge interpolation (KRI) weights:
KRI Weights Satisfy Design Goals Kernel ridge interpolation (KRI) weights: affinity:
KRI Weights Satisfy Design Goals Kernel ridge interpolation (KRI) weights: diversity:
KRI Weights Satisfy Design Goals Kernel ridge interpolation (KRI) weights:
KRI Weights Satisfy Design Goals Kernel ridge interpolation (KRI) weights: Remove the constraints on the weights: Can show equivalent to local ridge regression: KRR weights.
Weighted k-NN: Example 1 KRI weights KRR weights
Weighted k-NN: Example 2 KRI weights KRR weights
Weighted k-NN: Example 3 KRI weights KRR weights