This week: overview on pattern recognition (related to machine learning)

1. This week: overview on pattern recognition (related to machine learning)

2. Non-review of chapters 6/7 • Z-transforms • Convolution • Sampling/aliasing • Linear difference equations • Resonances • FIR/IIR filtering • DFT/FFT

3. Speech Pattern Recognition • Soft pattern classification plus temporal sequenceintegration • Supervised pattern classification: class labels used in training • Unsupervised pattern classification: labels not available (or at least not used)

5. Training and testing • Training: learning parameters of classifier • Testing: classify independent test set, compare with labels, and score

6. F1 and F2 for various vowels

7. Swedish basketball players vs.speech recognition researchers

8. Feature extraction criteria • Class discrimination • Generalization • Parsimony (efficiency)

11. Feature vector size • Best representations for discrimination on training set are highly dimensioned • Best representations for generalization to test set tend to be succinct

12. Dimensionality reduction • Principal components (i.e., SVD, KL transform, eigenanalysis ...) • Linear Discriminant Analysis (LDA) • Application-specific knowledge • Feature Selection via PR Evaluation

15. PR Methods • Minimum distance • Discriminant Functions • Linear • Nonlinear(e.g., quadratic, neural networks) • Some aspects of each - SVMs • Statistical Discriminant Functions

16. Minimum Distance • Vector or matrix representing element • Define distance function • Collect examples for each class • In testing, choose the class of the closest example • Choice of distance equivalent to an implicit statistical assumption • Signals (e.g. speech) add temporal variability

18. Limitations • Variable scale of dimensions • Variable importance of dimensions • For high dimensions, sparsely sampled space • For tough problems, resource limitations (storage, computation, memory access)

19. Decision Rule for Min Distance • Nearest Neighbor (NN) - in the limit of infinite samples, at most twice the error of optimum classifier • k-Nearest Neighbor (kNN) • lots of storage for large problems; potentially largesearches

20. Some Opinions • Better to throw away bad data than to reduce itsweight • Dimensionality-reduction based on variance often abad choice for supervised pattern recognition • Both of these are only true sometimes

21. Discriminant Analysis • Discriminant functions max for correct class, min for others • Decision surface between classes • Linear decision surface for 2-dim is aline, for 3 is a plane; generally called hyperplane • For 2 classes, surface at wTx + w0 = 0 • 2-class quadratic case, surface at xTWx+ wTx + w0 = 0

22. Two prototype example • Di2 = (x – zi)T (x – zi) = xTx + ziTzi– 2 xTzi • D12 – D22 = 2 xTz2 – 2 xTz1 + z1Tz1– z2Tz2 • At decision surface, distances are equal, so • 2 xTz2 – 2 xTz1= z2Tz2 – z1Tz1 Or xT(z2 – z1) = ½ (z2Tz2 – z1Tz1) And if prototypes are all normalized to 1, Decision surface is • xT(z2 – z1) = 0 • And each discriminant function is xTzi

23. System to discriminate between two classes

24. Training Discriminant Functions • Minimum distance • Fisher linear discriminant • Gradient learning

25. Generalized Discriminators - ANNs • McCulloch Pitts neural model • Rosenblatt Perceptron • Multilayer Systems

29. Typical unit for MLP Σ

31. Typical MLP

32. Support Vector Machines (SVMs) • High dimensional feature vectors • Transformed from simple features (e.g., polynomial) • Can potentially classify training set arbitrarily well • Improve generalization by maximizing margin • Via “Kernel trick”, don’t need explicit high dim • Inner product function between 2 points in space • Slack variables allow for imperfect classification

33. Maximum margin decision boundary, SVM

34. Unsupervised clustering • Large and diverse literature • Many methods • Next time one method explained in the context of a larger, statistical system

35. Some PR/machine learning Issues • Testing on the training set • Training on the test set • # parameters vs # training examples: overfittingand overtraining • For much more on machine learning, CS 281A/B