380 likes | 391 Views
Learn about hidden variables, tracking, regression, classification, universal approximators, structured modeling, training algorithms, error control, validation testing, regularization, and more in acoustic signal processing.
E N D
Mark Hasegawa-Johnson ECE Department and Beckman Institute University of Illinois at Urbana-Champaign Pattern Recognition in Acoustic Signal Processing
Motivation Perfect knowledge is unattainable. Solution: Identify variables with unknown values (“hidden variables” or “parameters”) Devise an algorithm that will learn them, in situ Inference vs. Tracking: Inference: learn fixed but unknown system parameters Tracking: track rapidly changing hidden variables Regression vs. Classification Regression: hidden variables are continuous-valued Classification: hidden variables are discrete-valued
Outline Regression examples Classification examples Inference Universal approximators Generalization error: Constraints vs. regularization Structured modeling: Constraints vs. regularization Tracking Convex programming: dynamics as constraints Bayesian formulation: dynamics as regularization
Universal Approximators Show sigmoid and RBF networks, mixture Gaussian PDFs, Parzen windows
Training Corpus Error Algorithms for minimizing training error for sigmoid, RBF networks
Law of Large Numbers Bounds on generalization error
Parameter Count Show that dVC is less than or equal to the number of parameters in the classifier Suggest hyperplane classifiers with few parameters
Parameter Volume Show that dVC is less than or equal to data volume divided by parameter volume, for a hyperplane SVM
Validation Testing Validation testing is a method for controlling generalization error
Error Control: Three Paths Describe the dichotomy: constrained vs. regularized learning vs. cross-validation
Validation-Based Learning Outline of typical validation-based learning, e.g., Mixture Gaussian Neural net
Constraint-Based Learning Outline of typical constrained learning, e.g., Model space limited by prior knowledge, e.g., phones, imaging classes
Regularized Learning Bayesian/MAP learning MDL-type regularization terms SVM/SVR AdaBoost Emphasize: each of these is ultimately confirmed by validation testing!!
Bayesian Regularization Examples where MAP is appropriate, e.g., speaker adaptation, speaker ID, language ID
MDL Regularization Model selection from limited data, e.g., determining whether prosodically tagged allophones should be modeled separately or jointly (Sarah's old paper)
Margin Regularization: SVM Example: landmark classification (Niyogi, Juneja, Borys)
Margin Regularization: SVR Example: acoustics to articulation using SVR (Vikram's work)
Margin Regularization:AdaBoost Example: AdaBoost feature selection for acoustic event detection (Zhuang and Zhou)
Inference Summary Some aspects of inference, difficult to control using validation, are controlled by other methods Feature selection: knowledge, or boosting Feature design: knowledge, or SVM, or SVR Class definitions: knowledge, or MDL Some aspects of inference easier to control using validation e.g., how many Gaussians? How many features?
Discrete Tracking Example:Speech Recognition Speech recognition
Continuous Tracking Ex.:Passive Imaging Source tracking
Bayesian Formulation General Bayesian formulation of the tracking problem
Discrete Example:Acoustic Event Detection Zhuang and Zhou AED system description
Continuous Example:Articulatory Inference Mixture Gaussian regression w/state
Constrained Tracking Example: passive imaging with discrete number of dynamic options (cite the paper?)
Long-Term Constraints Introduce Roth & Yih's ILP methods for NLP
Conclusions conclusions