110 likes | 449 Views
Ronan Collobert Jason Weston Leon Bottou Michael Karlen Koray Kavukcouglu Pavel Kuksa. NATURAL LANGUAGE PROCESSING (ALMOST) FROM SCRATCH. INTRODUCTION. Common Approaches in NLP Using Task-specific features Knowledge injection about structure of data Expertise from Linguists
E N D
Ronan Collobert Jason Weston Leon Bottou Michael Karlen KorayKavukcouglu PavelKuksa NATURAL LANGUAGE PROCESSING (ALMOST) FROM SCRATCH
INTRODUCTION • Common Approaches in NLP • Using Task-specific features • Knowledge injection about structure of data • Expertise from Linguists • Approach used in the paper • No task-specific feature engineering • Minimal prior Knowledge
Common Benchmark Datasets • Part-Of-Speech (POS) Tagging • Syntactic Parsing • Chunking • Shallow Parsing • Named Entity Recognition • Person, Location etc. • Semantic Role Labelling
The Network • Words to Feature Vectors • Look up Table • Random initialization vs Unsupervised Pre-training • Extending to any Discrete Features • Extracting Higher level Features from Word Feature Vectors • Window Approach • Sentence Approach
Training Schemes • Word-Level Log Likelihood • Only words are taken independently for optimizing the weights • Sentence-Level Log Likelihood • Optimization function takes into account all the tags as well as transitions between tags • Stochastic Gradient • Standard Optimization Algorithm