450 likes | 459 Views
Learn about generative modeling and how it can be used for unsupervised learning beyond clustering. Explore Bayesian networks and parameter estimation techniques for model fitting.
E N D
Generative (Bayesian) modeling 06/04/2016
Slides by (credit to): David M. Blei Andrew Y. Ng, Michael I. Jordan, IdoAbramovich, L. Fei-Fei, P. Perona,J. Sivic, B. Russell, A. Efros, A. Zisserman, B. Freeman,Tomasz Malisiewicz, Thomas Huffman, Tom Landauer and Peter Foltz, Melanie Martin, Hsuan-Sheng Chiu, HaiyanQiao, Jonathan Huang Thankyou!
Generative modeling • unsupervised learning • … beyond clustering • How can we describe/model the world for the computer? • Bayesian networks!
Bayesian networks • Directed acyclic graphs (DAG) whose nodes represent random variables • Arcs represent (directed) dependence between random variables • Filled nodes: observable variables • Empty nodes: hidden (not observable) variables Zi wi1 w2i w3i w4i
Collapsed notation of Bayesian networks • Frames indicates multiplications • E.g. N features and M instances: Zi wi1 w2i w3i w4i
Generative (Bayesian) modeling Find the parameters of the given model which explains/reconstruct the observed data Model „Generative story” DATA
Model „Generative story” • Model = Bayesian network • The structure of the network is given by the human engineer • The form of the nodes’ distribution (conditioned on their parents) is given as well • The parameters of the distributions have to be estimated from data
Parameter estimation in Bayesian network – only observable variables • Bayesian network assumes that the variables only (directly) dependent from their parents → parameter estimation at each node can be carried out separetly • Maximum Likelihood (or Bayesian estimation)
Expectation-Maximisation (EM) • The extension of Maximum Likelihood parameter estimation if hidden variables are present • We search for the parameter vector Φ which maximises the likelihood of the joint of observable variables X and hidden ones Z
Expectation-Maximisation (EM) • Iterativealgorithm. Step l: • (E)xpectationstep: estimatethevalues of Z (calculate expectedvalues) usingΦl • (M)aximizationstep: Maximum likelihoodestimetionbyusing Z
EM example There are two coins. We drop them together but we can observe only the sum of the heads: h(0)=4 h(1)=9 h(2)=2 What is the bias of the coins? Φ1=P1(H), Φ2=P2(H) ?
EM example a single z hidden variable: what is the proportion of the first coin out of h(1)=9 init Φ10=0.2 Φ20=0.5 E-step
EM example M-step
Text classification/clustering • E.g. recognition of documents’ topic or clustering images based on their content • „Bag-of-words model” • The term-document matrix:
Image Bag-of-”words”
N documents: D={d1, … ,dN} • The dictionaryconsists of M words • W={w 1 , … ,w M} • The size of theterm-documentmatrix is N * M and itcontainsthenumber of occurancesof a certainwordinacertaindocument
Document clustering –unigram model • Let’s assign a „topic” to each document • The topics are hidden variables
Generative story of „unigram model” • Howdocumentsgenerate? • „Drop” a topic (and a size) • Foreachwordpositiondrop a wordaccordingtothetopic’sdistribution TOPIC ... Word Word
Unigram model Zi wi1 w2i w3i w4i Each M documents, • Drop a topic z. • Drop a word (independently from the others) from a multinomial distribution conditiond on z
EM for clustering • E-step • M-step
Drawbacks of the bag-of-words model • Word order is ignored • Synonyms: We refer to a concept (object) by multiple words, e.g: álmos-kialvatlan → low recall • Polysemy: most of words have multiple senses, pl: bank, puska → low precision
pLSA ProbabilisticLatentSemanticAnalysis • Weassign a distribution of topicstoeachofthedocument • Topicsstillhave a distrbution over words • The distributionsfoundareinterpretable
Relation to clustering… • A document can belong to multiple clusters • We’re interested in the distribution of topics rather than pushing each doc into a cluster → more flexible
Generative story of pLSA • Howdocumentsgenerate? • Generatethedocument’stopicdistribution • Foreachwordpositiondrop a topicfromthedoc’stopicdistribution • drop a wordaccordingtothetopic’sdistribution TOPICdistribution ... TOPIC TOPIC ... word word
Example money money loan bank DOCUMENT 1: money1 bank1 bank1 loan1river2 stream2bank1 money1river2 bank1 money1 bank1 loan1money1 stream2bank1 money1 bank1 bank1 loan1river2 stream2bank1 money1river2 bank1 money1 bank1 loan1bank1 money1 stream2 .8 loan bank bank loan .2 TOPIC 1 .3 DOCUMENT 2: river2 stream2 bank2 stream2 bank2money1loan1 river2 stream2loan1 bank2 river2 bank2bank1stream2 river2loan1 bank2 stream2 bank2money1loan1river2 stream2 bank2 stream2 bank2money1river2 stream2loan1 bank2 river2 bank2money1bank1stream2 river2 bank2 stream2 bank2money1 river bank .7 river stream river bank stream TOPIC 2
Parameter estimation (model fitting, training) ? DOCUMENT 1: money? bank? bank? loan? river? stream? bank? money? river? bank? money? bank? loan? money? stream? bank? money? bank? bank? loan? river? stream? bank? money? river? bank? money? bank? loan? bank? money? stream? ? TOPIC 1 DOCUMENT 2: river? stream? bank? stream? bank? money?loan? river? stream? loan? bank? river? bank? bank? stream? river?loan? bank? stream? bank? money?loan? river? stream? bank? stream? bank? money?river? stream?loan? bank? river? bank? money?bank? stream? river? bank? stream? bank? money? ? TOPIC 2
pLSA Observable data Topic distributions For documents Term distributions over topics Slide credit: Josef Sivic
Generative story of pLSA • Howdocumentsgenerate? • Generatethedocument’stopicdistribution • Foreachwordpositiondrop a topicfromthedoc’stopicdistribution • drop a wordaccordingtothetopic’sdistribution TOPICdistribution ... TOPIC TOPIC ... word word
pLSA modell For each document d and word position: • For each word position drop a topic from the doc’s topic distribution • drop a word according to the topic’s distribution d zd1 zd2 zd3 zd4 wd1 wd2 wd3 wd4
pLSAforimages (example) z d w N D “eye” Sivic et al. ICCV 2005
pLSA – E-step What is the expected value of hidden variables (topics z) if the parameter values are fixed
pLSA – M-step We use the values of hidden hidden variables p(z|d,w)
EM algorithm • It can converge to local optimum • Stopping condition?
Approximate inference • The E-step in huge networks is unfeasible • There are approaches which are fast but do only approximate inference (=E-step) rather than exact one • The most popular approximate inferece method is: • Drop samples according to Bayesian network • The average of the samples can be used as expected values of hidden variables
Markov Chain Monte Carlo method (MCMC) • MCMC is an approaximate inference method • The samples are not independent from each other but they are generated one by one based on the previous sample (form a chain) • Gibbs sampling is the most famous MCMC method: • The next sample is generated by fixing all but variables and drop a value for the non-fixed one conditioned on the other ones
Drawbacks of pLSA • It can be recalculated from scratch if a new document arrives • The number of parameters increases as the function of the number of instances • d is just an index, it doesn’t fit well into the generative story
1990 1999 2003
An even more complex task Recogniseobjectsinsideimageswithoutanysupervision Aftertrainingthemodelcan be appliedforunknownimagesaswell
Summary • Generative (Bayesian) modelingenablestodefineanydescription/model of theworldwithanycomplexity (clustering is only an exampleforit) • EM algorithm is a generaltoolforsolvingparameterestimationproblemswherelatentvariables is incorporated