Query-oriented Multi-document Summarization via Unsupervised Deep Learning

1. Query-oriented Multi-document Summarization via Unsupervised Deep Learning Yan LIU, Sheng-hua ZHONG, Wenjie LI Department of Computing The Hong Kong Polytechnic University www.comp.polyu.edu.hk/~csshzhong

2. Outline • Problem • Extractive style query-oriented multi-document summarization • Idea • Push out important concepts layer by layer by inheriting extraction ability from deep learning • Methodology • Deep models, consistent with human cortex • Proposed technique • Query-oriented deep extraction • Experiments • DUC 2005, DUC 2006 and DUC 2007 • Conclusion • Provide a human-like document summarization Query-oriented Multi-document Summarization via Unsupervised Deep Learning

3. Outline • Problem • Extractive style query-oriented multi-document summarization • Idea • Extract important concepts layer by layer by inheriting extraction ability from deep learning • Methodology • Deep models, consistent with human cortex • Proposed technique • Query-oriented deep extraction • Experiments • DUC 2005, DUC 2006 and DUC 2007 • Conclusion • Provide a human-like document summarization Query-oriented Multi-document Summarization via Unsupervised Deep Learning

4. Query-oriented Multi-document Summarization • Extractive style query-oriented multi-document summarization • Generate the summary by extracting a proper set of sentences from multiple documents based on the pre-given query • Important in both information retrieval and natural language processing • Multi-document summarization remains a well-known challenge • Automatic generic text summarization  query oriented document summarization • Single-document  multi-document summarization • Extractive approach is the mainstream • Humans do not have difficulty with multi-document summarization • How does the neocortex process the lexical-semantic task? • Aim of this paper • This is the first paper of utilizing deep learning in document summarization • Provide human-like judgment by referencing the architecture of the human neocortex Query-oriented Multi-document Summarization via Unsupervised Deep Learning

5. Outline • Problem • Extractive style query-oriented multi-document summarization • Idea • Extract important concepts layer by layer by inheriting extraction ability from deep learning • Methodology • Deep models, consistent with human cortex • Proposed technique • Query-oriented deep extraction • Experiments • DUC 2005, DUC 2006 and DUC 2007 • Conclusion • Provide a human-like document summarization Query-oriented Multi-document Summarization via Unsupervised Deep Learning

6. Deep Learning • Physical structure • Human: dozens of cortical layers are involved in even the simplest lexical-semantic processing • Deep: model the problem using multiple layers of parameterized nonlinear modules • Evolution of intelligence • Human: the multi-layers structure began to appear in the neocortex starting from old world monkeys about 40 million years ago • Deep: the development of intelligence follows with the multi-layer structure • Propagation of information • Human: several reasons for believing that our lexical-semantic systems contain multi-layer generative models • Deep: layer-wise reconstruction to learn multiple levels of representation and abstraction that helps to make sense of data Query-oriented Multi-document Summarization via Unsupervised Deep Learning

7. Human Neocortex and Deep Architectures (a) Human neocortex (b) An example of deep architectures Query-oriented Multi-document Summarization via Unsupervised Deep Learning

8. Proposed Model in Vision Recognition Samples of first layer weights Examples represent “strokes” of digital (b) Resemble to V1 response (a) Proposed BDBN (c) Saliency map based on BDBN Query-oriented Multi-document Summarization via Unsupervised Deep Learning

9. Outline • Problem • Extractive style query-oriented multi-document summarization • Idea • Extract important concepts layer by layer by inheriting extraction ability from deep learning • Methodology • Deep models, consistent with human cortex • Proposed technique • Query-oriented deep extraction • Experiments • DUC 2005, DUC 2006 and DUC 2007 • Conclusion • Provide a human-like document summarization Query-oriented Multi-document Summarization via Unsupervised Deep Learning

10. Three-stage Learning of QODE • Information • Input: tf value of every word in document • Output: summary • Three-stage learning • Query-oriented concept extraction • Hidden layers are used to abstract the documents using greedy layer-wise extraction algorithm • Reconstruction validation for global adjustment • Reconstruct the data distribution by fine-tuning the whole deep architecture globally • Summary generation via dynamic programming • Dynamic programming is utilized to maximize the importance of the summary with the length constraint Query-oriented Multi-document Summarization via Unsupervised Deep Learning

11. Query-oriented Concept Extraction • A joint configuration ( , ) of the visible layer and the first hidden layer has energy • Utilize Contrastive Divergence algorithm to update the parameter space Query-oriented Multi-document Summarization via Unsupervised Deep Learning

12. Reconstruction Validation for Global Adjustment • Backpropagation adjusts the entire deep network to find good local optimum parameters by minimizing the cross-entropy error • Before backpropagation, a good region in the whole parameter space has been found • The convergence obtained from backpropagation learning is not slow • The result generally converge to a good local minimum on the error surface Query-oriented Multi-document Summarization via Unsupervised Deep Learning

13. Summary Generation via Dynamic Programming • Dynamic programming • Simplify a complicated problem by breaking it down into simpler sub-problems in a recursive manner • Maximize the importance of the summary with the length constraint • The importance of the sentence • The summary length is defined as • The objective function • Dynamic programming function Query-oriented Multi-document Summarization via Unsupervised Deep Learning

14. Outline • Problem • Extractive style query-oriented multi-document summarization • Idea • Extract important concepts layer by layer by inheriting extraction ability from deep learning • Methodology • Deep models, consistent with human cortex • Proposed technique • Query-oriented deep extraction • Experiments • DUC 2005, DUC 2006 and DUC 2007 • Conclusion • Provide a human-like document summarization Query-oriented Multi-document Summarization via Unsupervised Deep Learning

15. Experiment Setting • Database • DUC 2005, DUC 2006 and DUC 2007 • Benchmark datasets of multi-document summarization task evaluation in the Document Understanding Conference (DUC) • Produce query-oriented multi-document summarization with allowance of 250 words • Evaluation standards • ROUGE-1, ROUGE-2 and ROUGE-SU4 • Compared algorithms • Graph-based sentence ranking algorithms • Manifold-ranking model [Wan & Xiao, 2009] • Multiple-modality model [Wan, 2009] • Document-sensitive model [Wei et al, 2010] • Supervised learning based sentence ranking models • SVM classification [Vapnik, 1995] • Ranking SVM[Jochims et al, 2002] • Regression[Ouyang et al, 2011] • Classical relevance and redundancy based selection algorithms • Maximum marginal relevance (Goldstein et al, 2000) • Greedy search (Filatova & Hatzivassiloglou, 2004) • Integer linear program (Mcdonald, 2007) • NIST baseline system (Dang, 2005) Query-oriented Multi-document Summarization via Unsupervised Deep Learning

16. Average Recall Scores Comparison Comparison to representative algorithms on the DUC 2005 Query Oriented Contribution Analysis 1. Query oriented initial weight setting, 2. Query oriented penalty process, 3. Summary importance maximization by DP Query-oriented Multi-document Summarization via Unsupervised Deep Learning

17. Average Recall Scores Comparison Comparison to representative algorithms on the DUC 2006 Comparison to representative algorithms on the DUC 2007 Query-oriented Multi-document Summarization via Unsupervised Deep Learning

18. Query Word Importance Analysis (a) ROUGE-2 Recall performance vs. (b) ROUGE-4 Recall performance vs. Query-oriented Multi-document Summarization via Unsupervised Deep Learning

19. Filtering Out Words in First Layer of QODE The statistical analysis of words in layer Query-oriented Multi-document Summarization via Unsupervised Deep Learning

20. Key Words Discovery of QODE The statistical analysis of words in layer Query-oriented Multi-document Summarization via Unsupervised Deep Learning

21. Candidate Sentence Extraction of QODE Query-oriented Multi-document Summarization via Unsupervised Deep Learning

22. Advanced Extraction Ability of QODE Candidate sentence extracted in layer Query-oriented Multi-document Summarization via Unsupervised Deep Learning

23. Outline • Problem • Extractive style query-oriented multi-document summarization • Idea • Extract important concepts layer by layer by inheriting extraction ability from deep learning • Methodology • Deep models, consistent with human cortex • Proposed technique • Query-oriented deep extraction • Experiments • DUC 2005, DUC 2006 and DUC 2007 • Conclusion • Provide a human-like document summarization Query-oriented Multi-document Summarization via Unsupervised Deep Learning

24. Conclusion and Future Work • Novel document summarization architecture • Simulate the multi-layer physical structure of the cortex • First layer is utilized to filter out not important words • Second layer is utilized to discover key words • Third layer is utilized to extract candidate sentences • Three-stage learning • Simulate the procedure of lexical-semantic processing by human beings • Dynamic programming is utilized to maximize the importance of the summary with the length constraint • Future work • Propose novel deep learning model by referring more characters of human cortex Query-oriented Multi-document Summarization via Unsupervised Deep Learning

25. Q & A Thanks! • Yan LIU, Sheng-hua ZHONG, Wenjie LI • Department of Computing • The Hong Kong Polytechnic University • www.comp.polyu.edu.hk/~csshzhong Query-oriented Multi-document Summarization via Unsupervised Deep Learning