Multimodal Information Access and Synthesis Research Review

1. Multimodal Information Access and Synthesis Research Review Dan Roth Department of Computer Science University of Illinois at Urbana/Champaign

2. MIAS Mission • Most of the data today is unstructured • books, newspaper articles, journal publications, reports, images, and audio and video streams. • How to deal with the huge amount of unstructured data as if it was organized in a database with a known schema. • how to locate, organize, access and analyze unstructured data. MIAS Mission: • develop the theories, algorithms, and tools for analysts to • access a variety of data formats and models • integrate them with existing resources • transform raw data into useful and understandable information.

3. Task Perspective • In the next decade, somepeople will need to • Monitor a multimodal stream of interesting events, entities, threats; • Formulate and evaluate hypotheses with respect to them. • Impossible to touch even a fraction of the data available • Requires interaction, at the appropriate level of semantic abstraction, with a system that can • synthesize, summarize and interpret vast amounts of multimodal information, • integrate observed data with multimodal domain models and background information in multiple formats, • propose hypotheses, and help verify them.

4. Information Access & Management • The key to allowing access to information is to • Match methods to evoke an information need and to • Transform information from one form to another to make it more accessible to the abilities available to a user. • The solution lies in understanding the meaning of information and of peoples’ interaction with it. • Current tools tend to ignore the meaning of information, and operate on surface phenomena (particular words, image segments, and so on). • Doing so complicates all stages of the information use process. • The key is matching the demand of interacting with information to the abilities available to a user when they do it. • This will allow access to information under unusual, but valuable, circumstances (eg., when some modalities cannot be used temporarily)

5. Scenario • Consider an intelligence analyst researching a problem • Iranian nuclear program – generate a list of Iranian nuclear scientists, affiliations, specialties, biographies, photos, and notable recent activities. • Current technologies have solved the problem of collecting and storing huge amounts of information; it would be reasonable to assume that the information she is after does exist; • However, multiple barriers exist on the way to successful analysis, synthesis and decision support, posing significant research challenges. • Medical treatment – what is known about it; who are the experts; what do users say about it; what side effects have been reported • Disease Outbreaks: what is known about it (say, Ebola, Sars); who are the experts; evidence for outbreak; side effects reported, where, when. • Food Protection, Water Safety, Societal Infrastructures and Development project

6. Multimodal Information Access & Synthesis Saeed Zakeri Saeed Zakeri Online Data Sources Analysis & Synthesis Focused Multimodal Data Retrieval Web pages attended * * News Articles Specific Web sites Text Repositories Relational databases Surveillance Videos ... Discovering unusual events, entities, trends threats & associations. Tracking of events, entities & associations Rapid retrieval of all multi modal information about a particular entity Mapping to & augmenting institutional resources Efficient search, querying, question answering, browsing. * * * * * visited Elkhan Factory Text documents U. of Tehran * * Elkhan Factory * * * * * * Relational data loc = Northern Iran name = Elkhan Factory topics = fertilizer, enrichment Semantic categories Temporal categories Subjectivity/Opinions Images Infer Metadata: Semantic entities Discover Trends of & Relations Between Semantic Entities Support Information Analysis, Knowledge Discovery, Monitoring Semantic Disambiguation & Integration across multiple sources and modalities Meaning Based Transformation of Data for Presentation and Analysis Support.

7. MIAS Processes Tools Text Processing& Analysis Semantic Analysis & Information Extraction Information Integration Machine Learning & Knowledge Discovery Integrating Text & Images • Focused data retrieval and integration • Identify and collect relevant data from multiple sources • Semantic data enrichment: Real world Entities and Relations among them • Infer semantics from unstructured data and images; • Identify real-world entities and relations among them • Extraction of attributes and relations features into a common framework (generalized graphs) • Relate them to existing institutional resources for information integration • Trend Analysis • Tracking of events, social content, entities and topics • Knowledge discovery and hypotheses generation and verification • Construct the rich semantic structure and hidden networks of entity linkages • Multifaceted output • Information extraction • Allow semantic based navigation and search across disparate data modalities; • KR: Multi-view representation of the information as input to visualization tools. • Unique Stress on Multidisciplinary Foundations • Machine Learning/Statistical/Data and Knowledge Discovery • integration between Natural Language Processing and DBs • Multimodal: vision and and text

8. Integrated Mission: Research & Education • Develop diverse human resources to enhance the scientific research, educational, and governmental workforce in MIAS • Educational and Outreach Initiatives: • Target students from small research programs & minority-serving • Expose them to the national labs • Open opportunities for bigger impact • A comprehensive education program designed to increase participation in the study and practice of MIAS topics: • Provide substantive training for a new generation of experts in the field, • Serve as a tool for recruiting an experienced group of undergraduates into graduate study in one of the broad fields of information science • Be an intellectual community center, where participants at all levels of expertise come together in an enriched environment of collaboration.

9. Data Science Summer Institute at UIUC 1st DSSI: May 2007 Huge Success 2nd DSSI: May 2008 • Intensive Course • in • The Math of Data • Sciences • Probability and Statistics • Linear Algebra • Data Structures and Algorithms • Optimization • Learning & Clustering • Research Projects • Led by co-PIs and • Grad Students • Topics: • Virtual Web: Focused Crawling • Relations and Entities • Text and Images 8 weeks course 27 students Faculty from UIUC, Kansas State, UTSA, UTEP Advanced MIAS Related Tutorials Speaker Series

10. Machine Learning & KDD Databases & Information Integration Computer Vision Natural Language Processing & Information Extraction Information Retrieval & Web Information Access Data Science Summer Institute at UIUC Advanced MIAS Related Tutorials

11. Data Science Summer Institute at UIUC

12. Our Team • Leading researchers in intelligent information access & analysis and its foundations: • Machine Learning • Data Bases, Data Integration and Knowledge Discovery • Information Retrieval • Natural Language Processing • Machine Vision • Knowledge Representation and Reasoning • A large number of affiliates/consultants covering all areas of interest to the MIAS center.

13. Kevin C. Chang: Data Integration and Retrieval Deep Web MetaQuerier: Large-scale integration over deep Web • pioneered the “holistic integration” paradigm • Widely published at SIGMOD, VLDB, ICDE • System building and demo at CIDR, SIGMOD, VLDB, … • PC/editor/organizers of SIGMOD, ICDE, WWW, SIGKDD special issue, WIRI06, IIWeb06 workshops • Awards: NSF CAREER, IBM Faculty, NCSA Faculty; VLDB00 Best Paper Selection

14. AnHai Doan: Data Integration • Data integration • Matching Schemas, Ontologies, Entities • Integrating databases and text • ACM Doctoral Dissertation Award 2004 • Sloan Fellowship 2006 • Edited special issues on Data Integration • Co-chaired workshops on data integration, Web technologies, machine learning • Co-writing a book titled “Data Integration” Monitoring People and Events

15. David Forsyth: Computer Vision and Learning Linking Text and Images Labeling images via (a lot of) caption text • Leading Computer Vision researcher, • Over 110 papers on vision, graphics, learning applications • Program Chair, CVPR 2000, General chair CVPR 2006, regular member of PC in all major vision conferences • IEEE Technical Achievement Award, 2006 • Lead author of main textbook, widely adopted

16. German supermodel Claudia Schiffer gave birth to a baby boy by Caesarian section January 30, 2003, her spokeswoman said. The baby is the first child for both Schiffer, 32, and her husband, British film producer Matthew Vaughn, who was at her side for the birth. Schiffer is seen on the German television show 'Bet It...?!' ('Wetten Dass...?!') in Braunschweig, on January 26, 2002. (Alexandra Winkler/Reuters) British director Sam Mendes and his partner actress Kate Winslet arrive at the London premiere of 'The Road to Perdition', September 18, 2002. The films stars Tom Hanks as a Chicago hit man who has a separate family life and co-stars Paul Newman and Jude Law. REUTERS/Dan Chung US President George W. Bush (L) makes remarks while Secretary of State Colin Powell (R) listens before signing the US Leadership Against HIV /AIDS , Tuberculosis and Malaria Act of 2003 at the Department of State in Washington, DC. The five-year plan is designed to help prevent and treat AIDS, especially in more than a dozen African and Caribbean nations(AFP/Luke Frazza)

18. Jiawei Han: Knowledge Discovery • Patterns analysis and knowledge discovery from massive data • Research focus: Data streams, frequent patterns, sequential patterns, graph patterns, and their applications • Privacy preserving Data Analysis • Developed many popular data mining algorithms, e.g., FPgrowth, PrefixSpan, gSpan, StarCubing, CrossMine, RankingCube, and CrossClus • Over 300 research papers published in conferences and journals • Editor-in-Chief, ACM Transactions on Knowledge Discovery from Data • Textbook, “Data mining: Concepts and Techniques,” adopted worldwide

19. Cinda Heeren: Knowledge Discovery, Education • MIAS Summer School Director • Mathematical Foundation of Data Science Discrete UIUC CS Department Director of Diversity Programs • Lecturer for Discrete Math, Data Structures courses at UIUC • One of the leading teachers and educational leaders at UIUC. • Research in Algorithmic Data Analysis and Data bases • Speaker and regular presenter at conferences for young women, including: • GAMES and WYSE summer camps • Expanding Your Horizons careers conference • Grace Hopper Celebration of Women in Computing 2004 - panel on best practices for recruiting women into undergraduate programs in CS. • SIGCSE 2006 - workshop, “How to host your own Small Regional Celebration of Women in Computing.” Summer School

20. ChengXiang Zhai: Information Retrieval and Text Analysis • Probabilistic Paradigms for Information Retrieval • Personalized/Context Dependent Search • Relation Identification and Data Integration • Leading expert in information retrieval and search technologies • Recipient of the 2004 Presidential Early Career Award for Scientists and Engineers (PECASE), • Main architect and key contributor of the Lemur Toolkit (being used by many research groups and IR companies around the world) • ACM SIGIR’04 best paper award • Selected services include Program Chair of ACM CIKM 2004; HLT/NAACL 06; SIGIR’09

21. Dan Roth: Machine Learning, NLP, Inference • Semantic Analysis and Data enrichment • Entity and Relation Identification and Integration • Textual Entailment • Machine Learning Methods for NLP and IE • Leading Researcher in Machine Learning, NLP, AI • Developed Popular Machine Learning system and machine learning based NLP tools used in industry and NLP classes. • Program Chair, ACL’03, CoNLL’02; Regular senior PC member in all major Machine Learning, NLP and AI conferences • Associate Editor: Journal of Artificial Intelligence Research; Machine Learning • Multiple papers awards

22. Machine Learning, NLP, Reasoning & Optimization • Foundations • Learning Theory: Algorithmic and representational Issues; high Dimensions; dimensionality reduction • Learning protocols: how to minimize interaction (supervision); how to map domain/task information to supervision; semi-supervised learning; active learning; ranking; adaptation. • Constrained Conditional Models: Global decisions in which several local decisions play a role but there are mutual dependencies on their outcome; NLP Inference as Constrained Optimization. • Natural Language Processing • Semantic Parsing • Question answering • Semantic Entailment • Intelligent Information Access • Information Extraction • Named Entities and Relations • Matching Entities Mentions within and across documents and data bases • Software • Many NLP and IE tools that are being used in research labs and industry • Basic tools development: SNoW, FEX; shallow parser, pos tagger, semantic parser; NER, … • Learning Based Programming

23. Textual Entailment Phrasal verb paraphrasing [Connor&Roth’07] Entity matching [Li et. al, AAAI’04, NAACL’04] • Given: Q: Who acquired Overture? • Determine: A: Eyeing the huge market potential, currently led by Google, Yahoo took over search company Overture Services Inc last year. Semantic Role Labeling Inference for Entailment AAAI’05;TE’07 Is it true that…? (Textual Entailment) Eyeing the huge market potential, currently led by Google, Yahoo took over search company Overture Services Inc. last year  Yahoo acquired Overture Overture is a search company Google is a search company Google owns Overture ……….

24. Penalty for violating the constraint. Weight Vector for “local” models How far away is y from a “legal” assignment A collection of Classifiers; Log-linear models (HMM, CRF) or a combination Constrained Conditional Models Subject to constraints (Soft) constraints component How to solve (for best assignment) ? This is an Integer Linear Program Solve using ILP packages gives an exact solution. Search techniques are also possible How to train? How to decompose global objective function? Should we incorporate constraints in the learning process?

25. Semantic Categories • Information Access and Extraction requires the identification of semantic categories in text. Query: Aids Treatment Federal health officials are recommending aggressive use of a newly approved drug that protects people infected with the AIDS virus against a form of pneumonia that is the No.1 killer of AIDS victims. (AP890616-0048, TIPSTER VOL. 1) Relevant documents may mention specific types of treatments for AIDS Hemophiliacs lack a protein, called factor VIII, that is essential for making blood clots. As a result, they frequently suffer internal bleeding and must receive infusions of clotting protein derived from human blood. During the early 1980s, these treatments were often tainted with the AIDS virus. (AP890118-0146, TIPSTER Vol. 1) Many irrelevant documents mention AIDS and treatments for other diseases • There is a need to identify that this phrase represent a name of an organization, a name of a person, a name of a disease, a medicine, etc. • A narrow version of the problem is called: named entity recognition (NER)

26. Adaptation of Named Entity Recognition • Entities are inherently ambiguous (e.g. JFK can be both location and a person depending on the context) • Can appear in various forms ; Can be nested. • Using lists is not sufficient • New entities are always being introduced New NE seen • A lot of Machine Learning work – significant over fitting • Key difficulties – Adaptation to: • New domains/corpora • Slightly new definition of an entity • New languages • New types of entities . NE seen • How to reduce the requirements on the resources needed to produce a semantic categorization for a new domain/new language/new type of entities

27. NER Tools Screen shot from a CCG demo http://L2R.cs.uiuc.edu/~cogcomp • Work in progress: • Un-supervised discovery of entities in other languages • Quick adaptation to new entity types and new domains.

28. Extracting Relations • Information Access and Extraction requires the identification of relations between concepts in text. • Relations expressed within a single sentence or paragraph • Relations uncovered by processing large quantities of text (over time) • There is a need to identify concepts (e.g., entities) and relations that hold between them in a given sentence. • Closed set of relations: • [A causes B] • [A works for B] • [A prevents B] • [A lives in B] • Open ended set of relations • Every predicate can be a relation

29. Extracting Relations via Semantic Analysis Screen shot from a CCG demo http://L2R.cs.uiuc.edu/~cogcomp • Semantic parsing reveals several relations in the sentence along with their arguments. • This level of analysis, however, cannot abstract over the inherent variability in expressing the relations. . • Kill and Explode can be expressed in many different ways.

30. Information extraction [ACL’07] with Background Knowledge (Constraints) Lars Ole Andersen . Program analysis and specialization for the C Programming language. PhD thesis. DIKU , University of Copenhagen, May 1994 . Prediction result of a trained HMM Lars Ole Andersen . Program analysis and specialization for the C Programming language . PhD thesis . DIKU , University of Copenhagen , May 1994 . [AUTHOR] [TITLE] [EDITOR] [BOOKTITLE] [TECH-REPORT] [INSTITUTION] [DATE] Violates lots of constraints!

31. Information Extraction [ACL’07]with (background Knowledge) Constraints • Learn simple models. • Add constraints, to improve model expressivity and getcorrectresults! • [AUTHOR]Lars Ole Andersen . [TITLE]Program analysis andspecialization for the C Programming language . [TECH-REPORT] PhD thesis . [INSTITUTION] DIKU , University of Copenhagen , [DATE] May, 1994 . • If incorporated into semi-supervised training, better results mean • Better Feedback!

32. Variability Ambiguity Why is it difficult? Meaning Language

33. Kennedy The Reference Problem The same problem exists with other types of entities Document 1:The Justice Department has officially ended its inquiry into the assassinations ofJohn F. Kennedyand Martin Luther King Jr., finding ``no persuasive evidence'' to support conspiracy theories, according to department documents. The House Assassinations Committee concluded in 1978 thatKennedywas ``probably'' assassinated as the result of a conspiracy involving a second gunman, a finding that broke from the Warren Commission's belief that Lee Harvey Oswald acted alone in Dallas on Nov. 22, 1963. Document 2: In 1953, MassachusettsSen. John F. Kennedymarried Jacqueline Lee Bouvier in Newport, R.I. In 1960, Democratic presidential candidate John F. Kennedy confronted the issue of his Roman Catholic faith by telling a Protestant group in Houston, ``I do not speak for my church on public matters, and the church does not speak for me.'‘ Document 3:David Kennedywas born in Leicester, England in 1959.  …Kennedyco-edited The New Poetry (Bloodaxe Books 1993), and is the author of New Relations: The Refashioning Of British Poetry 1980-1994 (Seren 1996). 

34. Entity/Concept Identification in Text • Goal: Given names in text documents and their semantic types, identify real-world entities they represent. • A similarity measure between names [entity type dependent] • A way to group different looking strings into one group • A context sensitive way to distinguish between identical/similar strings that represent different entities • A generative Model [Li, Morie, Roth, NAACL’04] • A discriminative approach [Li, Morie, Roth, AAAI’04] • Summary: AI Magazine Special Issue on Semantic Integration’05 • Goal:Semantic Integration: Text, Databases and Institutional Recourses • Map concepts identified in text to entries in databases. • Construct/augment databases from textual information. • Aid discovery in text using existing knowledge bases.

35. Demo Screen shot from a CCG demo http://L2R.cs.uiuc.edu/~cogcomp More work on this problem: Scaling up Integration with DBs Temporal Integration/Inference …… Related Entities – Context

36. MIAS Processes Tools Text Processing& Analysis Semantic Analysis & Information Extraction Information Integration Machine Learning & Data Mining Integrating Text & Images • Focused data retrieval and integration • Identify and collect relevant data from multiple sources • Semantic data enrichment: Real world Entities and Relations among them • Infer semantics from unstructured data and images; • Identify real-world entities and relations among them • Extraction of attributes and relations features into a common framework (generalized graphs) • Relate them to existing institutional resources for information integration • Trend Analysis • Tracking of events, social content, entities and topics • Knowledge discovery and hypotheses generation and verification • Construct the rich semantic structure and hidden networks of entity linkages • Multifaceted output • Information extraction • Allow semantic based navigation and search across disparate data modalities; • KR: Multi-view representation of the information as input to visualization tools. • Unique Stress on Multidisciplinary Foundations • Machine Learning/Statistical/Data and Knowledge Discovery • integration between Natural Language Processing and DBs • Multimodal: vision and and text

37. MIAS - DSSI • Thank You

38. MIAS Mission • Most of the data today is unstructured • books, newspaper articles, journal publications, reports, images, and audio and video streams. • How to deal with the huge amount of unstructured data as if it was organized in a database with a known schema. • how to locate, organize, access and analyze unstructured data. MIAS Mission: • develop the theories, algorithms, and tools for analysts to • access a variety of data formats and models • integrate them with existing resources • transform raw data into useful and understandable information.