SEASR & Meandre for Second Generation Digital Libraries

1. SEASR & Meandre for Second Generation Digital Libraries National Center for Supercomputing ApplicationsUniversity of Illinois at Urbana-Champaign Automated Learning Group

2. SEASR • This project focuses on developing, integrating, deploying, and sustaining a set of reusable and expandable software components and a supporting framework, SEASR will benefit a broad set of data mining applications for scholars in humanities • The key goals established for this effort are a set of software centric directives: • Support the development of a state-of-the-art software environment for unstructured data management and analysis of digital libraries, repositories and archives, as well as educational platforms that are expected to contribute to many of the humanities breakthroughs of the 21st century. • Support the continued development, expansion, and maintenance of end-to-end software system – user interfaces, workflow engines, data management, analysis and visualization tools, collaborative tools, and other software integrated into a complete environment SEASR – to bring the full power of data analytics to the scholars.  • Support education and training for use of this software environment for analysis through workshops to promote its usage among scholars

3. The SEASR Picture

4. SEASR Enables Scholarly Research • Discovery • What hypothesis or rules can be generated by the “features” of the corpus? • What “features” or language of the corpus best describes the corpus? • What are the “similarities” between elements, documents, or corpuses to each other.

5. SEASR Architecture

6. SEASR: Reach + Relevance + Reuse + Repeatability SEASR emphasizes flexibility, scalability, modularity, and access to heterogeneous data and computational systems • Semantic driven environment for SOA interoperability • Encourages sharing and participation for building communities • Modular construction allows flows to be modified and configured to encourage reusability within and across domains • Enables a mashup and integration of tools • Data-intensive flows can be executed on a simple desktop or a large cluster(s) without modification • Computation can be created for distributed execution on servers where the content lives

7. SEASR @ Work – Tag Cloud Counts tokens Several different filtering options supported

8. SEASR @ Work – Dunning Loglikelihood Example showing over-represented Analysis Set: The Project Gutenberg EBook of A Tale of Two Cities, by Charles Dickens Reference Set: The Project Gutenberg EBook of Great Expectations, by Charles Dickens Feature Comparison of Tokens Specify an analysis document/collection Specify a reference document/collection Perform Statistics comparison using Dunning Loglikelihood

9. SEASR @ Work – Date Entities to Simile Timeline Entity Extraction with OpenNLP Dates viewed on Simile Timeline Locations viewed on Google Map

10. SEASR @ Work – Text Clustering Clustering of Text by token counts Filtering options for stop words, Part of Speech Dendogram Visualization

11. SEASR @ Work – Audio Analysis • NEMA: Executes a SEASR flow for each run • Loads audio data • Extracts features for every 10 sec moving window of audio • Loads and applies the models • Sends results back to the WebUI • NESTER: Annotation of Audio via Spectral Analysis

12. SEASR @ Work – Zotero • Plugin to Firefox • Zotero manages the collection • Launch SEASR Analytics • Citation Analysis uses the JUNG network importance algorithms to rank the authors in the citation network that is exported as RDF data from Zotero to SEASR • Zotero Export to Fedora through SEASR • Saves results from SEASR Analytics to a Collection

13. SEASR @ Work – Fedora Repository Search & Browse Interactive Web Application Web Service Zotero Upload to Repository

14. Meandre: Infrastructure • SEASR/Meandre Infrastructure: • Dataflow execution paradigm • Semantic-web driven • Web Oriented • Supports publishing services • Modular components • Encapsulation and execution mechanism • Promotes reuse, sharing, and collaboration The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

15. SEASR Apps SEASR Plugins SEASR Web Apps SEASR Services Meandre Data-Intensive Flows SEASR Components Data Analytics Visualization Gateway Connections Data Persistence Data Transformation Natural Lang Processing Descriptive Statistics Predictive Modeling Discovery Graphing Modeling Vis Info Vis (small multiples) Developer Tools Component Repository Component Discovery Meandre Infrastructure Shared Stores File Systems Metadata Stores SOA Gateways Virtualization Infrastructure

16. Meandre: Data Driven Execution • Execution Paradigms • Conventional programs perform computational tasks by executing a sequence of instructions. • Data driven execution revolves around the idea of applying transformation operations to a flow or stream of data when it is available. • Dataflow Approach • May have zero to many inputs • May have zero to many outputs • Performs a logical operation when data is available The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

17. Meandre: Dataflow Example Output Inputs Value1 Sum Value2 Logical Operation • Dataflow Addition Example • Logical Operation ‘+’ • Requires two inputs • Produces one output • When two inputs are available • Logical operation can be preformed • Sum is output • When output is produced • Reset internal values • Wait for two new input values to become available The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

18. Meandre: The Dataflow Component Data dictates component execution semantics Descriptor in RDFof its behavior The component implementation Inputs Outputs Component P The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

19. Meandre: Semantic Web Concepts • Relies on the usage of the resource description framework (RDF) which uses simple notation to express graph relations written usually as XML to provide a set of conventions and common means to exchange information • Provides a common framework to share and reuse data across application, enterprise, and community boundaries • Focuses on common formats for integration and combination of data drawn from diverse sources • Pays special attention to the language used for recording how the data relates to real world objects • Allows navigation to sets of data resources that are semantically connected. The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

20. Meandre: Metadata Ontologies • Meandre's metadata relies on three ontologies: • The RDF ontology serves as a base for defining Meandre descriptors • The Dublin Core Elements ontology provides basic publishing and descriptive capabilities in the description of Meandre descriptors The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

21. Meandre: Components in RDF ExistingStandards @prefix meandre: <http://www.meandre.org/ontology/> . @prefix xsd: <http://www.w3.org/2001/XMLSchema#> . @prefix dc: <http://purl.org/dc/elements/1.1/> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> . @prefix : <#> . <http://dita.ncsa.uiuc.edu/meandre/e2k/components/limited-iterations> meandre:name "Limited iterations"^^xsd:string ; rdf:type meandre:executable_component ; dc:creator "Xavier Llora"^^xsd:string ; dc:date "2007-11-17T00:32:35"^^xsd:date ; dc:description "Allows only a limited number of iterations"^^xsd:string ; dc:format "java/class"^^xsd:string ; dc:rights "University of Illinois/NCSA Open Source License"^^xsd:string ; meandre:execution_context <http://norma.ncsa.uiuc.edu/public-dav/Meandre/demos/E2K/V1/resources/colt.jar> , <http://norma.ncsa.uiuc.edu/public-dav/Meandre/demos/E2K/V1/resources/gacore.jar> , <http://dita.ncsa.uiuc.edu/meandre/e2k/components/limited-iterations/implementation/> , <http://norma.ncsa.uiuc.edu/public-dav/Meandre/demos/E2K/V1/resources/gacore-meandre.jar> , <http://norma.ncsa.uiuc.edu/public-dav/Meandre/demos/E2K/V1/resources/formj2.0.jar> ; ... The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

22. Meandre: Components Types • Components are the basic building block of any computational task. • There are two kinds of Meandre components: • Executable components • Perform computational tasks that require no human interactions during runtime • Processes are initialized during flow startup and are fired when in accordance to the policies defined for it. • Control components • Used to pause dataflow during user interaction cycles • WebUI may be a HTML Form, Applet, or Other user interface The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

23. Meandre: Create, Publish, & Share • “Components” and “Flows” have RDF descriptors • Easily shared, fosters sharing, & reuse • Allow machines to read and interpret • Independent of the implementations • Combine different implementation & platforms • Components: Java, Python, Lisp, Web Services • Execution: On a Laptop or a High Performance Cluster • A “Location” is RDF descriptor of one to many components, one to many flows, and their implementations The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

24. Meandre: Repository & Locations • Each location represents a set components/flows • Users can • Combine different locations together • Create components • Assemble flows • Share components and flows • Repositories Help • Administrate complex environments • Organize components and flows The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

25. Meandre: Metadata Properties Components and Flows share properties such as component name, creator, creation date, description, tags, and rights. Components specific metadata to describe the components' behavior, it’s location, type of implementation, firing policy, runnable, format, resource location, and execution context Flow specific metadata describes the directed graph of components, components instances, connectors, connector instance data port source, connector, instance data port target, connector instance source, connector instance target, instance name The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

26. Meandre: Programming Paradigm • The programming paradigm creates complex tasks by linking together a bunch of specialized components. Meandre's publishing mechanism allows components developed by third parties to be assembled in a new flow. • There are two ways to develop flows : • Meandre’s Workbench visual programming tool • Meandre’s ZigZag scripting language The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

27. Meandre: Workbench Existing Flow Components Flows Locations Web-based UI Components and flows are retrieved from server Additional locations of components and flows can be added to server Create flow using a graphical drag and drop interface Change property values Execute the flow The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

28. Meandre: ZigZag Script Language • ZigZag is a simple language for describing data-intensive flows • Modeled on Python for simplicity. • ZigZag is declarative language for expressing the directed graphs that describe flows. • Command-line tools allow ZigZag files to compile and execute. • A compiler is provided to transform a ZigZag program (.zz) into Meandre archive unit (.mau). • Mau(s) can then be executed by a Meandre engine. The SEASR project and its Meandre infrastructureare sponsored by The Andrew W. Mellon Foundation

29. DEMO Perform analysis using "Tag Cloud Viewer" on a hard coded web page • Open browser and go to http://seasr.org/documentation/example-flows/tag-cloud-viewer/ • Click on the "Execute" button to launch the creation of a tag cloud view for "Emma" by Jane Austen retrieved from Project Gutenberg

30. Discussion Questions What are data repositories that you utilize in your scholarly research? What tools or applications are being utilized against these repositories?