1 / 18

Apache Tika End-to-End

An introduction to Apache Tika, and integrating it to your application. Apache Tika End-to-End. Software Engineer Alfresco. Nick Burch. Apache Tika http://tika.apache.org/. Project which started in 2006 Grew out of the Lucene community, now widely used

rocco
Download Presentation

Apache Tika End-to-End

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An introduction to Apache Tika, and integrating it to your application Apache TikaEnd-to-End

  2. Software Engineer Alfresco Nick Burch

  3. Apache Tikahttp://tika.apache.org/ Project which started in 2006 Grew out of the Lucene community, now widely used Provides detection of files – eg this binary blob is really a word file, that one is UTF-8 plain text Plain text, HTML and XHTML versions of a wide range of different file formats Consistent Metadata from different files Tika hides the complexity of the different formats and their libraries, instead presents a simple, powerful API Easy to use and extend

  4. What's new? Lots of new parsers – text, office formats, publishing formats, images, audio, CAD, fonts etc Long standing parsers improved – better HTML from word for example Embedded resources and containers Use expanding – used by many SOLR users, Alfresco, lots of people crunching masses of data on Hadoop

  5. Supported Formats Page 1 Audio – WAV, RIFF, MIDI DWG (CAD) Epub RSS and ATOM Feeds True Type Fonts HTML Images – JPEG, GIF, PNG, TIFF, Bitmap (including EXIF where found) iWork (Keynote, Pages etc) RFC822 mbox Mail

  6. Supported Formats Page 2 Microsoft Outlook .msg Email Microsoft Office (Binary) – Word, PowerPoint, Excel, Visio, Publisher, Works Microsoft Office (OOXML) – Word, PowerPoint, Excel MP3 (id3 v1 and v2) CDF (Scientific Data) Open Document Format (Open Office) Old-style Open Office (.sxw etc) PDF

  7. Supported Formats Page 3 Zip and Tar archives RDF Plain Text FLV Video XML Java class files And I probably forgot one...!

  8. Metadata Tika provides consistent metadata across the range of parsers No need to know if it's “Last Author”, “Last Editor” or “Previous Author” in a file format, they all come back with the same metadata key Keys and values are strings, but strongly typed metadata entries provide converters to dates, ints etc

  9. Text Content Tika generates HTML-like SAX events as it parses Uses Java SAX API Events can be captured or transformed Body Content Handler used for plain text HTML and XHTML available Can customise with your own handler, with XSLT or with E4X from JavaScript eg HTML Table → CSV

  10. Calling Tika

  11. // Get a content detector, and an auto-selecting Parser TikaConfig config = TikaConfig.getDefaultConfig(); ContainerAwareDetector detector = new ContainerAwareDetector( config.getMimeRepository() ); Parser parser = new AutoDetectParser(detector); // We’ll only want the plain text contents ContentHandler handler = new BodyContentHandler(); // Tell the parser what we have Metadata metadata = new Metadata(); metadata.set(Metadata.RESOURCE_NAME_KEY, filename); // Have it processed parser.parse(input, handler, metadata, new ParseContext());

  12. // Plain text only content handler ContentHandler handler = new BodyContentHandler(); String text = handler.toString(); // XHTML content handler SAXTransformerFactory factory = SAXTransformerFactory.newInstance(); TransformerHandler handler = factory.newTransformerHandler(); handler.getTransformer().setOutputProperty(OutputKeys.METHOD, "xml"); handler.getTransformer().setOutputProperty(OutputKeys.INDENT, "yes"); StringWriter sw = new StringWriter(); handler.setResult(new StreamResult(sw)); String text = sw.toString();

  13. Tika Parsers

  14. Parser Interface Two key methods – what mime types are supported, and do the parsing public interface Parser { Set<MediaType> getSupportedTypes(ParseContext context); void parse(InputStream stream, ContentHandler handler, Metadata metadata, ParseContext context) throws IOException, SAXException, TikaException; }

  15. public class HelloWorldParser implements Parser { public Set<MediaType> getSupportedTypes(ParseContext context) { Set<MediaType> types = new HashSet<MediaType>(); types.add(MediaType.parse("hello/world")); return types; } public void parse(InputStream stream, ContentHandler handler, Metadata metadata, ParseContext context) throws SAXException { XHTMLContentHandler xhtml = new XHTMLContentHandler(handler, metadata); xhtml.startDocument(); xhtml.startElement("h1"); xhtml.characters("Hello, World!"); xhtml.endElement("h1"); xhtml.endDocument(); metadata.set("hello","world"); metadata.set("title","Hello World!"); } }

  16. Demo: Tika-App

  17. Demo: Geo-Tagged Images in Alfresco Share via Tika

  18. Any Questions?

More Related