Groundtruthing for Performance Evaluation of Document Image Analysis Systems: a primer

1. Groundtruthing for Performance Evaluation of Document Image Analysis Systems: a primer Mathieu Delalandre mathieu.delalandre@univ-tours.fr Pattern Recognition and Image Analysis Group Laboratory of Computer Science François Rabelais UniversityTours city, France Digidoc meeting, 6th of January 2012

2. Results dATA Data Results Results Data Groundtruth Groundtruth Groundtruth Groundtruthing for Performance Evaluation of Document Image Analysis Systems: a primer“Introduction” Degradation Performance evaluation is a particular cross-disciplinary research field in a variety of domains. Its purpose is the development of frameworks to evaluate and compare a set of methods in order to select the best-suited for a given application. Training data System Groundtruth must be reliable (i.e. 100% recognition rate) and exhaustive (label, localization, geometric transforms, noise estimation, metadata, etc.) Groundtruthing Characterisation Considering the document image analysis field (apart of the graphics), five main approaches exist. Performance evaluation

3. Groundtruthing for Performance Evaluation of Document Image Analysis Systems: a primer“GUI based groundtruthing” Principles:GUI plugged to a DIA systems, based on user correction. e.g.TrueViz [Kan’01], Xmillum [Hitz’00], PinkPanther [Yanikoglu’01], PerfectDoc [Yacoub’05], etc. Pros: Discussion about groundtruth formalism Cons: Time consuming considering the user correction, specific DIA chains must be designed for every application, groundtruth is not still reliable.

4. Groundtruthing for Performance Evaluation of Document Image Analysis Systems: a primer“Semi-automatic transcription” Principles: To exploit the context and user interaction to make more robust the recognition process. Transcription is achieved at metadata level, without considering the images. e.g.[Bal’ 08], [Lebourgeois’ 01] Algorithms:binarization and connected component labeling, shape context, image distance, clustering, etc. Pros: Interesting idea, 5% of labeling could result in 95% of correct transcription. Cons: What about the robustness, are we sure of a complete transcription, what about the impact of the segmentation, robustness of the approach is not proved yet.

5. Groundtruthing for Performance Evaluation of Document Image Analysis Systems: a primer“Electronic document mapping Principles:A registration algorithm estimates the global geometric transformation and then performs a robust local bitmap match to register an ideal document image to its corresponding scanned version. e.g.[Kan’96], [Hobby’98], [Beusekom’08], [Kim’02] Algorithms: Registration for transformation estimation, RAST (Recognition using Adaptive Subdivision of Transformation space), branch-and-bound algorithm Pros: The strongest approach of the literature. Cons: Can’t be applied with “old” documents, as an electronic version is mandatory.

6. Groundtruthing for Performance Evaluation of Document Image Analysis Systems: a primer“Transcript mapping” Principles:Transcript mapping eases the construction of document image segmentation ground truth that includes text-image alignment. e.g. [Stamatopoulos’10], [Zinger’09], [Jawahar’07], etc. Algorithms: HHM, DTW Pros: When no electronic documents exist, certainly the only valid way to obtain a groundtruth at the graphical level. Cons: Depends of the quality of transcriptions, producing transcriptions is time consuming, the approach is more sensitive to segmentation errors.

7. Groundtruthing for Performance Evaluation of Document Image Analysis Systems: a primer“Generation of synthetic document” Principles:In such a system, the test documents are generated by an automatic system which combines pre-defined models of document components in a pseudo-random way. As documents are synthetically generated, the groundtruth becomes automatically available. e.g.[Heroux’07], [Zi’05], etc. Pros: No previous data is mandatory, efficient and exhaustive groundtruth is generated automatically. Cons: Synthetic is not real, to prove similarity between synthetic and real data is not so simple.