Tagging of digital historical images

1. Tagging of digital historical images Authors: A. N. Talbonen (antal@sampo.ru) A. A. Rogov (rogov@psu.karelia.ru) Petrozavodsk state university

2. General tagging model Tag DB Object selection Tag attribution Indexing Full-text index ObjectDB Image collection

3. General research features • Research is based on analysis of image collection of White Sea-Baltic Sea Canal provided by National museum of Karelia • Collection consists of about 8k images with resolution 75 dpi.

4. 1. Face taggingGeneral features • Predominance of small-sized objects (width is less than 40 pixels) • No database • Available expert Distribution of object’s size

5. 1. Face taggingGeneral algorithm • Object (face) detection. • Computing of pairwise distances between objects. • Tagging (for each object): • The system displays a list of the most similar objects. • The expert determines a relationship between objects • Object tags are specified

6. 1. Face taggingFace detection features • There is OpenCV library (OpenCvSharp in C#) and it’smethod cv::CascadeClassifier::detectMultiScale (haarDetectObject in C#) (Viola-Jones implementation) being used for face detection • Viola-Jones method parameters are used to affect on precision and recall on face detection results • There is face recognition method based on Local Binary Patterns being used to improve the quality of Viola-Jones results

7. 1. Face taggingFace detection diagram Source image Face detection Object Training set Face objects Recognition Fake objects Object is a face Yes Insert in result collection

8. 1. Face taggingLocal binary patterns (LBP) Original LBP filter Advanced LBP filters

9. 1. Face taggingLocal binary patterns Uniform codes (patterns) Rotation invariant codes

10. 1. Face taggingLocal binary patterns Computing of face object histogram Weight matrix

11. 1. Face taggingFace detection experiment • The purpose is to find the LBP modification with the best detection rates • Experiment features: • Sample of 1070 images • Assessing features • Fake object when: • Object is not a face • Faces are recognized weakly • Faces turned at an angle greater than 90 degrees • Face object when: • Object is a face • Object is an image of people: portraits, paintings, sculptures • 12 different LBP modifications were used

12. 1. Face taggingFace detection experiment results

13. 1. Face taggingFace recognition experiment • Purpose is to find the LBP modification with the best face recognition rates • Experiment features • Training set contains 19 objects including 3 relevant pairs of face objects and 1 relevant pair of fake objects • 10 LBP modifications were used

14. 1. Face taggingFace recognition experiment Pairs: {1, 15}, {3, 14}, {4, 13}, {7, 9}

15. 1. Face taggingFace recognition experiment results Взвешенный Взвешенный Взвешенный Взвешенный

16. 1. Face taggingFace comparing Training set object’s histograms: Objects at position (row, col): (1,1) and (3, 4) correspond to fake objects and have similar histograms very different from the rest

17. 2. Texture taggingGeneral features • The classifier with tags based on moments is built • Texture searching is based on the built classifier • Search involves finding the segments corresponding to different textures • Minimal segment size to be include in result is 100 pixels

18. 2. Texture taggingMoment-based segmentation Moment calculation function: Source image I Moment imageM00 Moment imageM10 Moment imageM01

19. 2. Texture taggingMoment-based segmentation Moment feature calculation function: F00 Binary segmentation example F10 Precision: 96,7 % F01

20. 2. Texture segmentationImplementation features • Each moment is an image • Moment computing is based on library OpenCV and it’s methodcv::filter2D • Parameter seek is based on developed experiment

21. 2. Texture taggingParameter seek example

22. 2. Texture taggingClassifier features • Set of textures of several classes is given • Each class is assigned a set of tags • Each image is subjected to a separate texture search • Each texture found adds appropriate set of tags to the source image

23. 2. Texture taggingExample Source image

24. 2. Texture taggingExample Classifier example Classifier textures example

27. 2. Texture taggingExperiment • Purpose is to evaluate the search quality • Experiment features • Sample of 100 images • Classifier contains 2 textures

28. 2. Texture taggingSearch quality evaluate method Single texture estimations: • Flag of belonging to • assessed collection • Flag of belonging to • search result General estimations: Flag of relevance

29. 2. Texture taggingExperiment results

30. Thanks for your attention!