1 / 109

Computer Assisted Visual InterActive Recognition (CAVIAR)

Computer Assisted Visual InterActive Recognition (CAVIAR). Jie Zou RPI ECSE DocLab. Advisor: Prof. George Nagy Committee: Prof. Qiang Ji Prof. Robert B. Kelley Prof. Mukkai Krishnamoorthy. Agenda. Introduction Related research CAVIAR methodology Interactive segmentation

gochoa
Download Presentation

Computer Assisted Visual InterActive Recognition (CAVIAR)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Computer Assisted Visual InterActive Recognition(CAVIAR) Jie Zou RPI ECSE DocLab Advisor: Prof. George Nagy Committee: Prof. Qiang Ji Prof. Robert B. Kelley Prof. Mukkai Krishnamoorthy Jie Zou, Doclab ECSE, RPI

  2. Agenda • Introduction • Related research • CAVIAR methodology • Interactive segmentation • CAVIAR – flower recognition system • CAVIAR – face recognition system • Conclusions Jie Zou, Doclab ECSE, RPI

  3. Agenda • Introduction • Related research • CAVIAR methodology • Interactive segmentation • CAVIAR – flower recognition system • CAVIAR – face recognition system • Conclusions Jie Zou, Doclab ECSE, RPI

  4. Motivation • All operational systems require human assistance (preprocessing, handling rejects). • CAVIAR makes parsimonious use of human visual talent throughout the process rather than only at the beginning or the end. Jie Zou, Doclab ECSE, RPI

  5. Scope of CAVIAR • Visual pattern recognition only • Each CAVIAR system addresses a specific domain • Many class classification • Low throughput Jie Zou, Doclab ECSE, RPI

  6. Research Goals • Allocation of human and machine responsibilities • Mathematical model • Framework and design principles • Prototype CAVIAR systems Jie Zou, Doclab ECSE, RPI

  7. Agenda • Introduction • Related research • CAVIAR methodology • Interactive segmentation • CAVIAR – flower recognition system • CAVIAR – face recognition system • Conclusions Jie Zou, Doclab ECSE, RPI

  8. Content-Based Image Retrieval • Typical search of CBIR system • Submit a query image. • Specify the relative importance of the features. • Relevance feedback (label the retrieved images as acceptable or not-acceptable). • Iterates until user finds the desired image. Jie Zou, Doclab ECSE, RPI

  9. CBIR vs. CAVIAR CBIR CAVIAR Subjective retrieval Objective classification User judges retrieval results Statistical decision boundary Machine weights features User weights features Narrow domain Broad domain Relevance feedback Relevance feedback Model adjustment Jie Zou, Doclab ECSE, RPI

  10. Flower Recognition • Little research on automatic flower recognition • M. Das, R. Manmatha, and E.M. Riseman, “Indexing flower patent images using domain knowledge," IEEE Intelligent Systems, vol. 14, no. 5, pp. 24-33, 1999. Jie Zou, Doclab ECSE, RPI

  11. Face Recognition • Started in 1960’s. Now, most active pattern recognition application • Eigenface, dominant method • Geometrical feature models are appropriate for interactive recognition Jie Zou, Doclab ECSE, RPI

  12. Agenda • Introduction • Related research • CAVIAR methodology • Interactive segmentation • CAVIAR – flower recognition system • CAVIAR – face recognition system • Conclusions Jie Zou, Doclab ECSE, RPI

  13. Psychophysics • Attneave (1954): “the nature of redundancy in visual stimulations”, and “information is concentrated along contours.” • Miller (1956): plus or minus 7 Jie Zou, Doclab ECSE, RPI

  14. Allocation of Human and Machine Responsibilities Conventional System CAVIAR Jie Zou, Doclab ECSE, RPI

  15. Notation CAVIAR state Model parameters Features Index vector Training set Label Jie Zou, Doclab ECSE, RPI

  16. Formal Description (1) • Finite state machine • Initial state created by: • Model building • Feature extraction • Indexing Jie Zou, Doclab ECSE, RPI

  17. Formal Description (2) • Model manipulation leads to a state transition from state n to state n+1: • Model building , • Feature extraction • Indexing Jie Zou, Doclab ECSE, RPI

  18. Formal Description (3) • The task can terminate at any state by identification. Jie Zou, Doclab ECSE, RPI

  19. Illustration (Video) Jie Zou, Doclab ECSE, RPI

  20. Agenda • Introduction • Related research • CAVIAR methodology • Interactive segmentation • CAVIAR – flower recognition system • CAVIAR – face recognition system • Conclusions Jie Zou, Doclab ECSE, RPI

  21. Notation • Parametric boundary • Exact boundary • Foreground region or • Background region or • Radius vectorintersectsat , andat . Jie Zou, Doclab ECSE, RPI

  22. Training – Color Distributions Jie Zou, Doclab ECSE, RPI

  23. Training – Circle Parameter Distributions Jie Zou, Doclab ECSE, RPI

  24. Training – Deviation of Circular Model From Exact Boundary ß=5.52 Jie Zou, Doclab ECSE, RPI

  25. Automatic Segmentation – Circle Partition Use a circle to isolate a region, which contains mostly flower colors. Jie Zou, Doclab ECSE, RPI

  26. Automatic Segmentation – Generate Boundary Likelihood Map Distance to the circle Magnitude of color gradient Boundary pixels should be close to the circle, and have high color gradient. Jie Zou, Doclab ECSE, RPI

  27. Automatic Segmentation – Deformation on BLM • Circle center = a foreground seed, and four corner pixels = background seeds • Foreground and background regions compete with each other to expand. • Eventually, converge to the watershed of the seed pixels on the BLM. Jie Zou, Doclab ECSE, RPI

  28. Advantage of BLM over Gradient Map Jie Zou, Doclab ECSE, RPI

  29. Examples of the Result of Automatic Segmentation Jie Zou, Doclab ECSE, RPI

  30. Interactive Correction (Video) Jie Zou, Doclab ECSE, RPI

  31. Segment Flower Pictures with Interactive Correction • 1078 flowers from 113 speciesBorjan Gagoski and Adam Callahan • 5.7 seed pixels, 15.2 seconds per picture Greenie Cheng Jie Zou, Doclab ECSE, RPI

  32. Agenda • Motivation • Related research • Interactive segmentation • CAVIAR methodology • CAVIAR – flower recognition system • CAVIAR – face recognition system • Conclusions Jie Zou, Doclab ECSE, RPI

  33. Flower Database • 320 by 240 resolution • Highly variable illumination, and complex background • 216 samples from 29 classes for development • 612 samples from 102 classes for evaluation Jie Zou, Doclab ECSE, RPI

  34. Rose Curve Model • Parametric curve withsix parameters. • Flowers are composed of petals, which havecircular symmetry. • When n=0, rose curvereduces to circle. Jie Zou, Doclab ECSE, RPI

  35. Classification Features number of petals. the ratio of outer to inner radius. first three order moments of the hue andsaturation histograms Jie Zou, Doclab ECSE, RPI

  36. Adjust inner circle radius by dragging this control point. Move the rose curve by dragging this control point. Change the petal number using this ComboBox. Adjust outer circle radius by dragging this control point. CAVIAR-Flower GUI Jie Zou, Doclab ECSE, RPI

  37. CAVIAR-Flower (Video) Jie Zou, Doclab ECSE, RPI

  38. Evaluation • CAVIAR compared to human-alone and machine-alone • Machine learning • Decision directed approximation • Finite state machine calibration Jie Zou, Doclab ECSE, RPI

  39. Experimental Protocol Thanks to Borjan Gagoski Jie Zou, Doclab ECSE, RPI

  40. CAVIAR Compared to Human-Alone and Machine-Alone • Significantly reduce the recognition time compared to human-alone • Significantly increase the accuracy compared to machine-alone Jie Zou, Doclab ECSE, RPI

  41. Accuracy of Initial Automatic Recognition T2 – 5 labeled T3 – 1 labeled T4 – 1 labeled + 2 pseudo T5 – 1 labeled + 4 pseudo Jie Zou, Doclab ECSE, RPI

  42. Rank Order after Initial Automatic Recognition T2 – 5 labeled T3 – 1 labeled T4 – 1 labeled + 2 pseudo T5 – 1 labeled + 4 pseudo Jie Zou, Doclab ECSE, RPI

  43. Time of Interactive Recognition T2 – 5 labeled T3 – 1 labeled T4 – 1 labeled + 2 pseudo T5 – 1 labeled + 4 pseudo Jie Zou, Doclab ECSE, RPI

  44. Accuracy of Interactive Recognition T2 – 5 labeled T3 – 1 labeled T4 – 1 labeled + 2 pseudo T5 – 1 labeled + 4 pseudo Jie Zou, Doclab ECSE, RPI

  45. Observations about Machine Learning • Initialized with a single training samples per class. • Self-learning: user classified pseudo-labeled samples improve the performance. • Performance of T5 is close to T2, suggesting that instead of initializing with many training samples, we can trust the system’s self learning. Jie Zou, Doclab ECSE, RPI

  46. Calibration of Finite State Machine • 52% samples are immediately confirmed. • 90% samples are identified by 3 adjustments. • The probability of success on each adjustment is just over one half. Jie Zou, Doclab ECSE, RPI

  47. Summary of CAVIAR-Flower • Parameterized rose curve to model the flowers. • Display the rose curve and let user adjust it if necessary. • The evaluation of the system shows advantages of CAVIAR system. Jie Zou, Doclab ECSE, RPI

  48. Agenda • Motivation • Related research • Interactive segmentation • CAVIAR methodology • CAVIAR – flower recognition system • CAVIAR – face recognition system • Conclusions Jie Zou, Doclab ECSE, RPI

  49. CAVIAR - Face • Not to implement a state-of-the-art face recognition system. • To demonstrate the wider applicability of CAVIAR methodology. • 400 FERET pictures of 200 subjects (“ba” and “bk” series) Jie Zou, Doclab ECSE, RPI

  50. Face model and features • Face model contains only two pupils. • An automatic facial feature detection program locates the other 26 points. Thanks to Yan Tong, Zhiwei Zhu, and Dr. Qiang Ji. Jie Zou, Doclab ECSE, RPI

More Related