1 / 18

Human Computer Interaction Query by Sketch

Human Computer Interaction Query by Sketch. Chi-Ren Shyu. Department of Computer Engineering and Computer Science University of Missouri-Columbia Columbia, MO 65211, U.S.A. Query by Sketch -- an HCI topic?.

paul
Download Presentation

Human Computer Interaction Query by Sketch

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Human Computer InteractionQuery by Sketch Chi-Ren Shyu Department of Computer Engineering and Computer Science University of Missouri-Columbia Columbia, MO 65211, U.S.A.

  2. Query by Sketch -- an HCI topic? • Human side: query behavior, retrieval result evaluation, training of semantics and database indexing. • Computer side: query processing from human’s sketches, feature extraction from sketch objects, database search mechanisms, etc.

  3. Some Applications and Systems • Photo (natural scene images) IBM QBIC http://wwwqbic.almaden.ibm.com/ • GIS – Egenhofer at U. of Maine Spatial Query by sketch http://www.spatial.maine.edu/~max/RL.html • Medical Image – University of Missouri – Colmbia WebHIQS http://diglib1.cecs.missouri.edu:3243/FinalProj/index1.html (experimental website)

  4. QBS in Diagnostic Image Databases • Diagnoses often involve findings related to spatial relationship among lesions and landmarks. • Pathologies could be detected based on many visual patterns that are “sketchable”, such as shapes of tumors, distribution of nodules, ect. Query: “Retrieve database images that have similar lesion-landmark location with the sketched query (image)”

  5. Query methods in Diagnostic Image DBS • Sketch on a blank sheet – users are provided with a GUI for drawing organs, landmarks, and lesions. • Sketch on an organ template – users are provided with a template of an organ and a GUI for drawing landmarks and lesions. • Sketch on an existing image – users are provided with a real diagnostic image and a GUI for drawing landmarks and lesions.

  6. Research opportunity – Feature Extraction from Sketches • Shapes from sketch objects, texture from sketched regions, gray scale from sketched vicinity, etc. Domain expertise will help to design computer vision algorithms for this purpose. • Forming a feature vector for image retrieval. • Searching a high-dimensional database for retrieving the most similar sketches?

  7. Research opportunity – Spatial Modeling from Sketches Anatomical landmarks in HRCT lung: • Lung regions (Automatic Extraction) • Fissures (Human in the loop) • Lobe partitions (HIL/AE) Lesions vs. landmarks: • Interior • Adjacent • Across

  8. Research opportunity – Spatial Modeling from Sketches - 2 Query lesion Sketched by the user Why these three lesions should be retrieved?

  9. Degree of Truth for (inner-Adjacent, To-the-right) (0.34, 0.00) (0.15, 0.00) (0.55, 0.31) (1.00, 0.82) (1.00, 0.96) (0.83, 0.84) (0.21, 0.99) (0.00, 1.00) Spatial Modeling from Sketches - 3 What about multiple landmarks with multiple lesions?

  10. Research opportunity – Semantic Query from Sketches Semantic or so-called ontology is very subjective and user-oriented. Not good for “general purposed” database retrieval. GREAT for Sketches! The hope is: Sketches might be able to bring more desirable retrieval results from user’s aspects.

  11. Semantic Query from Sketches - 2

  12. Semantic Query from Sketches - 3 Sketch bronchial structures on an organ template and fill gray scales to different part of the object, such as bronchial Walls and lumens.

  13. Semantic Query from Sketches - 4 Users sketch on many training images and form a semantic tree structure for future retrieval.

  14. Semantic Query from Sketches - 5 For fuzzy gangs, this looks familiar. We can form membership functions for the semantic terms by using user’s Sketches.

  15. Semantic Query from Sketches - 6 Ranking database images by semantics

  16. Semantic Query from Sketches - 6

  17. Research opportunity – Query Languages for Sketches SELECT I.iid FROM LungImageDatabase I WHERE SIM( SKETCH(Qid,GUI), I.FE(iid)) < threshold ORDER BY SIM; A possible SQL extension for QBS:

  18. Summary Query by sketch is not new, however, it has high potential for many applications, especially in the areas of diagnostic image database, GIS, criminal investigation, etc. In the CECS department, we have many existing research projects that are highly correlated to the concept of QBS. We should have some sort of synergistic efforts to have some interesting projects in the near future.

More Related