1 / 26

IPAL – Image Perception, Access and Language UMI – CNRS, I2R A-STAR, NUS joint lab

IPAL – Image Perception, Access and Language UMI – CNRS, I2R A-STAR, NUS joint lab. Scientific axis MIIRAD M edical I mage I ndexing and R etrieval for A ssisted D iagnosis, medical research and teaching PI Singapore : Wee Kheng LEOW PI France : Daniel RACOCEANU.

urania
Download Presentation

IPAL – Image Perception, Access and Language UMI – CNRS, I2R A-STAR, NUS joint lab

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IPAL – Image Perception, Access and LanguageUMI – CNRS, I2R A-STAR, NUS joint lab Scientific axisMIIRAD Medical Image Indexing and Retrieval for Assisted Diagnosis, medical research and teaching PI Singapore : Wee Kheng LEOW PI France : Daniel RACOCEANU

  2. MIIRAD permanent participants • Wee Keng LEOW (IPAL), NUS, Singapore • Daniel RACOCEANU (IPAL), CNRS, France • Caroline LACOSTE (IPAL), CNRS, France • Joo-Hwee LIM (IPAL), I2R, Singapore • Jean-Pierre CHEVALLET (IPAL), CNRS, France • Xiong WEI (IPAL), I2R, Singapore + other SINGAPOUR & FRANCE - COLLABORATORS

  3. Medical Image Retrieval - Example

  4. Content-Based Image Retrieval (CBIR) and related areas Information retrieval Database system CBIR Pattern recognition and image processing

  5. Computer Aided Detection/Diagnosis Medical Education Medical Research MIIRAD – Core and Applications CBIR (similarity based access) PACS (archiving) DICOM (communication)

  6. Medical Image Retrieval What for ? • Medical assistance • Use of similar cases in order to assist medical detection/diagnosis, treatment, … • Difficult cases detection/diagnosis assistance • Treatment choice assistance • Education • Better understanding of diseases and phenomena using multiple similar examples • Better training by the recognition of similar symptoms – different origins cases • Updating and creating medical multimedia atlases • Research • Medical Image Data mining • New pathological trends, • Extract new global characteristics by type of population and geographic regions

  7. Brief state of the art in medical image retrieval

  8. About the medical images retrieval … • Common CBIR algorithms like: • IBM-QBIC (Query by Image Content) – Niblack et al., 1993 • CORE (Content-Based Retrieval Engine) – Wu et al., 1995 • Blobworld – Carson et al.,2002 give poor results when applied to medical images. • Two or three semantic layers only, seems to be insufficient to model medical knowledge for image retrieval

  9. About some specific medical images retrieval approaches … • Korn et al. (1998) - a system for fast and effective retrieval of tumor shapes in mammogram x-rays • Restrictions on image modality / anatomy / pathology • ASSERT (Automatic Search and Selection Engine with Retrieval Tools) – (Shyu et al., 1999) • Operate only on high-resolution CT of the lung • A physician delineates the region bearing a pathology • High data entry cost, which prohibits its application for clinical routine

  10. ASSERT – a specific approach(Shyu, C, et al., 1999)

  11. medGIFT (Univ. of Geneva) • Project for content-based search in medical image databases • Goals of the project • Better management of visual medical data (retrieval) • Visual Knowledge Management • Textual and visual data • Diagnostic aid • Specialized retrieval (lung CTs, fractures, dermatologic images) • Access to PACS data • In the short term: • Research, Teaching

  12. Knowledge Layer (KL) Gen Concept Type Abstraction Hierarchy (TAH) Contours Spatial relationship characteristics Temporal sequences Gen Concept Concept Concept Concept Hierarchical TAH Classification Using Context and user sensitive attributes (specified by the user) SE Schema Layer (SL) Database schema Objects = VisualEntities (VEs) Spatial relationship among objects Stream entities (SE) VE VE VE VE VE VE VE VE Schema extraction Feature and Content Layer (FCL) Image Features Contours Spatial relationship characteristics Temporal sequences Segmentation Manually Semi automatically (active contours, …) Automatically (depending of the contrast) Spatial Feature Computation Shape modeling Spatial relationship modeling Visual attribute Raw Data Layer (RDL) Raw Medical Image Conventional attribute Encoding techniques Compression,,Encription, Special formating Knowledge-Based Image Retrieval Systembrain lesions - (Chuet al., 1998) Medical Image

  13. IRMA – Image Retrieval in Medical Applications(Lehmann et al., 2004) Query results Knowledge Layer Retrieval Query Object Layer Blob-trees Identification Scheme Layer Indexing Feature Layer Feature selection Feature vector Feature extraction RST - parameters Registration Registered Data Layer Categories Categorization Raw Data Layer (RDL) Medical Image

  14. Example of IRMA interface

  15. Remark : • The knowledge layer(s) generation needs an important quantity of information and knowledge from the physician, and strongly related to the medical image category. The cost and the time necessary to build an efficient system seems to be prohibitive • A considerable improvement could be added by the use of a “standard” medical knowledge like MeSH, UMLS, ...

  16. Actual challenges in CBIR for biomedical applications • What ? • Improve the retrieval systems efficiency/effectiveness (actually ~ 20-30 %) • Medical image access based on visual similarity • Intelligent analysis and mining extraction from large MI database • Context sensitive query and navigation • 3D … • How ? • By reducing the “semantic gap” using the medical knowledge • By using existing structured and updated medical knowledge, in order to facilitate the task of the physician (in loop) • By increasing the efficiency of the fusion between medical text and images • Difficulties ? • The complexity of the medical knowledge and medical queries • Medical knowledge/ontology strongly related to the specific medical imagecategory (modality, anatomy, pathology) • Difficulties to extract concepts from a medical image

  17. ONCO-MEDIAONtology COntext related MEDical Image Access ICT-ASIA 2006Project proposal

  18. QP Query processing ( Retrieval ) trend Mining Extraction High-Level Fusion Knowledge (UMLS) Semantic Extraction/Interpretation Semantic (Ontology) Stxt Simg … Simg Svid Structural Extraction/Interpretation Ltxt Limg … Structure (Objects) Limg Lvid Primitive Extraction Features (Segmentation) PEtxt PEimg … PEimg PEvid Categorization Text Image modality1 … Image modalityn Stream Local fusion Global approach User query INDEXING

  19. Document semantic analysis CUI1 Documenti MEDICAL ONTOLOGY MEDICAL ONTOLOGY CUIn CUI2 TEXT-BASED CONCEPTS MedicalDocument i - Indexes DICOM Medical Images ij (associated to Case i) DICOM_CUIk Medical Document i Case i DICOM Headerij ? DICOM_CUIm DICOM headers semantic analysis DICOM_CUIk CUI1 DICOM_CUIk DICOM Header -BASED CONCEPTS DICOM Header ij Indexes DICOM headers + Medical images Image visual analysis Complete Casei VB-Gen CUI1 CUIn MedImgij VB –Gen CUIk CUI2 VB-Gen CUIk VB –Gen CUIg ? VB-Gen CUIg TEXTUAL and VISUAL BASED FUZEDGENERALCONCEPTS of the Medical Case i IMAGE-BASED GENERAL CONCEPTS Medical image ij General Signature ONCO-MEDIAThe semantic indexing 1/2 TEXT-BASED CONCEPTS EXTRACTION Verification Fusion ??? DICOM HEADER TEXT-BASED CONCEPTS EXTRACTION Verification Fusion ??? GENERAL MEDICAL IMAGE VISUAL ANALYSIS GENERAL SEMANTIC INDEXING

  20. GENERAL SEMANTIC INDEXING ? CATEGORY Modality (MRI, X-Ray, CT, …) Anatomic part Pathology Template associated to the Modality, Anatomy, Pathology ? TB-CUI1 VB-Spec CUIk Complete Casei + Associate Medical Imagesij ? TB-CUIn VB-Gen CUIp Medical Images ij TB - CUI2 VB-Spec CUI1 MedImgij TB-CUIm COMPLETE TEXTUAL and VISUAL BASED FUZEDCONCEPTS of the Medical Case i VB-Spec CUIp VB-Spec CUIk IMAGE-BASED SPECIFIC CONCEPTS Medical image ij - Specific Signature ONCO-MEDIAThe semantic indexing 2/2 COMPLETE SEMANTIC INDEXING SPECIALIZED MEDICAL IMAGE VISUAL ANALYSIS

  21. Text “Find all the cases in which a tumor decrease in size for less than three month post treatment, then resumed a growth pattern after that period” MEDICAL ONTOLOGY Text + medical image “Find images with large-sized frontal lobes brain tumors for patients approximately 35 years old” + TEXT QUERY CONCEPTS EXTRACTION Medical image CUI1 Text query GENRAL AND SPECIALIZED QUERY MEDICAL IMAGE VISUAL ANALYSIS CUIn VB-Gen CUI1 CUI2 ImageiQuery QUERY TEXT-BASED CONCEPTS Textual query i - Indexes IMAGE-BASED ONTOLOGY VB-Spec CUIp VB-Spec CUIk QUERY IMAGE-BASED CONCEPTS Medical image ij - Specific Signature ONCO-MEDIAThe semantic retrieval 1/2 QUERY ? Verification Fusion ???

  22. TEXT-IMAGE SIMILARITY COMPARISON TB-CUI1 VB-Spec CUIk Query RETRIEVAL TB-CUIn VB-Gen CUIp TB - CUI2 TB-CUIm COMPLETE TEXTUAL and VISUAL BASED FUZED QUERY CONCEPTS ONCO-MEDIAThe semantic retrieval 2/2 COMPLETE SEMANTIC QUERY INDEXING

  23. Proposed modular management • WP1 : Global ONCO-MEDIA coordination • Coordinate a global coherent action • Inter-WP tasks management • WP2 : Medical textual documents and DICOM medical image text header semantic extraction / indexing using UMLS medical ontology • WP3 : Medical image based semantic extraction / indexing • 3.1. Generic approach based on UMLS medical ontology • 3.2. Specific category-dependent (modality, anatomy, pathology) approach • Lung CTs, Fractures, Dermatologic Images, Coronary Extraction, … (to be decided, according to the available data) • Need for a physician strongly involved • WP4 : Medical Text and Image fusion techniques. Similarity search approaches • WP5 : Prototype platform specification and development. Test on CLEF Medical Image benchmark and on a given hospital medical image databases (Singapore or/and Geneva Hospital)

  24. Our partners in this project • Dr. Henning Müller – Medical Informatics Service, Geneva University, HUG – Hopitaux Universitaires de Genève,Switzerland • Pr. Tianzi Jiang – Medical Imaging and Computing, National  Laboratory of Pattern Recognition, Institute of  Automation, Academy of Sciences – Beijing, China • MIIRAD/IPAL team (I2R A-STAR, CNRS France, NUS) • Pr. Gerard Finet, Dr. Isabelle Magnin – CREATIS, Lyon, France • Pr. Heng-Shuen Chen, Dept. of Medical Informatics, National Taiwan University Hospital, National Taiwan University, Taipei, Taiwan – to be confirmed • …

  25. References (Carson et al., 1999) C. Carson, M. Thomas, S. Belongie, J. M. Hellerstein, J. Malik, Blobworld: A system for region{based image indexing and retrieval, in: D. P. Huijsmans, A. W. M. Smeulders (Eds.), Third International Conference On Visual Information Systems (VISUAL'99), no. 1614 in Lecture Notes in Computer Science, Springer{Verlag, Amsterdam, The Netherlands, 1999, pp. 509-516. (Chu et al, 1998) W.W. Chu, F.C. Alfonso, and K.T. Ricky. Knowledge-based image retrieval with spatial and temporal constructs. IEEE Transactions on Knowledge and Data Engineering, vol. 10, no. 6, pp.872-888, 1998. (Lehmann et al, 2004) T. M. Lehmann, M. O. Güld, C. Thies, B. Fischer, K. Spitzer, D. Keysers, H. Ney, M. Kohnen, H. Schubert, and B. B. Wein, Content-based image retrieval in medical application,Methods of Information in Medicine, vol. 43, no. 4, pp. 354-361, 2004. (Niblack et al., 1993) W. Niblack, R. Barber, W. Equitz, M. D. Flickner, E. H. Glasman, D. Petkovic, P. Yanker, C. Faloutsos, G. Taubin, QBIC project: querying images by content, using color, texture, and shape, in: W. Niblack (Ed.), Storage and Retrieval for Image and Video Databases, Vol. 1908 of SPIE Proceedings, 1993, pp. 173{187. (Muller et al, 2004)H. Muller, N. Michoux, D. Bandon, and A. Geissbuhler. A review of content-based image retrieval systems in medical applications - clinical benefits and future directions. International Journal of Medical Informatics, vol. 73, no. 1, pp. 1-23, 2004. (Shyu et al., 1999) C.-R. Shyu, C. E. Brodley, A. C. Kak, A. Kosaka, A. M. Aisen, L. S. Broderick, ASSERT: A physician-in-the-loop content-based retrieval system for HRCT image databases, Computer Vision and Image Understanding (special issue on content-based access for image and video libraries) 75 (1/2) (1999) 111{132.

  26. Thanks !

More Related