Introduction to Imaging and Structural Informatics

INFO-I530 (Foundation of Health Informatics) Medical Imaging Systems Lecture #11 (extra)

Lecture in a Nutshell • Imaging and Structural Informatics • Introduction • Basic Concepts • Structural Imaging • Two-Dimensional Image Processing • Three-Dimensional Image Processing • Functional Imaging • Imaging Systems in Radiology • Introduction • Basic Concepts and Issues • Current Status

Imaging and Structural Informatics

Introduction • Although there are many imaging modalities, images of all types are increasingly being converted to or initially acquired in digital form. • Digital images have become a core data type that must be considered in many biomedical informatics applications. • We can develop general solutions that can be applied to all images, regardless of the source. • The common tasks addressed by imaging informatics can be classified as: • Image generation: process of generating the images and converting them to digital form if they are not intrinsically digital. • Image manipulation: uses preprocessing and post-processing methods to enhance, visualize, or analyze the images. • Image management: includes methods for storing, transmitting, displaying, retrieving, and organizing images. • Image integration: is the combination of images with other information needed for interpretation, management, and other tasks.

Basic Concepts • Digital Images • A digital image typically is represented in a computer by a two-dimensional array of numbers (a bit map); each element is called a pixel. • If we consider the image of a volume, then a three-dimensional array of numbers is required; each element is called a voxel. • We can store any image in a computer in this manner, either by converting it from an analog to a digital representation or by generating it directly in digital form. • Imaging Parameters • Spatial resolution: is related to the sharpness of the image; it is a measure of how well the imaging modality can distinguish points on the object that are close together (number of pixels) • Contrast resolution: is a measure of the ability to distinguish small differences in intensity, which in turn are related to differences in measurable parameters such as X-ray attenuation (bits per pixel) • Temporal resolution: is a measure of the time needed to create an image. An imaging procedure can be a real-time application, if it can generate images concurrent with the physical process (pixel per second)

Structural Imaging • Energy Source • Light: photographs, either of gross anatomic structures or, if a microscope was used, of histologic specimens. • X-Rays: were first discovered in 1895 by Wilhelm Conrad Roentgen. Film-based radiography is being replaced by computed and digital radiography. X-ray can be generated in real time (fluoroscopy). Their limitations are their relatively poor contrast resolution, their use of ionizing radiation, and their inability to depict physiologic function. Radioactive isotope can also be used (nuclear-medicine imaging). • Ultrasound: or ultrasonography uses pulses of high-frequency sound waves rather than ionizing radiation to image body structures. The system constructs two-dimensional images by displaying the echoes from pulses of multiple adjacent one-dimensional paths. • Nuclear Magnetic Resonance: Creation of images from NMR signals known as MRI had to await the development of computer-based reconstruction techniques, which represent one of the most spectacular applications of computers in medicine.

Structural Imaging cont. • Reconstruction Methods • Contrast radiography: the use of radiopaque contrast material to highlight the areas of interest (e.g., stomach, colon, urinary tract) was used as early as 1902 to address this problem. (angiography 1923) • In the early 1970s, Hounsfield led a team at the London-based EMI Corporation, which developed the first commercially viable CTscanner. Cormack and Hounsfield were awarded the 1979 Nobel Prize in Medicine. • Instead of depicting a directly measurable parameter (the absorption of X-ray beams as they pass through the body), computed tomography (CT) mathematically reconstructs an image from X-ray-attenuation values that have been measured from multiple angles. • After the invention of the CT scanner, this basic method of reconstruction from projections was applied to other energy sources, including magnetism (MRI), ultrasound (ultrasound-transmission tomography), and variants of nuclear-medicine imaging called positron-emission tomography (PET) and single-photon-emission computed tomography (SPECT).

Structural Imaging cont. • Higher Dimensionality • Reconstruction modalities—such as CT, PET, and MRI—all are either inherently three-dimensional or can be made three-dimensional by acquisition of a series of closely spaced parallel slices. • Ultrasound images, on the other hand, cannot be acquired as parallel slices because sound does not pass through bone or air. • Contrast Agents • Radiologic contrast agents and reconstruction techniques • Histologic staining agents such as hematoxylin and eosin (H&E) • Magnetic contrast agents such as gadolinium for MR images • Molecular Imaging: Advances in molecular biology have led to the ability to design contrast agents that are highly specific for individual molecules. Tagged molecules have been used for several years in vitro by such techniques as immunocytochemistry (binding of tagged antibodies to antigen) and in situ hybridization (binding of tagged nucleotide sequences to DNA or RNA)

Two-Dimensional Image Processing • Basic Concepts • Images can be enhanced to permit human viewing, to show views not present in the original images, to flag suspicious areas for closer examination by the clinician, to quantify the size and shape of an organ, and to prepare the images for integration with other information. • Most of these applications require one or more of the four basic image processing steps: global processing, segmentation, feature detection, and classification. • Global processing: involves computations on the entire image, withoutregard to specific local content. The purpose is to enhance an image for human visualization or for further analysis by the computer. • A simple but important example is gray-scale windowing of CT images. The CT scanner generates pixel values (Hounsfield numbers, or CT numbers) in the range of −1,000 to +3,000. Humans, however, cannot distinguish more than about 100 shades of gray. To appreciate the full precision available with a CT image, the operator can adjust the midpoint and the range of the displayed CT values.

Two-Dimensional Image Processing cont. • Segmentation: involves the extraction of regions of interest (ROIs) from the overall image. The ROIs usually correspond to anatomically meaningful structures, such as organs or parts of organs. The structures may be delineated by their borders, in which case edge-detection techniques (such as edge-following algorithms) are used, or by their composition on the image, in which case region-detection techniques (such as texture analysis) are used. • Because segmentation is difficult for a computer, it is often performed manually by a human operator or through a combination of automated and operatorinteractive approaches. It therefore remains a major bottleneck that prevents more widespread application of image-processing techniques. • Feature detection: is the process of extracting useful parameters from the segmented regions. • These parameters may themselves be informative—for example, the volume of the heart or the size of the fetus. • Classification: Extracted parameters also may be used as input into an automated classification procedure, which determines the type of object. • For example, small round regions on chest X-ray images might be classified as tumors, depending on such features as intensity, perimeter, and area.

Two-Dimensional Image Processing cont. • Examples • In general, routine techniques are available on the manufacturer’s workstations (e.g., an MR console or an ultrasound machine), whereas more advanced image-processing algorithms are available as software packages that run on independent workstations. The primary uses of two-dimensional image processing in the clinical environment are for: • Image enhancement:uses global processing to improve the appearance of the image either for human use or for subsequent processing by computer, such as CT windowing. Another technique is unsharp masking, in which a blurred image is subtracted from the original image to increase local contrast and to enhance the visibility of fine-detail (high-frequency) structures. • Histogram equalization: spreads the image gray levels throughout the visible range to maximize the visibility of those gray levels that are used frequently. Temporal subtraction subtracts a reference image from later images that are registered to the first. A common use of temporal subtraction is digital-subtraction angiography (DSA) in which a background image is subtracted from an image taken after the injection of contrast material.

Two-Dimensional Image Processing cont. • Screening:uses global processing, segmentation, feature detection, and classification to determine whether an image should be flagged for careful review by a radiologist or pathologist. If the number of flagged images is small compared with the total number of images, then automated screening procedures can be economically viable. Screening techniques have been applied successfully to mammography images for identifying mass lesions and clusters of microcalcifications, to chest X-rays for small cancerous nodules, and to Papanicolaou (Pap) smears for cancerous or precancerous cells. • Quantitation: (automatic measurements) uses global processing and segmentation to characterize meaningful regions of interest. For example, heart size, shape, and motion are subtle indicators of heart function and of the response of the heart to therapy. Similarly, fetal head size and femur length, as measured on ultrasound images, are valuable indicators of fetal well-being.

Three-Dimensional Image Processing The basic two-dimensional image-processing operations of global processing, segmentation, feature detection, and classification generalize to higher dimensions, and are usually part of any image-processing application. However, three-dimensional and higher dimensionality images give rise to additional informatics issues which include: • Registration: In order to accurately depict anatomy, the voxels must be accurately registered (or located) in the three-dimensional volume (voxel registration), and separately acquired image volumes from the same subject must be registered with each other (volume registration). • Voxel Registration: two-dimensional images can be converted to three-dimensional volumes by acquiring a set of closely spaced parallel sections through a tissue or whole specimen. In this case the problem is how to align the sections with each other. (fiducial markers, thick serials, stereo-matching)

Three-Dimensional Image Processing cont. • Volume Registration: A related problem to that of aligning individual sections is the problem of aligning separate image volumes from the same subject, i.e., intrasubject alignment. Because different image modalities provide complementary information, it is common to acquire more than one kind of image volume on the same individual such as a combination of PET and MRI. (intensity-based optimization, templates of the same modalities, landmark-based) • Spatial Representation of Anatomy: The reconstructed and registered three-dimensional image volumes can be visualized directly using volume rendering techniques. Extraction of spatial representations of anatomy, in the form of three-dimensional surfaces or volume regions, is accomplished by a three-dimensional generalization of the segmentation techniques. Popular segmentation and reconstruction techniques include: • Reconstruction from Serial Sections: The classic approach to extracting anatomy is to manually or semi-automatically trace the contours of structures of interest on each of a series of aligned image slices, then to tile a surface over the contours (triangular made surface mesh). Neither fully automatic contour tracing nor fully automatic tiling has been satisfactorily demonstrated in the general case.

Three-Dimensional Image Processing cont. • Region-Based Segmentation (segmentation at the macroscopic level): voxels are grouped into contiguous regions based on characteristics such as intensity ranges and similarity to neighboring voxels. A common initial approach to region-based segmentation is to first classify voxels into a small number of tissue classes such as gray matter, white matter, cerebrospinal fluid, and background, then to use these classifications as a basis for further segmentation. Another region-based approach is called region growing, in which regions are grown from seed voxels manually or automatically placed within candidate regions. • Edge-based segmentation: is the complement to region-based segmentation; intensity gradients are used to search for and link organ boundaries. In the two-dimensional case, contour-following methods connects adjacent points on the boundary. In the three-dimensional case, isosurface-following or marching-cubes methods connect border voxels in a region into a three-dimensional surface mesh.

Three-Dimensional Image Processing cont. • Model- and Knowledge-Based Segmentation: The most popular current method for medical image segmentation is the use of deformable models (snakes method). In the two-dimensional case the deformable model is a contour, often represented as a simple set of linear segments or a spline, which is initialized to approximate the contour on the image. The contour is then deformed according to a cost function that includes both intrinsic terms limiting how much the contour can distort, and extrinsic terms that reward closeness to image borders. In the three-dimensional case, a three-dimensional surface (often a triangular mesh) is deformed in a similar manner. An advantage of deformable models is that the cost function can include knowledge of the expected anatomy of the brain. An alternative knowledge-based approach explicitly records shape information in a geometric constraint network (GCN), which encodes local shape variation based on a training set. The shape constraints define search regions on the image in which to search for edges. • Combined Methods: Most brain segmentation packages use a combination of methods in a sequential pipeline.

Functional Imaging Many imaging techniques not only show the structure of the body but also the function. For imaging purposes, function can be inferred by observing changes of structure over time. A particularly profound application of functional imaging is the understanding of cognitive activity in the brain. Functional brain-imaging modalities can be classified as image-based or non-image-based. In both cases the functional data must be mapped to the individual subject’s anatomy, where the anatomy is extracted from structural images. Remote visualization of integrated structural and functional brain data mapped onto a single patient’s brain.

Imaging Systems in Radiology

Introduction • In this section methods for managing and integrating images, focusing on how images are acquired from imaging equipment, stored, transmitted, presented for interpretation will be discussed. • Radiology departments in some institutions are thus referred to alternatively as Departments of Medical Imaging. • Medical center-based Radiology: Many Radiology departments are becoming highly distributed enterprises, with acquisition sites in intensive care unit areas, regular patient floors, emergency departments and etc. • Distributed medical center-based Radiology: Increasingly, due to high-speed network availability, interpretation can be done at central sites, or according to different methods of organization, since image acquisition and interpretation can be effectively decoupled.

Basic Concepts • Roles for Imaging in Biomedicine • Detection and diagnosis: focuses on identifying the presence of an abnormality, but in the case in which the findings are not sufficiently specific to be characteristic of a particular disease, other methods must be used for actual diagnosis (e.g. mammography). • Assessment and Planning: imaging is often used to assess a patient’s health status in terms of progression of a disease process (such as determination of tumor stage), response to treatment, and estimation of prognosis. • Guidance of Procedures: Images can provide real-time guidance when virtual-reality methods are used to superimpose a surgeon’s visual perspective on the appropriate image view in the projection that demonstrates the abnormality. • Communication: Communicating digital images is essential to enable remote viewing, interpretation, and consultation, as in techniques such as teleradiology, telepathology, and teledermatology, collectively referred to as telemedicine. • Education and Training: medical diagnosis and treatment depends on imaging and on the skills needed to interpret such images. • Research: e.g. The quantitative study of morphometrics, or growth and development, depends on the use of imaging methods.

Basic Concepts cont. • The Radiologic Process and Its Interaction • Diagnostic studies in the Radiology department are provided at the request of referring clinicians, who then use the information for decision-making. The radiologist provides the primary analysis and interpretation of the radiologic findings. • The radiologic process is characterized by seven kinds of tasks, each of which involves information exchange and which can be augmented and enhanced by information technology. The first five tasks occur in sequence, whereas the final two are ongoing and support the other five: • Assess clinical problem • Request/schedule exam • Perform exam • Analyze findings • Communicate results and recommendations • Assess quality and monitor performance • Educate, train and provide feedback

Basic Concepts cont. The radiologic process

Basic Concepts cont. • Image Management and Display • Picture-Archiving and Communication Systems (PACS) • Images are acquired by facilities specific to the various imaging modalities (such as CT, MRI, angiography, ultrasound, or nuclear medicine), and the facilities are operated largely by radiology (or imaging) technologists (denoted R.T.). Local printing of images on film or paper may be performed. • Image procedures are scheduled through a radiology information system (RIS), and patient-identification and schedule information is transmitted to the modality workstation through a DICOM RIS gateway. Remote imaging centers operate in the same way. • The images produced are transmitted through a DICOM gateway to a server (autorouter), which is responsible for sending the images where they are needed and for managing workflow. The autorouter validates the linkage of images to the appropriate study (by interaction with a validation server) and distributes them according to rules.

Basic Concepts cont. • Images can be viewed on interpretation workstations, both special purpose and generic, and manipulated by imaging professionals who use built-in workstation tools and invoke special processing functions through servers (e.g., for three-dimensional rendering, registration, and fusion of data from images obtained by two different modalities, for feature extraction, or for computer aided detection). • Multiple images from a particular examination need to be associated, and both prior and other associated studies and reports may need to be available. This linkage is accomplished by the validation server, which is able to query for and retrieve information from the RIS or hospital information system (HIS), as well as from the PACS archives. The validation server is responsible for coordinating the association of image and non-image information. • Referring physicians may access images at internal and Internet based workstations, typically through a Web interface. Access from a browser through a Web server permits the user to obtain study information, including reports and images, through the validation server, which interacts with the PACS archive and with the HIS and RIS.

Basic Concepts cont. Architecture of a typical picture-archiving and communication system (PACS)

Basic Concepts cont. • Storage Requirements • On-line digital archiving of image data for a busy radiology department requires vast amounts of storage. Image modalities differ substantially in their storage requirements, depending on the contrast and spatial resolution needed, the number of images or the size of the data sets, whether raw or processed data are stored, and whether data-compression techniques are used:

Basic Concepts cont. • Considering that a typical radiology department performs 250 examinations per day, and nominally assuming 10 megabytes per study, then, in an average day, approximately 2.5 gigabytes of data must be transmitted from the image-acquisition nodes to the image archive. Assuming 250 working days per year (ignoring weekends for simplicity), we estimate that the storage requirements per year for examination image data are on the order of 625 gigabytes. • Currently a two tier system is affordable by large magnetic disks of 500 gigabytes (0.5 terabytes) or more, sufficient to store approximately 1 year’s image data in a typical department; and a digital-tape library functioning for long-term storage. • Data compression: Lossless compression (RLE 2:1), Lossy compression (JPEG 20:1), Wavelet compression (80:1) • Another consideration in storage relates to speed of access. Thus, hierarchies of storage and algorithms for deciding where to place image data based on patterns of expected use and network traffic are required for smooth functioning of a PACS.

Basic Concepts cont. • Image Transmission • The principal media for image transmission and networking are broadband coaxial cable and fiber-optic cable. The network configuration (both WAN and LAN) and the capacity of each part must be planned in relation to considerations such as patterns of expected use and cost. • Standardization of Formats • TCP/IP is the dominant low-level protocol used in medical imaging. Transmission of data about a medical imaging procedure, including patient, examination, and image data, require higher level messaging formats. • Digital Imaging and Communications in Medicine (DICOM), has been adopted as a network communication protocol to a worldwide extend for both radiological and other medical images. • DICOM is intended to ensure that a wide variety of equipment can be interfaced with the network and that the data can be recognized and interpreted correctly by all the nodes on the network. • DICOM was developed by the American College of Radiology (ACR) and the National Equipment Manufacturers Association (NEMA).

Basic Concepts cont. • Display Capabilities • The design of image-viewing consoles that are suitable for interpretation of examinations by radiologists poses a host of technical and human engineering problems. • Mammography presents especially difficult challenges for soft-copy interpretation because mammography requires extremely high resolution for microcalcification detection. • An advantage for digital mammography is the ability to preprocess images with computer-aided detection (CAD) algorithms for highlighting potential nodules and microcalcifications. • The display of three-dimensional imaging data places even greater demands on consoles to calculate and redisplay oblique slices and rotated views rapidly. • Cost • Image management and PACS development have significant benefits for radiology departments in terms of reductions in film-library space and personnel time, as well as immediacy of access to images, although digitizing film based systems is not cost effective.

Basic Concepts cont. • Integration with Other Healthcare Information • Radiology Information Systems (RIS) • Specialized functionality—such as for image manipulation or interpretation, for education, or for decision support—as well as more comprehensive capabilities embodied in HISs have tended to develop in relative isolation over many years or even decades. • As a result, the software architectures of systems that implement these capabilities often are incompatible and frequently are inflexible with respect to ease of integration with one another. • Management of work flow in a radiology department is a complex activity that involves not only maintenance of the film library and digital archive but also scheduling of examinations, registration of patients, performance of examinations, review and analysis of studies by radiologists, creation of interpretations, transcription of dictated reports (or generation of structured reports directly by radiologists), distribution of radiology reports to referring physicians, and billing for services.

Basic Concepts cont. • RISs have been implemented either as standalone systems or as components of HISs. In either case, an RIS must be integrated with other information systems within an institution to allow reconciliation of patient data, to support examination scheduling and results reporting, and to facilitate patient billing. • Picture-archiving and communication system (PACS) image-management functions must be integrated with RISs and HISs. Because an RIS (or, in some cases, an HIS) keeps track of examinations and associates them with patients, and a PACS keeps track of images and associates them with examinations, the task is to provide coordination between the examination data on the two systems: • For example the path to the images can be stored directly with the examination record on the RIS (or HIS), or the examination data can be duplicated on the PACS, where pointers to the images for each examination are maintained. • Alternatively, the PACS can be augmented with patient-lookup and examination-lookup capabilities and the databases from an RIS or HIS duplicated on it. Whenever a user application submits a query about images, the query is sent to a PACS server; queries about other clinical information are sent to the RIS or HIS. • A Web-based front end could integrate with appropriate back-end services of HIS, RIS, and PACS to eliminate such duplication.

Basic Concepts cont. • Reporting Methodology • Dictation and transcription • Dictation and voice recognition • Dictation and voice recognition and transcription • Structured reporting using computers (e.g. obstetrical ultrasound) • Enterprise Integration • In an integrated delivery network (IDN) environment, clinical data and images typically are obtained from multiple sources (hospitals, offices, imaging centers) and distributed for interpretation (to the imaging specialist) and for review (to the requesting clinician and other specialists). • Multiway consultations can be carried out, with all parties concurrently or asynchronously viewing and perhaps annotating the images. Images from different sources may need to be fused or used in image-guided treatment. • Component based architecture: Middleware components are being developed that support access, processing, analysis, and composition of lower-level.

Basic Concepts cont. New architectures for imaging workstation applications (TF: Teaching file)

Current Status • Picture Archiving and Communication Systems • The completely filmless radiology department is still limited by the fact that film mammography has not yet been supplanted by digital modalities to a significant extent. • A popular misconception is that barriers to soft copy interpretation are removed by higher resolution display screens. If a 20-inch diagonal monitor can display 1,000 × 1,000 pixels, the question is whether it is inherently inferior to a 2,000 × 2,000 monitor or a 4,000 × 4,000 monitor, with a screen of the same 20-inch physical size. • If individual pixels are sufficiently close together on a 1,000 × 1,000 monitor so that the eye cannot resolve them at normal viewing distance (as is the case), however, then the additional resolution of the 2,0002 or 4,0002 monitors will not improve perceived image quality. • The question of optimal resolution for interpretation of most studies has not yet been satisfactorily answered.

Current Status cont. • Teleradiology • It is now common for radiology departments to enable their radiologists to provide coverage from home, especially for interpretations of CT and MRI examinations obtained during nights or weekends and review of interpretations made by radiology residents before finalization of reports. • In the United States, expansion of teleradiology is still limited by the requirement that interpreting radiologists be licensed in the state in which the images are acquired. • Client-based viewing applications are often augmented by Java to provide needed user-interface features for image manipulation. • At the back end of the systems is generally an image repository of images stored in a DICOM-compatible format, and often compressed by wavelet methods. • Patient-identification information permits linkage to an RIS or HIS, associating images with specific examinations and to reports.

Current Status cont. • Indexing and Image Retrieval • In a typical PACS system, images are archived on a file server. Identification data, usually obtained from the DICOM header, are stored in an associated relational database. • Specific images are retrieved based on information that was entered in the database at the same time that the images were acquired, either directly or via linkage to an RIS. For routine radiology this information is usually sufficient, because current and previous images are retrieved mostly by patient identifier and imaging modality. • For research or education, however, it is often desirable to retrieve images that “look like this one”—for example, to show other examples of a particular disease process or to perform a retrospective research study. The standard approach to this task, is to index images manually according to keywords. The DICOM standard has provisions to include these keywords and controlled vocabularies, such as the Systematized Nomenclature of Medicine (SNOMED), Neuro-names, and the Digital Anatomist Foundational Model (all part of UMLS), provide many of the needed source keywords.

Current Status cont. • PACS, HIS, and RIS Integration • The evolution of radiology at the Brigham and Women’s Hospital is probably representative of that at many institutions. The Brigham is part of an IDN, the Partners HealthCare System, serving eastern Massachusetts. • The Brigham has a large and successful HIS, known as BICS, and an IDXRad RIS. A homegrown PACS was developed for acquisition and archiving of the digital modalities, later replaced by a commercial system from General Electric Corp. • The HIS and RIS are interlinked for transfer of patient demographic, billing, and report information, and the RIS and PACS are interlinked to enable the RIS to keep track of images on the PACS associated with particular examinations. Ultrasound reports are generated by separate reporting modules and are integrated into the RIS through an HL7 interface. • Other semi-independent reporting modules can be linked to the RIS in similar fashion. • Voice-recognition systems for reporting have been introduced. Mostly, these are used to provide draft reports to transcription editors who revise them as necessary.

Current Status cont. • DICOM and HL7 • Integration among PACS, RIS, and HIS has been aided by the evolution of two standards, DICOM and HL7. • DICOMis a standard format for transmitting image information, including patient, exam, and study series information. It has been important in making it possible for different imaging modality devices and consoles to transmit images to common PACS archives, and enabling the images to be manipulated on common interpretation workstations, while maintaining the association between images and the series and exams of which they are part, for specific patients. • Patient demographic and exam ordering information, result reporting, and billing, on the other hand, have been the province of messaging standards developed by HL7. • Thus RISs and PACS must communicate using both DICOM and HL7 standards, in order to fully integrate the above information. • The HL7 and DICOM standards development communities, in fact, participate jointly in an Image Management Special Interest Group of HL7, to resolve areas of interface and overlap.

Summary • Imaging and Structural Informatics • Introduction • Basic Concepts • Structural Imaging • Two-Dimensional Image Processing • Three-Dimensional Image Processing • Functional Imaging • Imaging Systems in Radiology • Introduction • Basic Concepts and Issues • Current Status

Introduction to Imaging and Structural Informatics