Intensity-based deformable registration of 2D fluoroscopic X-ray images to a 3D CT model

1. Intensity-based deformable registration of 2D fluoroscopic X-ray images to a 3D CT model Aviv Hurvitz Advisor: Prof. Leo Joskowicz

2. Navigation in orthopedic surgery Applications • Determine position of surgical tools relative to anatomy • Position surgical robots • Match pre-operative model to anatomy [hss.edu] [medtronic.com]

3. Navigation in orthopedic surgery Related registration methods • Registration to fiducials • Implanted fiducials • On skin • Contact based registration (Point-cloud to surface registration) • Registration of fluoroscopic X-ray to CT

4. Registration of fluoroscopic X-ray to CT [LaRose 2001; Tomazevic 2003; Knaan 2003; Livyatan 2003; and others] Preoperatively: • Acquire CT of anatomy of interest Intraoperatively: • Fix a tracking marker to the bone • Acquire 2-5 fluoroscopic X-ray images from various camera poses • For each image, record: • T(trackercamera) - camera position at acquisition time • T(trackerbone) - bone marker position at acquisition time • Obtain an initial transformation estimate T(boneCT) • Estimate transformation T(boneCT) with the algorithm we describe next Tracker updates T(trackerbone) continuously, enabling real-time navigation.

5. Registration of fluoroscopic X-ray to CT Input: • For 2-5 X-ray images: • T(trackercamera) at acquisition time • T(trackerbone) at acquisition time • Initial T(boneCT) estimate Repeat until convergence: • For each camera position: • Define a virtual camera positioned relative to CT • Create DRR • Rate similarity of DRRs to X-ray images by a similarity metric • Determine the next T(boneCT) by an optimization algorithm

6. DRR creation [Knaan et al., 03]

7. Is it possible to do the registration without a CT? Advantages: • Save time before operation • Save costs • Decrease radiation exposure Disadvantages: • A patient-specific CT provides more information

8. Is it possible to create a patient-specific model without a CT? Approach • Use prior information – everyone’s bones are similar • Deduce exact shapes of bones from the fluoroscopic images In absence of CT, need to search for both bone pose (6 d.o.f.) and the patient-specific bone shape

9. Image analysis-by-synthesis Define a model of the bone which is parameterized by pose and by shape. Repeat until convergence: • Simulate X-ray imaging process to create DRRs • Rate similarity of DRRs to fluoroscopic images by similarity metric • Modify pose and shape parameters by optimization algorithm When the DRRs match the fluoroscopic images • We found the pose and the shape of the bone • We have registration: a transformation from every point in the model to its corresponding point in the patient anatomy. • We can map data from the model to the patient anatomy. E.g., map the surface of the bone.

10. Active Appearance Model (AAM) [Cootes et al., 98; Matthews and Baker, 04] • Model shape variations and appearance variations compactly (with few parameters) • Can be trained from a dataset of samples • A generative model – can generate new images which are similar to the training images • Image analysis-by-synthesis [Edwards, Taylor and Cootes, 98]

11. Shape • Manually label landmark points on training images • Define a triangular mesh between the landmarks [Matthews and Baker, 04] • Each shape is represented by a vector of landmark coordinates

12. Procrustes analysis - align meshes to cancel global variations in translation, rotation and scale • PCA on shape vectors

13. Appearance • Warp each training image to the base mesh so. This creates a set of shape-normalized images. • Represent pixels inside the base mesh so as vectors • PCA on shape-normalized image vectors:

14. The resulting model [Matthews and Baker, 04]

15. Instantiating the model • Input: • Shape parameters • Appearance parameters • Computation: • Instance’s shape-normalized appearance • Instance’s shape mesh

16. Instantiating the model [Matthews and Baker, 04]

17. Fitting AAMs to images • Input: • Image I(x) • AAM • Goal: find shape p and appearance λ that yield an AAM image similar to I(x). • Minimize SSD over λ and p: Sum differences over all pixels x in base mesh

18. The optimization is nonlinear in p and linear in λ. • The parameters are found by iterative gradient descent. • Standard gradient descent is slow because it requires calculating the partial derivatives at every iteration. • There are tricks to estimate gradient quickly [Cootes, 98] , or use a pre-computed gradient.[Matthews and Baker, 04]. • After improvements, algorithm is fast enough for real-time face tracking

19. Using AAMs for 2D-3D registration • The training dataset is CTs (3D images). ROIs of femoral head approximately 100x100x100 pixels. • The shapes are defined by tetrahedral meshes (approx. 3,000 nodes and 20,000 tetras). • It is too difficult to find landmarks manually.

20. Finding landmarks automatically • Select one CT arbitrarily as a template. • Define a tetrahedral mesh on the template. • Perform 3D-3D deformable registration of each CT to the template. Defines a transformation Ti:TemplateCTi • Apply transformation to template mesh transfer mesh to the corresponding landmarks in each CT.

21. Finding landmarks automatically • 3D-3D registration performed with Elastix software (an ITK wrapper) [Klein and Staring, 07] • Registration procedure • Preparation: match histogram to template • Rigid registration: • Initialize by aligning centerpoints of ROIs • Metric: Normalized Cross Correlation • Optimizer: Standard gradient descent • Deformable registration • Transform: 3D B-spline. (Grid spacing 8x8x8 voxels) • Metric: Sum of squared differences (SSD) • Optimizer: Conjugate gradient • Use image pyramids (3 levels) • Unsupervised • Each registration runs approx. 10 mins. • Compute all registrations in parallel on bmos cluster (mosix)

22. Building AAM of femur CTs • Input: • 14 CTs and 14 corresponding meshes • Output: • Mean shape mesh and 6 significant basis vectors • Mean appearance image and 10 significant basis vectors

23. Mean volume image - A0

24. Shape model S0 s0 + 2 std s1 s0 + 2 std s2

25. Using the AAM in 2D-3D registration Repeat until convergence: • Generate 2D DRRs out of AAM using current pose, shape, and appearance parameter estimates. • Rate similarity of DRRs to fluoroscopic images by similarity metric • Modify AAM shape parameters and the pose parameters by optimization algorithm. (We ignore appearance variations)

26. Generating DRRs from AAM – Algorithm 1 • Generate a 3D image of the AAM instance for each pixel in 3D image: • find tetrahedron containing pixel center (use KD-tree) • find corresponding point in base mesh (an affine transform) • read value from appearance image • Proceed with standard ray casting of 3D image [Knaan et al., 03]

27. Instance mesh Base mesh Shape-normalized volume image Generating DRRs from AAM – Algorithm 2 • Cast rays directly through tetrahedral mesh of AAM instance • Sample points at uniform intervals on ray • Find corresponding points in base mesh • Read values from appearance image • Requires identifying all ray-tetrahedron intersection points. We implemented algorithm of [Marmitt and Slusallek, 06]

28. Parameter search issues • Accuracy and Robustness • Search from various initial locations • Genetic algorithm • Downhill simplex optimizer (ITK amoeba) • Speed • Multi-resolution • Limit DRR generation to ROIs

29. Current status • The naïve search converges to unsatisfactory result (local minimum) • Need to improve both accuracy and speed • Search from multiple initialization points • ROIs • Think about a smarter optimization algorithm Future: • Experiment on real fluoroscopic images

30. Intensity-based registration of 2D X-ray images to a 3D deformable model Thank you for listening

31. Principal Component Analysis (PCA)