Image-guided therapy and medical robotics tutorial using a LEGO Mindstorms NXT robot and 3D Slicer

1. Image-guided therapy and medical robotics tutorial using a LEGO Mindstorms NXT robot and 3D Slicer Danielle F. Pace, B.CmpH Ron Kikinis, M.D. Nobuhiko Hata, Ph.D Surgical Planning Laboratory, Brigham and Women’s Hospital andHarvard Medical School

2. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 Funding Information This tutorial was made possible by grant numbers 5U41RR019703, 5P01CA067165 and 5U54EB005149 from the National Institutes of Health (NIH). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH. This tutorial was also in part supported by National Science Foundation (NSF) grant number 9731748 and the Center for Integration of Medicine and Innovative Technology (CIMIT).

3. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 Thanks to • Terry Peters, Ph.D (Robarts Research Institute, University of Western Ontario) • Steven Canvin (The LEGO Group) • Cory Walker (NXT++ Developer) • G. Wade Johnson (Device::USB Developer) • Steve Pieper, Ph.D (Surgical Planning Laboratory) • Haiying Liu (Surgical Planning Laboratory) • Junichi Tokuda, Ph.D (Surgical Planning Laboratory) • Christoph Ruetz (Surgical Planning Laboratory) • Philip Mewes (Surgical Planning Laboratory) • The entire LEGO Mindstorms community

4. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 Goals of the LEGO IGT and medical robotics tutorial • To emphasize Slicer3’s support for image-guided therapy (IGT) using medical robots • To demonstrate the typical steps of an image-guided therapy or medical robotics procedure in a hands-on manner The example procedure that we will use to do this is a needle biopsy.

5. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 An Example of IGT in Action: Needle Biopsy • In suspected cancer cases (such as for prostate or breast cancer) a needle biopsy is often performed as part of the diagnostic process. • A needle is used toextract small pieces oftissue for analysis. Goal: Hit the tumour so that it can be detected! (image from R. Alterovitz, K. Goldberg and A. Okamura, Proceedings of the 2005 IEEE International Conference on Robotics and Automation, Barcelona, Spain, April 2005, pp. 1652-1657)

6. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 Medical Robotics • Medical robots are increasingly popular for procedures such as needle biopsy in prostate cancer • In medical robotics, we use the same steps as in IGT: imaging, planning, registration, tracking and navigation • To the right is a prostate biopsy robot that operates within an MRI machine (being tested on a melon!) (JHU/SPL MRI prostate robot)

7. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 Tutorial Supplies • One LEGO Mindstorms NXT robotics kit • One LEGO Deluxe Brick Box • Two pom-poms • CT volume of the phantom • 3D Slicer LEGO tutorial module • A Linux computer with root access • Two sheets of white paper and tape • Assembly instructions for the robot and phantom • Phantom placement guide

8. (-22, 84, 41) z y z x y x Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 IGT Workflow Step 1:Imaging (image modified from S. Haker et al., Top Magn. Reson. Imaging 16(5), October 2005, pp. 355-368) Step 2:Planning Step 3:Registration ICS (image from http://www.ncigt.org/ourwork/core_mrguidedtherapy.html) Step 4:Tracking and Navigation PCS

9. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 1. Set up the Robot and Phantom Position the phantom relative to the LEGO robot such that: • The needle can reach both pom-pom targets • The registration pillars will be in view of the LEGO robot’s ultrasonic sensor

10. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 2. Imaging - Load the CT Volume of the Phantom

11. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 3. Connect Slicer to the robot Slicer3 connects to the robot via a USB cable

12. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 4. Registration The position of the phantom relative to the LEGO robot is not known. We must determine the relationship between the image coordinate system (ICS) and the patient (robot) coordinate system (PCS). We must perform registration!

13. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 The registration procedure • Find the eight fiducials in the patient (robot) coordinate system (PCS) • Find the eight fiducials in the image coordinate system (ICS) • Use a landmark registration algorithm to determine the rigid transformation from the image coordinate system to the patient (robot) coordinate system: PCS3x1 = R3x3 • ICS3x1 + t3x1

14. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 The eight fiducials 6 5 2 1 back 4 3 8 7 front(towardsrobot)

15. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 Finding the fiducials in the PCS z z x • Two views of the patient (robot) coordinate system: x y y

16. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 Finding the fiducials in the PCS four “registration pillars” • The LEGO robot’s ultrasonic sensor detects distance to the nearest obstacle. • We will find the phantom by using the robot’s ultrasonic sensor to find the “registration pillars” on the phantom.

17. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 Finding the fiducials in the PCS • Use the ultrasonic sensor to find the registration pillars: the 4 angles shown • Average the 4 angles to find  • We know d because the robot’s needle can hit both pom-poms • Use , d and knowledge of the phantom’s structure to find the eight fiducials on the phantom y d  x PCS

18. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 Finding the fiducials in the PCS • The robotic arm moves from left to right while polling the ultrasonic sensor for distance and the motor’s rotation sensor for the number of degrees • This occurs twice - once for the “top” pillars and once for the “bottom” pillars. y

19. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 Finding the fiducials in the PCS • Profiles of the number of degrees that the motor has turned vs. distance are generated for the top and the bottom swipes • The local minima represent the registration pillars and are automatically detected y

20. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 Finding the fiducials in the ICS Select the corresponding fiducials in the image coordinate system by clicking on the points on the CT volume.

21. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 5. Preoperative Planning - select the target coordinate • The target coordinate is the location where you would like the tip of the robot’s needle to be at the end of the “biopsy”. • Let’s try to have the robot biopsy the red pom-pom. • We need to find the center of the red pom-pom on the CT volume.

22. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 5. Planning - select the target coordinate Click on the target coordinate: the center of the red pom-pom.

23. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 6) Execute the biopsy Tracking is done through targeting during the robot’s movement The display of the final needle position in the PCS provides Navigation Transformation from the ICS to the PCS The target in the patient (robot) coordinate system The final needle position in the patient (robot) coordinate system

24. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 After completing this tutorial: • You have seen an example of medical robotics within Slicer3 • You have learned about the following steps in IGT and medical robotics in a hands-on manner: • Imaging • Preoperative planning • Targeting and tracking • Navigation • Registration • Tutorial website:http://wiki.na-mic.org/Wiki/index.php/LEGO_IGT_and_ Medical_Robotics_Tutorial

25. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 Additional references • Learn about image registration by reading: • B. Zitová and J. Flusser. Image Registration Methods: A Survey. Image and Vision Computing, 21:977-1000, 2003. • Brainstorm about the components of the total error that occur from the selection of the target to the final position of the robot needle tip. How could you measure those errors? • Start by learning about error in registration by reading: J.B. West, M.J. Fitzpatrick, S.A. Toms, C.R. Maurer, and R.J. Maciunas. Fiducial Point Placement and the Accuracy of Point-based, Rigid Body Registration. Neurosurgery, 48(4):810-817, 2001.