Programming for Image Processing/Analysis and Visualization using The Visualization Toolkit

1. Programming for Image Processing/Analysis and Visualization using The Visualization Toolkit Xenios Papademetris papad@noodle.med.yale.edu BML 325, 5-7294 http://noodle.med.yale.edu/seminar/seminar.html

2. Course Structure • Using VTK using scripting languages • Understand Toolkit Structure • Use existing algorithms (incl. local extensions) • Learn Tcl/Tk • Extending VTK • C++/Object Oriented Programming/Design • Cross-Platform Issues • Coding Guidelines

3. Schedule – Part 2 • Review of Part 1 and Course Overview • C++ Pointers/Classes, Object Oriented Programming • Adding new VTK Commands/Cmake • Image-to-image filters/ surface to surface filters • Case Study I -- Iterative Closest Point surface matching • Case Study II – A Simple Segmentation Algortihm

4. VTK Pipeline (I) File Output Sources Filters Mappers Props vtkDataSet e.g. Images – vtkImageData Surfaces – vtkPointData vtkDataSet

5. VTK Pipeline (II) Props Render Window Renderer Props Props (e.g. Actor/Volume) vtkCamera, vtkLight vtkRenderWindowInteractor vtkProperty

6. Datasets • Organizing structure plus attributes • Structured points • Rectilinear Grid • Structured Grid

7. Data Representation(vtkDataSet) Point Attributes (vtkPointData) Point Properties (e.g. intensity) Points (vtkPoints) Define Location Arrays of Numbers (one per point or cell) vtkDataArray Data Set Cells (vtkCellArray) Define Topology Cell Attributes (vtkPointData) Cell Properties (e.g. normal)

8. Cells • Cell is defined by an ordered list of points • Triangle, quadrilateral points specified counter clockwise • Others as shown 6 3 7 5 4 1 3 2 2 Hexahedron 1 Tetrahedron 0 0

9. Representing arrays with vtkDataArray • vtkDataArray is an abstract superclass for classes representing arrays of vectors called tuples (or numbers treated as vectors of length 1). Each tuple consists of a set of numbers or components. • Derived Classes includevtkUnsignedCharArray, vtkShortArray, vtkFloatArray, vtkDoubleArray etc. • Can function either as a dynamic array (lower performance) or a fixed length array • All data in VTK is stored ultimately in one of the many derived classes of vtkDataArray, e.g. in the case of vtkImageData the intensities are stored in a vtkDataArray having dimensions equal to the number of voxels and vector length typically equal to 1 (3 for color images, Number of Frames for multiframe data such as fMRI cardiac etc.)

10. vtkDataArray Hierarchy

11. A concrete example vtkFloatArray Mode 1 – Fixed Length Array: To create in TCL type vtkFloatArray arr arr SetNumberOfComponents 1 arr SetNumberOfTuples 20 arr SetComponent 10 0 10.0 arr SetTuple1 11 9.0 set b [ arr GetComponent 10 0 ] This creates an array of 20 (Number of Tuples) vectors each having size 1 (Number of Components) We access elements in this array be using the SetComponent and GetComponent methods. All indices start at 0.

12. A concrete example vtkFloatArray Mode 2 – Dynamic Array To create in TCL type vtkFloatArray arr arr SetNumberOfComponents 1 arr InsertNextTuple1 5 arr InsertNextTuple1 10 set b [ arr GetComponent 1 0 ] This creates a dynamic array of vectors each having size 1 (Number of Components). The InsertNextTuple command allocates memory dynamically and places the value there Note also (InsertNextTuple2, InsertNextTuple3,4,9 for multi-component arrays)

13. Implicitly Defined by Image Specification Rarely Used Images are Simple Data-sets Point Attributes (vtkPointData) Point Properties (e.g. intensity) Points (vtkPoints) Define Location Arrays of Numbers (one per point or cell) vtkDataArray vtkImageData Cells (vtkCellArray) Define Topology Cell Attributes (vtkCellData) Cell Properties (e.g. normal)

14. vtkImageData • vtkImageData is the basic VTK class for storing images. • It is defined by 4 key elements • Dimensions -- these define the size of the image • Origin -- the position in 3D space of point 0 0 0 • Spacing -- the voxel dimensions • Scalar Type -- the type of the image (e.g. float, short etc) • An 4x4x4 image has 4x4x4=64 points and 3x3x3=27 cubic cells (both are implicitly defined)

15. Manually Creating an Image in TCL vtkImageData img img SetDimensions 10 10 2 img SetOrigin 0 0 0 img SetSpacing 0.78 0.78 1.5 img SetScalarType $VTK_SHORT img SetNumberOfScalarComponents 1 img AllocateScalars Intensity values can be accessed using the scalar array i.e. Point 0 is (0,0,0), point 1 is (1,0,0), point 10 is (0,1,0), point 100 is (0,0,1) set data [ [ img GetPointData ] GetScalars ] $data SetComponent 10 0 5.0 set v [ $data GetComponent 10 0 ] Set v2 [ $img GetScalarComponentAsFloat 0 1 0 0 ] ( this unfortunately is the nearest vtk comes to a getvoxel, no set voxel command)

16. Our own Tcl/Tk Code (or Python, Java) Our own C++ Code Tcl / Tk (or Python/Tk or Java) VTK Open GL Computational Aspects 3D Graphics Graphical User Interface (GUI) Multi-Platform Program Structure Hardware Layer (Windows/Linux/Unix/Mac OS X)

17. A First Example(http://www.vtk.org/example-code.php) Creating a sphere in Tcl, C++ or Java, using the underlying C++ VTK libraries.

18. Source Mapper Actor TCL – part 1 # First we include the VTK Tcl packages which will make available # all of the vtk commands from Tcl. The vtkinteraction package defines # a simple Tcl/Tk interactor widget. package require vtk; package require vtkinteraction # create sphere geometry vtkSphereSource sphere sphere SetRadius 1.0 sphere SetThetaResolution 18 sphere SetPhiResolution 18 # map to graphics library vtkPolyDataMapper map; map SetInput [sphere GetOutput] # actor coordinates geometry, properties, transformation vtkActor aSphere aSphere SetMapper map [aSphere GetProperty] SetColor 0 0 1; # blue

19. Mapper Actor Renderer Render Window vtkRenderWindowInteractor TCL – part 2 # actor coordinates geometry, properties, transformation vtkActor aSphere aSphere SetMapper map [aSphere GetProperty] SetColor 0 0 1; # blue # create a renderer add the sphere vtkRenderer ren1 ren1 AddActor aSphere ren1 SetBackground 1 1 1;# Background color white # create a window to render into vtkRenderWindow renWin renWin AddRenderer ren1 # create an interactor vtkRenderWindowInteractor iren iren SetRenderWindow renWin # Render an image; since no lights/cameras specified, created automatically renWin Render

20. Source Mapper Actor C++ – part 1 #include "vtkSphereSource.h" #include "vtkPolyDataMapper.h" #include "vtkActor.h" #include "vtkRenderWindow.h" #include "vtkRenderer.h" #include "vtkRenderWindowInteractor.h" void main () { // create sphere geometry vtkSphereSource *sphere = vtkSphereSource::New(); sphere->SetRadius(1.0); sphere->SetThetaResolution(18); sphere->SetPhiResolution(18); vtkPolyDataMapper *map = vtkPolyDataMapper::New(); map->SetInput(sphere->GetOutput()); // actor coordinates geometry, properties, transformation vtkActor *aSphere = vtkActor::New(); aSphere->SetMapper(map); aSphere->GetProperty()->SetColor(0,0,1); // sphere color blue … }

21. Source Mapper Actor Java – part 1 import vtk.*; public class test { // in the static constructor we load in the native code // The libraries must be in your path to work static { System.loadLibrary("vtkCommonJava"); System.loadLibrary("vtkFilteringJava"); System.loadLibrary("vtkIOJava"); System.loadLibrary("vtkImagingJava"); System.loadLibrary("vtkGraphicsJava"); System.loadLibrary("vtkRenderingJava"); } public static void main (String[] args) { // create sphere geometry vtkSphereSource sphere = new vtkSphereSource(); sphere.SetRadius(1.0); sphere.SetThetaResolution(18); sphere.SetPhiResolution(18); // map to graphics objects vtkPolyDataMapper map = new vtkPolyDataMapper(); map.SetInput(sphere.GetOutput()); // actor coordinates geometry, properties, transformation vtkActor aSphere = new vtkActor(); aSphere.SetMapper(map); aSphere.GetProperty().SetColor(0,0,1); // color blue

22. Comment on Language Selection • VTK can be used from any one of C++/Java/Tcl and Python • Some features not accessible from the scripting languages • In Java/Tcl/Python our own code can: • Be used to join standard VTK elements in a pipeline • Call existing VTK routines • In C++ we can do all of the above plus • Write new filters/sources etc as required (which can in turn be used from the other languages)

23. Programming Languages • Interpreted vs Compiled Languages • Note that classic distinction is blurring thanks to on-the-fly compilers (e.g. Tcl 8.x) • Interpreted Languages • e.g. BASIC, Matlab, Tcl, Python, Perl … • Fast development, computationally inefficient • Compiled Languages • e.g. C/C++, Fortran, Java(?) • Computationally more efficient but extra step in development cycle

24. Interpreted Languages • Program execution is performed by the interpreter or shell e.g. • TCL (tcsh/wish) • Allows for interactive command execution • Faster development cycle because there is no lengthy compilation step • Slower performance as each command needs to be parsed before it is executed • Simpler to use!

25. Compiled Languages • Compiler converts source code into executable. • Typically allows for standalone programs (especially if statically linked – more later) • Slower development cycle because there is a compilation step. • Less prone to syntax errors as compiler flags these out. • More complex to use some understanding of lower level computer fundamentals is necessary.

26. Compiling a simple C++ program(single source file cosine.c) #include <math.h> #include <stdio.h> int main(int argv,char** argv) { printf(”Sin(40)=%5.3f\n”, sin(40.0*M_PI/180.0)); } (In case you are unsure this program simply prints the value of the cosine of 40 degrees, the ugly 40.0*M_PI/180.0 construct is needed as the cos function requires its arguments to be in radians)

27. Step 1 – creating the object file(compiling) (This is assuming a UNIX setup) In the same directory as the file cosine.c type g++ –c cosine.c This if everything works out will produce the object file cosine.o

28. Step 2 – creating the executable(linking) (Again assuming a UNIX setup) In the same directory as the file cosine.o type g++ –o cosine cosine.o -lm Statement to link in the math library libm.so. In unix shared libraries are assumed to begin with `lib’ which is not included in the link directive. Output File

29. Step 3 -- Execution Typing `cosine’ on the command prompt, will (roughly) 1. Load the program cosine into memory 2. Load the system shared libraries into memory (e.g. libC and libm) 3. Start the program by calling the main function.

30. More on Libraries • Typically many different programs are built using standard building blocks e.g. • Math Library • I/O Library • Graphics Library • Custom libraries e.g. to read/write certain medical image formats • Libraries can be created as static or dynamic. • Static: each program has its own copy of the library embedded in the executable (typically .a or .lib) • Dynamic: one copy of the library for the whole system (typically .dll or .so)

31. Compiling a more complicated C++ program and library Project consists of 5 files: • utility.h -- header file defining the specifications of the utility code • Utility.cpp – actual code for the utility code • Print.h – header file for the printing code • print.cpp – implementation of the printing code • Main.cpp -- main program that uses code in utility.cpp and print.cpp Plan: • Create a library containing the code in utility.cpp and readwrite.cpp • Link the library with the object file main.o to create the executable

32. Utility.h and Utility.cpp Utility.h #include <math.h> float deg2rad(float t); Utility.cpp #include “utility.h” float deg2rad(float t) { return t*M_PI/180.0; }

33. print.h and print.cpp print.h #include <math.h> void printnumber(float t); print.cpp #inlcude “print.h” void printnumber(float t) { printf(”The number is %5.2f\n”,t); }

34. Main.cpp main.cpp #include “print.h” #include “utility.h” #include “math.h” int main (int argc,char **argv) { float t=40.0; float trad=deg2rad(t); float costrad=cos(trad); printnumber(costrad); }

35. Compling and LinkingMethod 1 – no libaries g++ -c utility.c produces utility.o g++ -c print.c produces print.o g++ -c main.c produces main.o g++ -o main main.o utility.o print.o –lm This yields the final executable main Note that the following one-step procedure is also possible but only useful for really small projects: g++ -o main main.cpp utility.cpp print.cpp –lm

36. Compling and LinkingMethod 2 – static library method g++ -c utility.c produces utility.o g++ -c print.c produces print.o ar -cr libutil.a print.o utility.o produces libutil.a (a ranlib step may also be needed) g++ -c main.c produces main.o g++ -o main -lutil –lm This yields the final executable main by linking it with the library libutil.a and the math library libm.so. The size of main is approximately equal to Size(main,o) + size(libutil.a) + overhead The executable main will run just fine even if libutil.a is deleted.

37. Compling and LinkingMethod 3 – shared library method g++ -c utility.c produces utility.o g++ -c print.c produces print.o g++ -shared -o libutil.so print.o utility.o produces libutil.so g++ -c main.c produces main.o g++ -o main -lutil –lm This yields the final executable main by linking it with the library libutil.so and the math library libm.so. The size of main is approximately equal to Size(main,o) + overhead The executable main will NOT run if libutil.a is deleted. The LD_LIBRARY_PATH variable also needs to be set to point to the location of libutil.so

38. Automating compiling/linkingThe make command The compiling process of even the simplest project involves the typing of a large number of commands. • The process is automated on UNIX using the make command and its corresponding macro file (makefile) • On Windows (using Visual C++) the same is done using either nmake (and makefiles) or project files in Visual Studio • VTK uses a higher level program called cmake (www.cmake.org) which produces output for either unix makefiles or windows visual C++ project files/makefiles. • We will only describe cmake next week.

39. Extending VTK – the gameplan • Gather our own code into one or more shared libraries • Automatically wrap the library to generate interface code to make our library accessible from TCL (Java/Python also possible) • Load our library into the tcl interpreter in effect making our code appear as additional TCL commands • Use TCL as a glue language to test/call/debug our C++ code.

40. Next Week • Discuss pointers and memory allocation issues • Discuss Object-Oriented Design Philosophy in more detail • Constructors/Destructors • Inheritance • Abstract Classes • Virtual Functions • Overriding Functions • Discuss reference counted allocation/de-allocation

41. “Homework’’ • If you are not familiar with basic C/C++ get a book and read about it. I will discuss the object stuff in detail but you should be familiar with • Basic syntax • Loop/conditional operators • Procedures and functions • Input/output statements • Variable types • If you have not already go through the slides from the last seminar series and freshen up on some of the material there. I will be assuming some familiarity with it.