Video Summarization of Key Events Stage I - The Critical View

1. Video Summarization of Key EventsStage I - The Critical View Michael A. Grasso, MD, PhD University of Maryland School of Medicine UMBC Computer Science MichaelGrasso.com

2. Abstract Laparoscopic surgery is a minimally invasive technique with unique training requirements. Video-assisted evaluation is one method that surgical residents can use to demonstrate competence. Automated video summarization can increase the efficiency of evaluations by directing the senior surgeon to key portions of a surgical procedure. We are using image classification techniques to segment videos of laparoscopic cholecystectomies to assist with surgical training and evaluation.

3. Overview • Background • Laparoscopic Surgery • Image Classification • Methods • Discussion

4. Laparoscopic Surgery • Minimally Invasive Surgery. • First performed in 1987. • Used in many surgical procedures. • Gall bladder removal (cholecystectomy). • Esophageal surgery (fundoplication). • Colon surgery (colectomy). • Others.

5. Laparoscopic Approach • Narrow tubes (trocars) are inserted into the abdomen through small incisions. www.fda.gov

6. Laparoscopic Procedure • Camera is passed through trocar. • Procedure is often videotaped. • Carbon dioxide is infused through trocar. • Instruments are passed through the trocars to cut, manipulate, and sew.

7. Laparoscopic Aftercare • Compared with an open procedure. • Smaller scars. • Reduced pain. • Quicker recovery. http://www.nlm.nih.gov/medlineplus/ency/presentations/100166_1.htm

8. Technical Challenges • Access limited to small incisions. • Long instruments with only the tips visible. • Two-dimensional video. • Limited tactile feedback. British Journal of Surgery. 2004 Dec;91(12):1549-1558

9. Laparoscopic Training • Traditional apprenticeship model. • Acquire skills during actual procedures. • Not sufficient for laparoscopic skills. • Other methods. • Box trainer with animal or synthetic models. • Virtual reality simulator. • Video-based assessment.

10. Assessment of Skills • Trainee must demonstrate competency. • Evaluation by a senior surgeon. • Direct observation of the trainee. • Video-based assessment. • Question: Can we organize video in order to assist in video-based assessment? American Journal of Surgery. 1991 Mar;161(3):399-403

11. Video Segments Frames Objective • Identity key portions of surgical procedure to aid in video-based assessment. • Stage I is to identify the "critical view".

12. Overview • Background • Laparoscopic Surgery • Image Classification • Methods • Discussion • Summary: Organize surgical video to make it easier for expert to review.

13. The Critical View • Helps ensure that the anatomy has been properly identified. • Occurs after dissecting anatomy. • Occurs before clipping the cystic artery and cystic duct.

14. The Critical View Fundus Cystic artery Liver Cystic duct Netter's Atlas of Human Anatomy

15. The Critical View

16. Image Classification - Human • Features a person might use. • Spectral features. • Tonal variations. • Textural features. • Spatial distribution of tonal variations. • Contextual features. • Features from surrounding areas.

17. Image Classification - Computed • Features extracted from image. • Spectral features. • Distribution, size, width. • Textural features. • Homogeneity, contrast, correlation. • Similarity/distance metrics. • Jaccard coefficient, Jeffrey divergence. Journal of WSCG. 2003; 11(1):269-273 IEEE Transaction on Systems, Man, and Cybernetics. 1973 Nov; 3(6):610-621

18. Color Histogram • Red, green, blue, or gray. • Count number of pixels for each tone. • One 28 set for an 8-bit image for each color. • Does not vary with translation and rotation. • Ignores shape and texture. • 4x4 image. • 4 gray tones. • H = {5, 4, 5, 2}

19. Binary Histogram • Quantize values for each tone to 0 or 1. • Background color given less weight. • Subtle changes given more weight. • HB = {1, 0, 1, 1}

20. 3D Histogram • Distribution within a 3D color-space. • 3D color space (red, green, blue). • Used in object recognition & image retrieval. • n3 entries, where n = number of tones. • Example. • Quantized to 3 tones for each color.

21. Co-occurrence matrix. Co-occurring values (0o, 45o, 90o, 135o). Four 28 x 28 matrices for 8-bit image. Spatial-Dependency Matrix M0 =

22. Additional Spectral Features • Location of the distribution. • Mean = Σ (bin*freq) / Σ (freq). • Mode = bin of the max freq. • Size of the distribution. • Standard deviation. • Width of the distribution. • Max(bin) - Min(bin).

23. Additional Textural Features • Homogeneity. • Number of tone transitions. • Contrast. • Amount of local variation. • Correlation. • Measure of linear dependencies.

24. Similarity/Distance Metrics • Jaccard Coefficient. • Similarity of two sample sets. |A  B| / |A  B| • Two binary sets. M11 / (M01 + M10 + M11) • Jeffrey Divergence. • Distance between two vector spaces. Σ (xi log(xi/avgi) + yi log(yi/avgi)) n i=1

25. Other Distance Metrics • City Block or Manhattan Distance. • Euclidean Distance. • Chi-Square. • Canberra Distance. Proceedings ACM SAC. 2008;:1225-1230

26. Related Efforts - Hysteroscopy • Use Jeffrey divergence on color histogram to identify segments. • Relevant segments based on image redundancy. • No understanding of the content of each segment. Mayo Clinic Proceedings 27th IEEE-EMBS. 2005;:5680-5683

27. Related Efforts - Echocardiogram • Use cosine similarity and edge change ratio to identify video segments. • State-based modeling. • Identify states in each video segment. • Diastole (resting). • Systole (contracting). Medline Plus IEEE Transaction on Information Technology in Biomedicine. 2008 May;12(3):366-376

28. Overview • Background • Laparoscopic Surgery • Image Classification • Methods • Discussion • Summary: Spectral and textural features compared with similarity metrics.

29. Video Segments Frames Methods • Our objective. • Identity key portions of surgical procedure to aid in video-based assessment. • Stage I is to identify the "critical view".

30. Tools • FFmpeg • http://ffmpeg.mplayerhq.hu/ • Extract JPEG images. • ImageJ • http://rsbweb.nih.gov/ij/ • Macros and Java plugins.

31. Work Plan • Identify videos for analysis. • Convert videos to JPG. • Evaluate ability to identify critical view. • Color histogram. • Binary histogram. • 3D histogram. • Spatial-dependency matrix. • Jaccard coefficient, Jeffrey divergence.

32. Algorithm Color Histograms Binary Histograms 3D Histograms Spatial-Dependency Matrices Feature Extraction ImageJ Random Image Image Extraction FFmpeg Critical View Similarity Metric Critical View?

33. Overview • Background • Laparoscopic Surgery • Image Classification • Methods • Discussion • Summary: Attempt to identify the critical view by comparing image features with similarity metrics.

34. Discussion • Color and binary histograms do not correlate with the critical view. • They do, however, predict when we are in the abdomen. • Currently working on 3D histograms and spatial-dependency matrices. • NIH grant application under development.

35. Challenges • Live tissue (vs. solid objects). • Deformable. • Normal variation. • Disease states. • May need to consider. • Temporal information. • Relevant clinical data of the patient. • Critical view "rectangle" (contextual).

36. Summary • We are comparing image features with similarity metrics to identify the critical view. • This is a first step in automated video summarization, to help with video-assisted evaluation of laparoscopic surgery.

37. Thank You