Learning Patterns in Images

1. Learning Patterns in Images 전산과학과 인공지능연구실 이 재영

2. Concern • learning patterns in images & image sequences • using the obtained pattern for interpreting new images • Three problem areas • semantic interpretation of color image • detection of blasting caps in x-ray image • recognizing actions in video image sequences => Image formation processes => The choices of representation spaces

3. Introduction • Motivation of this research • vision system need learning capabilities for difficult problems • Current research on learning in vision • concentrated on neural network • Symbolic learning • insufficiently explored • potentially very promising research area

4. Semantic Interpretation of Color Images of Outdoor Scenes • MIST Methodology • Multi-level Image Sampling and Transformation • environment for applying diverse machine learning methods to computer vision • semantically interpret natural scenes • Three learning programs • AQ15c: learning decision rules from examples

5. NN: neural network learning • AQ-NN: multi-strategy learning combining symbolic and neural network methods

6. The MIST Methodology • Learning Mode • builds or updates the Image Knowledge Base • image area attributes => class label • developed by inductive inference from examples by a trainer • Interpretation Mode • learned image transformation procedure • new image => ASI(Annotated Symbolic Image) • ASI area <=> class label, annotations(additional information)

7. Original Image ASI

8. Description space generation & BK formulation • Training (Learning Mode) Event generation: area -> attribute vector ASI Learning or refinement -> class description Image interpretation and evaluation Image Knowledge Base

9. Description space generation & BK formulation • initialized by trainer • assign class names to area • define initial description spaces • initial attributes: hue, saturation, gradient, intensity, high frequency spots, etc… • procedures for the measurement of attributes • Event generation • using chosen procedures • sampled areas -> training examples (attribute vector) • computed by 5x5 windowing operator

10. Learning or refinement • applies selected machine learning program to the training examples • generate a class description (a kind of rule?) • Image interpretation and evaluation • applying developed descriptions to testing area • generate an Annotated Symbolic Image(ASI) • area <=> class labeling & annotations • compare ASI label with test area label • => stop train or continue iteration with ASI as input • complete class description: a sequence of image transformations that produce final ASI

11. Interpretation Mode • procedure new image => apply description => majority voting(3x3 window) => ASI with annotation(degree of confidence)

12. Experimental Result • AQ-NN • AQ • learn attributional decision rules from example • used to structure NN architecture • NN • further optimize the AQ induced descriptions

13. Detection of Blasting Caps in X-Ray Images • Problem • inspect a sequence of images for known objects • but little standardization of the class of objects • Focus • how vision and learning can be combined to find blasting caps • relationship between image characteristics and object functionality

14. X-ray of blasting caps l Secondary Explosive a 2r lseca

15. Blasting caps • various shape but same functionality • strongest feature • low intensity blob in the center of a rectangular ribbon of higher intensity • the intensities of both blob & ribbon are lowest along the axis of the blasting cap and highest along the occluding contour • blasting caps can be occluded by other objects • airport security scenario • detect blasting caps in x-ray image of luggage

16. Methods and experimental results • AQ15c inductive learning system was used to learn descriptions of blasting caps and non-blasting caps (geometric & intensity features) • First phase • detect candidate : find low intensity blobs • Second phase • a flexible matching routine is used to match the local model to the image characteristics • attempt to fit a local model to ribbon-like features surrounding the blob

17. Test luggage image • contains various objects • clothes, shoes, calculators, pens, batteries, etc… • Results

18. Recognizing Actions in Video Image Sequence • Recognizing the function of objects from its motion • based on characteristics such as shape, physics and causation • velocity, acceleration, force of impact from motion => strongly constrain possible function • object(motion) should not be evaluated in isolation, but in context

19. Primitive shapes • stick : a1a2 << a3 • strip : a1a2 ∧ a2a3 ∧ a1a3 • plate: a1  a2 >> a3 • blob : a1  a2  a3 • Primitive motions (ex. Knife) • stabbing • slicing • chopping

20. Inferring object function from primitive motions • object: a collection of primitives • knife : handle(stick) + blade(strip) • function depends on object’s motion • in object’s coordinate system & • in relative to the actee (object it acts on) direction of motion the main axis of the object the surface of the actee => determine intended function

21. Ex] knife motion • stab: parallel to the main axis of knife & perpendicular to the surface of the actee • chop: perpendicular to the main axis & perpendicular to the surface of the actee • slice: back-and-forth motion parallel to its main axis & parallel to the surface of the actee • Computing primitive motion • for both actor’s coordinate & actee’s coordinate • using optical flow with shape information(main axis, center of mass, …)

22. Parameterizing the Motion of a Stick or Strip   Motion direction 

23. Experiments • Knife • 25 frames /second for 5 seconds => 125 images • sampling • 11 evenly spaced samples, each composed of 3 consecutive images( 0-2, 10-12, …_ • 33 images for each experiment

24. Stabbing angle  0   -90 time

25. Chopping angle  0   -90 time

26. Slicing  angle  0  -90 time

27. Summary • Semantic interpretation of color images • apply machine learning to computer vision • AQ-NN • Detection of blasting caps • analysis of the functional properties of blasting caps (intensity & geometric features)

28. Recognizing actions in video image sequences • understanding of the way an object is being used by an agent • use combined information • the shape of the object • its motion • its relation to the actee