Learning

1. ICONIP 2007 Learning Hybrid Fuzzy Colour Processing and D.Playne, V.Mehta, N.Reyes, A.Barczak Computer Science, Massey University, Auckland, New Zealand

2. Introduction Presentation Highlights • Recognize robots via their colour patches and track them in real-time • Perform colour corrections for object recognition adaptively using Fuzzy Logic • Learn colour descriptors using successive frames • Extract the best fuzzy rules for any given colour automatically

3. Introduction Nature of the Object Source of Pictures: http://luminous-landscape.com/tutorials/color_and_vision.shtml

4. Introduction Human Eye, Rods and Cones R G B Source of Pictures: http://luminous-landscape.com/tutorials/color_and_vision.shtml

5. Introduction Red, Green & Blue Cones Spectral sensitivities of the RED, GREEN and BLUE CONES Source of Pictures: http://luminous-landscape.com/tutorials/color_and_vision.shtml

6. Introduction Human Visual System Fuzzy Colour Contrast Fusion Eye Light Source Object The colour sensed is the result of a combination of a multitude of factors.

7. Introduction Colour Constancy UNDER BLUISH ILLUMINANT UNDER YELLOWISH ILLUMINANT fundamental characteristic of the visual system which compensates for changes in illuminant color in order to keep object colours stable

8. Introduction Central Idea Fuzzy Inference System a priori knowledge of colour locus shifting in different illuminations + Humans tend to remember colours of familiar objects Adaptive contrast operations Different cones have different levels of sensitivity to Red, Green & Blue

9. Introduction Testbed Overhead Camera Fluorescent lamps Colour objects www.Fira.net Exploratory environment is indoor – room totally obstructed from sunlight Multiple monochromatic light sources – fluorescent / fluoride lamps Colour Object Recognition – speed is < 33ms, Camera height is approx. 2m.

10. Introduction Illumination Conditions Dark Other Factors: Lens focus Bright Object rotation Dim Quantum electrical effects Shadows Presence of similar colours We need to automatically compensate for the effects of varying illumination intensities in the scene of traversal *

11. Introduction Colours as captured by the camera Yellow object turns pale under strong white illumination Color is not captured by the camera as we humans see it. A Green object tends to appear more as a whitish yellow object under bright white illumination.

12. Colour Space Colour Space: Modified rg-Chromaticity Space g New Color attributes: Pie slice color decision region Radius: rg-Saturation Origin moved to the location of white. Angle: rg-Hue O r Pie-slice decision region was originally devised by Thomas et al.; however, they utilized the UV space. In this research, we transformed the rg-space to utilize the same pie-slice decision region.

13. Colour Space Colour Space • The new color descriptors, namely rg-Saturation and rg-Hue are derived as follows: • Compute rg-chromaticities from the camera RGB tristimulus: • Assign white as the new reference point or origin (Onew) of the new color space Onew(0.333, 0.333), then compute for rg-Saturation as the radius extending from the origin to any given pair of rg-chromaticities. • 3. Next, compute for the rg-Hue descriptor as the angle relative to the Xnew-axis, with origin at Onew:

14. Color decision region Classification Colour Classification g Ynew Chrominance Point rg-Saturation Color decision region for Color 1 rgray zone θ (Rg-Hue) Xnew Onew Gray zone e.g. PINK r If (rg-Saturation >= rmin) and (rg-Saturation <= rmax) (rg-Hue >= θ1 ) and (rg-Hue <= θ2) Then Color = Pink. Pie-slice decision region was originally devised by Thomas et al.; however, they utilized the UV space. In this research, we transformed the rg-space to utilize the same pie-slice decision region.

15. Classification Colour Classification Would the pie-slice decision region suffice for accurate color object identification in the newly transformed rg-chromaticity space?

16. Fuzzy Contrast Fusion Colours under varying illumination intensity Light Blue Patches Let’s zero-in on some of the colour identification targets… Presence of similarly coloured objects (i.e. Light Blue, Blue & Violet) Scene confounded with spatially varying illumination and highlights

17. g g g g g g g g r r r r r r r r Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion 1 2 3 4 Object 1 Object 2 Object 3 Object 4 6 7 All 5 Object 5 Object 6 Object 7 Objects 1-to-7

18. Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion There is a need to adaptively apply contrast operations on each of the colors comprising an object to localize the color locus formation at some fixed region. Pie-Slice Decision Region

19. Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion Excel File Source: T. Ross, Fuzzy Logic with Engineering Applications. Singapore: McGraw-Hill, Inc., (1997).

20. Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion Contrast Enhance: Logistic Function for 0 <= μα(y) <= 0.5 for 0.5 <= μα(y) <= 1 This operation acts in a combination of contraction and dilation. Source: T. Ross, Fuzzy Logic with Engineering Applications. Singapore: McGraw-Hill, Inc., (1997). Pal, S., and D. Majumder. (1986). In Fuzzy mathematical approach to pattern recognition, John Wiley & Sons, New York.

21. 71 71 71 99 99 99 56 56 56 51 51 51 38 38 38 122 122 122 117 117 117 115 115 115 51 51 51 26 26 26 153 153 153 166 166 166 112 112 112 56 56 56 31 31 31 166 166 166 166 166 166 107 107 107 74 74 74 23 23 23 158 158 158 158 158 158 120 120 120 71 71 71 18 18 18 Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion Image Process each colour channel in isolation Green Components Blue Components Red Components

22. Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion Example:Enhancing the Blue component of a pixel Value of Blue component

23. Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion Example:Enhancing the Blue component of a pixel Scaled Component = 115/255 0.45098

24. Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion Example:Enhancing the Blue component of a pixel Enhanced Component (Normalized) 0.4067666 NORMALIZED COLOR COMPONENT MATRIX AFTER APPLYING THE ENHANCEMENT ALGORITHM

25. Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion Example:Enhancing the Blue component of a pixel Enhanced component (ready for display)

26. Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion Raw Image Enhanced Image

27. Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion Excel File

28. Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion Contrast Degrade: Logit Function for 0 <= μα(y) <= 0.5 for 0.5 <= μα(y) <= 1 This operation pulls input signals towards 0.5

29. enhance degrade retain Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion • Performed separatelyon each of the RGB color values • Acts on either one of the following operations: Enhance, Degrade or Retain original value • Simultaneously accounts for all confounding factors found in natural scenes. Retain Enhance Degrade Colour Channel

30. Fuzzy Contrast Fusion General Architecture RNEW, GNEW, BNEW Camera RGB Adaptive Color Contrast Operations Color Contrast Fusion rg-chrom. space Rg-Hue, Rg-Sat Rg-Hue, Rg-Sat rg-chrom. space (Pie-Slice Decision Region) Perceived Color Target color Contrast Constraints This algorithm appears on: N. H. Reyes, E. P. Dadios,. "Dynamic Color Object Recognition", Int’l. Journal Of Advanced Computational Intelligence, Feb. 2004, Japan

31. Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion Pie-slice decision Rg-Hue, Rg-Saturation Light Blue Blue Violet …etc. Target Colour INPUTS (from Camera) Rules for Red Degrade R, G, B New Red OUTPUTS(Corrected Colour Values) Enhance Red Retain Rules for Green Degrade New Green Enhance Green Retain Rules for Blue Degrade New Blue Blue Enhance Retain Colour Contrast Constraints

32. Fuzzy Contrast Fusion Fuzzy Colour Contrast Fusion If (rg-Hue depicts Light Blue) Then Apply High Contrast Degrade on Red channel and Apply Low Contrast Enhance on Green channel and Apply Medium Contrast Degrade on Blue Channel. Example:Colour Contrast Rule for Light Blue Empirically derived!

33. Fuzzy Contrast Fusion Sample Results: Light Blue Targets Pie-slice decision region Pie-slice decision region + Contrast

34. Learning Colour Learning Initial Parameters Calibrated Parameters

35. Learning Colour Learning

36. Learning Colour Learning The centre and radius of the circle has now been found so the next part of the algorithm can run. The learning algorithm works on a moving average system combined with a decaying learning rate algorithm. The algorithm will run for a set number of iterations and keep moving average of the maximum and minimum rg-Hue and rg-Saturation:

37. Learning Colour Learning The idea of the algorithm is to move a robot with a colour patch or roll a ball around the board to calibrate the colour. Because the object will move through all of the different illumination conditions, the algorithm will calibrate the colour classifier to work for the entire board, accounting for all possible illumination conditions.

38. Rule Extraction Colour Contrast Rule Extraction

39. Rule Extraction Sample Targets: Yellow objects

40. Rule Extraction Colour Contrast Rule Extraction Automatic Rule Extraction: CCRE Algorithm Manual Rule Extraction

41. Robots in action The Fuzzy Vision algorithm employed in the game… Robots in Massey (IIMS Lab 7) Currently, we still have not incorporated our new Fuzzy obstacle avoidance and Target seeking algorithm yet. Nonetheless, the robots are able to play soccer already.

42. Conclusions Conclusions • We have presented a myriad of algorithms that allow colours to be enhanced or degraded through a hybrid fuzzy approach. Now colours can be processed by fuzzy colour contrast rules for more accurate colour object recognition. We have also incorporated rule extraction and colour learning algorithms for calibrating the colour contrast rules and target colour parameters automatically and more accurately.

43. Conclusions The End. We are happy to answer any of your questions. Thanks for listening!

44. Fuzzy Colour Contrast Fusion • Tested on 2 different cameras: • Samsung • AVT Marlin F-033C IEEE-1394 by Allied Vision Technologies • Tested on YUV, HSI and rg-Chromaticity colour spaces

45. * Summary We have presented an algorithm that resembles the colour discriminating feature of the human visual system. Where our memory serves as a guide in isolating an object’s colour from the scene, Fuzzy Colour contrast rules are used to perform colour corrections; thus, breaking the limits of accuracy of the pie slice-decision region for colour object identification.

46. Results & Analysis Results of Applying Colour Contrast Fusion in rg-chromaticity, YUV, and HSI Colour Spaces. *

47. Contrast Threshold Partitioning of the rg Space Threshold = 0.5 Red High Red Low Green High Green Low Blue High Blue Low

48. Colour Correction Experiments performed at IIMS Lab 7 The colour patches under bright illumination Image taken by an overhead camera 10 feet from the ground

49. Colour Correction Experiments performed at IIMS Lab 7 After turning off some lights Colours degrade considerably, reducing visibility of objects Applying the colour contrast operations to compensate for the effects of glare, hue and saturation drifting allows for better colour spotting Using the algorithm, Pink colours can be correctly classified

50. Colour Correction Experiments performed at IIMS Lab 7 To some extent, the algorithm can see in the dark Applying the colour contrast operations to compensate for the effects of glare, hue and saturation drifting also allows for colour correction