1 / 77

Learning More About NokiaCV A Mobile Based Computer Vision Algorithm Suite

Learning More About NokiaCV A Mobile Based Computer Vision Algorithm Suite. Current Scenario. Mobile cameras are widespread Main uses for camera: capturing images/videos Still significant barriers to create new applications There is not enough support for camera/imaging in S60 SDK

adelie
Download Presentation

Learning More About NokiaCV A Mobile Based Computer Vision Algorithm Suite

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Learning More About NokiaCVA Mobile Based Computer Vision Algorithm Suite

  2. Current Scenario • Mobile cameras are widespread • Main uses for camera: capturing images/videos • Still significant barriers to create new applications • There is not enough support for camera/imaging in S60 SDK • Only a handful of small companies making camera apps. Most of these are experts – computer vision expertise, image/signal processing experience

  3. Solution: Nokia Computer Vision Library (NokiaCV) • Developers do not have to start imaging/camera applications from scratch • Shared platform will encourage further and more complex work in imaging domain • Nokia Computer Vision library built on top of Symbian • Fundamental operations for imaging and camera processing

  4. Prior work • A few libraries available for mobile devices, including S60 • Symbian mainly supports bitmap drawing, image/video capture. • No processing support • Difficult/slow to access individual pixels – common operation in processing • Other S60 imaging libraries are research-quality code, no support

  5. Goal • Internally to be used in imaging products, as a research test bed, etc. • Externally to provide developers in binary form to facilitate third-party imaging • Intel’s OpenCV is the desktop version of what NCV will become for mobiles • Full functionality • Wide range of imaging operations, however some not possible on mobile (yet) • NokiaCV’s main difference: supports operations that are useful and fast right now.

  6. Use Cases

  7. Capture Enhancement • Image color conversions, imaging wrapper to modify images easily and manipulate pixels directly and easily • Building blocks present for developers to make high-quality stitching algorithms for panoramas • Building blocks present for developers to implement complex algorithms without requiring the building and debugging of a platform

  8. Gaming/UI • Provides robust motion estimation as a software component, ready for games and applications

  9. Post-capture editing • Image warping operation provided to create standard warping or in other entertainment uses • Image compositing provided to add items to captured images

  10. NokiaCV Library Overview • Library builds on OS, extending imaging capabilities • Image object provided standardizing OS image internals • Standard image operations provided • Linear algebra used in many advanced imaging applications, present in NCV • Building block for future advanced libraries, both externally and internally

  11. Functional Blocks

  12. CCamus • CEgoMovement • CEigenvalues • CFixed • CImageOp • CArithmeticOp • CColorConversionOp • CComparisonOp • CConvolutionOp • CCornerDetectionOp • CEdgeDetectionOp • CGaussianSmoothOp • CMedianFilterOp • CMorphologicalOp • CResizeOp • CRotationOp • CShearOp • CTemplateMatchOp • CThresholdOp • CWarpOp • CImageOperations • CImagePyramid • CImageStatistics • CMotionHistoryImage • CNokiaCVImage • CNokiaCVMath • CNokiaCVMatrix • CNokiaCVVector • COpticalFlow • CPixelAccess • CPixelColorModel • CSVD • TColorModel • CHsvColorModel • CLabColorModel • CLmsColorModel • CRgbColorModel • CXyzColorModel • CYuvColorModel

  13. API Overview • This library extends the image processing and math capabilities of the Symbian Series 60 platform, targeted for applications using computer vision techniques on camera phones. • The library provides an image class, CNokiaCVImage for handling RGB, grayscale and black and white images. Provides functions to perform arithmetic operations, transformations, feature extraction, color conversions. In addition, operations to compute image statistics such as histograms and pixel color means are provided. • The library has two algorithms (CCamus and CMotionHistoryImage) to determine motion estimates and history. • Matrix and vector classes CNokiaCVMatrix and CNokiaCVVector allow creation of matrices and vectors of arbitrary dimensionality. Math operations and some linear algebra techniques that work with these classes are available in the library.

  14. Library Features

  15. CNokiaCVImage Class • CNokiaCVImage is a wrapper for CFbsBitmap that is provided by the platform. This class provides DisplayMode independent access to the pixel data. CFbsBitmap* iInternalBitmap; • Usage examples: // load from file CNokiaCVImage* incvBitmap = CNokiaCVImage::NewL(); TRAPD(err1, incvBitmap->CreateL( *aImageFullName )); ... // draw bitmap CWindowGc& gc = SystemGc(); gc.BitBlt( TPoint( 0,0 ), incvBitmap-> Bitmap() );

  16. Functions • static IMPORT_C CNokiaCVImage* CNokiaCVImage::NewLC ( ) ; Implements two-phase construction functions (NewLC(), ConstructL()) to create a new instance of the object. Returns: CNokiaCVImage* • IMPORT_C void CNokiaCVImage::CreateL (const TDesC & aFileName  ) ; Create image from a file. • static IMPORT_C void CNokiaCVImage::Copy (CNokiaCVImage * aSource, CNokiaCVImage * aTarget, TInt aBlendingRatio = 100 ); Copy a source image to the target image.

  17. CPixelAccess Class • CPixelAccess class provides access to CNokiaCVImage on a pixel-level. Derived from TBitmapUtil. • Example: CPixelAccess* out = CPixelAccess::NewL(aTarget); CPixelAccess* in = CPixelAccess::NewL(iImage, out); ... in->SetPos( TPoint( rotX, rotY ) ); out->SetPos(TPoint( x, y ) ); out->SetRGB( in->GetRGB() ); // set pixel value from in to out ... delete out; delete in;

  18. Functions • static IMPORT_C CPixelAccess* CPixelAccess::NewLC (CNokiaCVImage * aImage  ) ; Constructor. Sets the current pixel position to (0,0) position. • IMPORT_C TUint32 CPixelAccess::GetRGB ( ) ; Return pixel from the current position as a RGB. • IMPORT_C void CPixelAccess::SetRGB (TUint32 aPixel  ) ; Set pixel to the current position. Parameters: aPixel RGB value as TUint32 in format used by TRgb: 0x00BBGGRR

  19. CPixelColorModel Class • IMPORT_C TUint8 CPixelColorModel::Red ( ) ; Get the red component; Returns: TUint8 Red-component • IMPORT_C CYuvColorModel CPixelColorModel::ToYUV ( ) ; Convert to YUV. Returns: CYuvColorModel. • IMPORT_C CHsvColorModel CPixelColorModel::ToHSV ( ) ; • Convert to HSV return CHsvColorModel

  20. TColorModel Class • class CRgbColorModel : public TColorModel { public: TInt iR; TInt iG; TInt iB; }; • class CYuvColorModel : public TColorModel { public: TInt iY; TInt iU; TInt iV; };

  21. CNokiaCVMath Class • This class gathers some utility functions in one place. • static IMPORT_C TInt SinLUT_Rad (TReal &aTrg, const TReal &aSrc) Computes Sin using lookup table. • static IMPORT_C TInt TanLUT_Rad (TReal &aTrg, const TReal &aSrc) Computes Tan using lookup table.

  22. CNokiaCVMatrix Class • Fixed point matrix representation. Usage: CNokiaCVMatrix* m = CNokiaCVMatrix->NewLC(5,5); (*m)(3,3) = 2; (*m)(2,3) = 4.23; CNokiaCVMatrix* n = CNokiaCVMatrix->Ones(5,5); CleanupStack::PushL(n); CNokiaCVMatrix *res = (*m)*(*n)*5; CleanupStack::PushL(res); CFixed det = res->Det(); ... CleanupStack::Pop(m); CleanupStack::Pop(n); CleanupStack::Pop(res); delete m; delete n; delete res;

  23. Functions • static IMPORT_C CNokiaCVMatrix * NewL (const TInt aRows, const TInt aCols) Creates a new matrix according to size given with initial value zero. • IMPORT_C CNokiaCVMatrix * Inverse () const Calculates the inverse matrix of this matrix. • IMPORT_C CFixed ** Eigenvalues () Calculates the eigenvalues of this matrix. • IMPORT_C CFixed * Singularvalues () Calculates the singular values of this matrix using ncvSVD.

  24. CSVD Class • Singular value decomposition of a given real matrix. • Householder bidiagonalization and a variant of the QR algorithm are used. • The SVD algorithm is derived from JAMA/TNT, the Java Matrix Package. • Usage: CNokiaCVMatrix *a = CNokiaCVMatrix::NewL(rows,cols); CSVD *svd; TRAPD(err, svd = CSVD::NewL(*a)) ... //Handle possible errors CNokiaCVMatrix *v = svd->V(); ... delete a; delete svd; delete v;

  25. CNokiaCVVector Class • Fixed point representation of a vector. • class CNokiaCVVector : public CNokiaCVMatrix { … • IMPORT_C TInt Length () Getter of the vector size/length. • IMPORT_C CNokiaCVVector * Normalize () Creates the normalization of this vector. • IMPORT_C CNokiaCVVector * CrossProduct (CNokiaCVVector &aVector) Calculates the cross product of this and the given vector. • IMPORT_C CFixedDotProduct (CNokiaCVVector &aVector) Calculates the dot product of to same size vectors.

  26. CMotionHistoryImage Class • This class computes a motion history image (MHI) for a series of input frames. The MHI is a grayscale bitmap in which recently moved pixels are represent with bright values and "older" with darker values. Frame-by-frame the values from previous frames are decayed linearly.

  27. Functions • static IMPORT_C CMotionHistoryImage * NewL (CNokiaCVImage &aImage, const TInt aHistorySize) Construct CMotionHistoryImage from the first image of motion history. Parameters: aImageSize Size of the images. aHistorySize Maximum number of images in the history. • IMPORT_C void AddImage (CNokiaCVImage &aImage) Add an image to the motion history. • IMPORT_C void GetImage (CNokiaCVImage &aImage) Get a CNokiaCVImage containing the motion history image.

  28. COpticalFlow Class • The optical flow class: motion estimations for each pixel. This class also implements methods for calculating higher level information from the optical flow. • Friends class CCamus: Camus algorithm for estimation of optical flow from a series of bitmaps. • Member CEgoMovement Class: class for pixel motion estimation values. CFixediX Movement along the x-axis. CFixediY Movement along the y-axis.

  29. Functions • static IMPORT_C COpticalFlow * NewL (const TSize &aImageSize) Factory method for creating an instance. Parameters: • aImageSize Size of the optical flow map • IMPORT_C CEgoMovementMean () const Calculate the mean value of the optical flow. Represents tilting and panning of the camera. • IMPORT_C CFixedRotation () Calculate the mean rotation of the optical flow in the image plane.

  30. Functions • void SetCamusOpts (const TInt aCamusOpts) Set runtime parameters of the Camus motion estimation algorithm. • IMPORT_C CEgoMovement (const CFixed aX, const CFixed aY) Constructor. • IMPORT_C CFixedNorm () Calculates the euclidian norm of the movement.

  31. CImagePyramid Class • Creates a Laplacian or Gaussian image pyramid from a given bitmap. • IMPORT_C CNokiaCVImage * LevelImage (TInt aLevel) Getter for an image of certain level. Returns: • CNokiaCVImage* Pointer to the level image • static IMPORT_C CImagePyramid * NewLC (CNokiaCVImage *aImage, TInt aLevels, TImagePyramidType aType=EGaussianPyramid) Constructor. Parameters: • aImage CNokiaCVImage to construct the pyramid from aLevels How many levels in the pyramid. If the number is too large it will be truncated. aType Which type of pyramid to construct. Default = Gaussian.

  32. CImageStatistics Class • Calculating various statistics from CNokiaCVImages. CImageStatistics provides the following statistical functions for CNokiaCVImages: Histogram Mean Standard deviation Image moments: • m00 • m01 • m10 • m11 • m20 • M02 • Usage: TReal mean; CImageStatistics::Mean(*iSourceBitmap, mean, (TChannel)aChannel);

  33. Functions • static IMPORT_C void Histogram (CNokiaCVImage &aImage, THistogram &aArray, TChannel aChannel=EGrayChannel) Calculate the histogram from a CNokiaCVImage. • static IMPORT_C void Mean (CNokiaCVImage &aImage, TReal &aMean, TChannel aChannel=EAllChannels) Calculate the mean of CNokiaCVImage. static IMPORT_C void StDev (CNokiaCVImage &aImage, TReal &aStDev, TReal &aMean, TChannel aChannel) Calculate the standard deviation of a CNokiaCVImage.

  34. CImageOperations Class • Provides an simplified access to image operations. Each operation is a static method that creates and returns the result bitmap. • Usage CNokiaCVImage* iTarget = CImageOperations::Rotate(iSource, 45); ... delete iTarget;

  35. Functions • static IMPORT_C CNokiaCVImage * Resize (CNokiaCVImage *aSource, TSize aSize, TBool aBilinear=EFalse) Resizes the source image. See CResizeOp for more information. • static IMPORT_C CNokiaCVImage * Convolution (CNokiaCVImage *aSource, CNokiaCVMatrix &aKernel, TInt aNormFactor) Performs convolution of CNokiaCVImage and a kernel (CNokiaCVMatrix). • static IMPORT_C CNokiaCVImage * Gaussian (CNokiaCVImage *aSource, TUint aSize) Gaussian smoothing filter. • static IMPORT_C CNokiaCVImage * EdgeDetectLaplacian (CNokiaCVImage *aSource) Laplacian edge detection filter. See CEdgeDetectionOp for more info.

  36. CCamus Class • Implements the Camus algorithm for estimation of optical flow from a series of bitmaps. This class creates an COpticalFlow object that contains an estimation of the motion vector (x and y motion in pixels) for each pixel of the input frame. • Can use the methods of COpticalFlow to extract higher level information such as estimated camera rotation/movement. • The algorithm is highly parameterizable. • Features of the algorithm and some hints: • Maximum estimated motion is 1 or 2 pixels/frame, so it is typically reasonable to use small bitmaps (e.g. 16x12 pixels). • History of several frames is used to achieve sub pixel accuracy • Based on template matching • Uses grayscale images • Large motions are hard so having a good frame rate is essential since the per-frame motion is inversely proportional to the frame rate • Small bitmaps are preferred (especially when estimation of quick movements is required) as its faster to compute; higher frame rate.

  37. Camus Algorithm Basic idea: • Simple technique to perform optical flow computations using correlation or patch-making methods. The idea here is to match features between images and computing the SD (sum-difference) of patches. • Reference: T. Camus, Real-Time Optical Flow, PhD Thesis, Brown University Technical Report CS-94-36, 1994.

  38. Functions static IMPORT_C CCamus* CCamus::NewL (CNokiaCVImage & aImage, const TInt aHistorySize = KDefaultCamusHistorySize ) IMPORT_C void CCamus::GetOpticalFlow (COpticalFlow & aOpticalFlow, CNokiaCVImage & aImage, const TInt aCamusOpts = KDefaultCamusOpts )

  39. How to Use CCamus COpticalFlow* iOpticalFlow = COpticalFlow::NewL(iFirstFrame->Size()); CNokiaCVImage* iCamus = CCamus::NewL(*iFirstFrame, 5); iCamus->GetOpticalFlow(*iOpticalFlow, *iSecondFrame, CAMUS_SETCAMUSOPTS(CCamus::EMedium, CCamus::EMedium, 3)); CEgoMovement mean = iOpticalFlow->Mean();

  40. CEgoMovement Class • Simple vector-like wrapper class for pixel motion estimation values.

  41. Functions • IMPORT_C CEgoMovement::CEgoMovement (const CFixedaX, const CFixedaY )Constructor. Parameters: • aX Movement along the x-axis. • aY Movement along the y-axis. • IMPORT_C CFixed CEgoMovement::Norm ( ) Calculates the Euclidian norm of the movement. Returns: • Norm (the length of the movement in cartesian coordinate system). • IMPORT_C CFixed CEgoMovement::DotProduct (const CEgoMovement & aMovement  ) Calculates a dot product. Parameters: • aMovement Vector movement. Returns: • Dot product value.

  42. CEigenValues Class • Eigenvalue decomposition of a real matrix. Usage: CNokiaCVMatrix * matrix = ...; CEigenvalues * eigs; TRAPD(err, eigs = CEigenvalues->NewL(matrix)); if(err != KErrNone) { //eigs not found do something } else { TReal * real = eigs->RealEigenvalues(); delete eigs; } ...

  43. Functions • IMPORT_C TReal* CEigenvalues::RealEigenvalues ( ) Getter for the real part of the eigenvalues (iD). Returns: • the real part of the eigenvalues in an array • IMPORT_C TReal* CEigenvalues::ImaginaryEigenvalues ( ) Getter for the imaginary part of the eigenvalues (iE). Returns: • the imaginary part of the eigenvalues in an array

  44. CFixed Class • Fixed point class. • The class implements basic arithmetic for fixed point numbers. • CFixed can be used instead of TReal when performance more critical than accuracy.

  45. CImageOp Class • Image operations interface. • This class represents an interface that all image operation classes must implement. Most of the operations implementing this interface are also included in CImageOperations. • There are various image operations that implement this interface, e.g. for rotating, scaling and filtering bitmap. Usage example (CRotationOp): iSource = .... iTargetMode = EGray256; iTarget1 = CNokiaCVImage::NewL(); iTarget1->CreateL(iCurrSource->Size(), iTargetMode); CRotationOp op(iCurrSource, 90); op.DoOperation(iTarget1); ...

  46. Functions • virtual IMPORT_C void CImageOp::DoOperation (CNokiaCVImage * aTarget  ) [pure virtual] • All image operations must implement this method. This method does the actual operation. • Typically you need to allocate and create the target bitmap yourself before doing the operation • Parameters: • aTarget target bitmap • Note: • the function can leave • Implemented in CArithmeticOp, CColorConversionOp, CComparisonOp, CConvolutionOp, CCornerDetectionOp, CEdgeDetectionOp, CGaussianSmoothOp, CMedianFilterOp, CMorphologicalOp, CResizeOp, CRotationOp, CShearOp, CTemplateMatchOp, CThresholdOp, and CWarpOp.

  47. CArithmeticOp • CArithmeticOp provides basic arithmetic operations for CNokiaCVImages. • Target and source bitmaps must be of same size. • Usage: IMPORT_C void CArithmeticOp::DoOperation (CNokiaCVImage * aTarget  )  Does the actual operation. Parameters: • aTargetCNokiaCVImage to store the result in. Must be allocated and created beforehand. Implements CImageOp.

  48. Image Arithmetic Examples

  49. CColorConversionOp Class • Provides CNokiaCVImage with color conversions.

  50. Functions • IMPORT_C CColorConversionOp::CColorConversionOp (CNokiaCVImage * aImage, TDisplayMode aTargetMode ) • Constructor. Converts given image to given display mode. • Parameters: • aImage Image to convert aTargetMode Target display mode Valid TDisplayModes for aTargetMode are: • EGray2 • EGray4 • EGray16 • EGray256 • EColor4K • EColor64K • EColor16M • IMPORT_C void CColorConversionOp::DoOperation (CNokiaCVImage * aTarget  )  • Parameters: • aTargetCNokiaCVImage to store the result in.

More Related