1 / 58

Framework for digital camera in Linux

Framework for digital camera in Linux. Dongsoo Kim ( dongsoo45.kim@samsung.com ) Heungjun Kim( riverful.kim@samsung.com ) Samsung Electronics. Contents. Introducing mobile camera devices SoC camera subsystem Mobile camera module devices Framework for digital camera

marietta-anglim
Download Presentation

Framework for digital camera in Linux

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Framework for digital camerain Linux Dongsoo Kim (dongsoo45.kim@samsung.com) Heungjun Kim(riverful.kim@samsung.com)Samsung Electronics

  2. Contents • Introducing mobile camera devices • SoC camera subsystem • Mobile camera module devices • Framework for digital camera • Video4Linux2 and camera device • New APIs for high-end mobile camera devices • Dual camera framework for 3G handset • Porting issues in SoC camera subsystem

  3. Terms • AP : Application Processor • Camera interface : interface device which is syncing image data from external camera device and gives some commands to them • Camera module : external camera device packaged with lens, sensor and ISP on it • ISP : Image Signal Processor. Also supports lots of additional functions • V4L2 : Video for Linux 2

  4. This document contains some animations. Slideshow mode is recommended

  5. What makes camera “a camera”? • Strobe support • Various program mode : slow sync, red-eye reduction and on…. • Exposure control • Aperture (Iris) • Shutter • ISO • Lens control • Focus • Zoom

  6. Introducing mobile camera devices

  7. History of camera phone First Camera Phone in 2000 with 3.5MP 1.3MP in 2003 5MP in2004 3MP in2004 7MP in 2005 2MP in 2004 10MP in 2006 12MP in 2009

  8. Mobile camera module devices • Old and low-end mobile camera modules

  9. Mobile camera module devices • Brand new and high-end mobile camera modules

  10. SoC camera subsystem • What is SoC camera subsystem? ARM SoC solution Peripherals External Camera module Camera interface Peripheral3 (USB) ARM Core Peripheral2 (MMC) Peripheral1(Display)

  11. SoC camera subsystem Request for data Image data in expected format Camera interface Command interface Resolution handling Data format handling (YUV/RAW/JPEG…) Buffer handling I2C RAW DATA Camera module Exposure control Lens control White balance control Effect control Face detect Zoom control Strobe control . . .

  12. SoC camera subsystem • Various ways of preview (Live view) in SoC camera subsystems User Application ARM core COPY! M e m o r y Frame buffer DATA Camera Interface DATA Camera Module OK Cancel

  13. SoC camera subsystem • Various ways of preview (Live view) in SoC camera subsystems User Application ARM core M e m o r y COPY! Save as file DATA D M A Camera Interface DATA Camera Module Frame buffer Through DMA OK Cancel

  14. SoC camera subsystem • How to capture still shots in SoC camera subsystem • Your camera module has a dedicated JPEG encoder on it ARM core User Application Camera module DATA COPY! RAW DATA PROCESS JPEG PROCESS M e m o r y DATA Save as JPEG file DATA Camera Interface MEMORY

  15. SoC camera subsystem • How to capture still shots in SoC camera subsystem • No JPEG encoder in your camera module User Application ARM core COPY! RAW DATA M e m o r y DATA Codec S/W Camera Interface DATA Camera Module Save as JPEG file

  16. SoC camera subsystem • How to take motion pictures in SoC camera subsystem COPY! Frame buffer ARM core User Application M e m o r y OK Cancel DATA Codec S/W Save as… Camera Interface DATA Camera Module COPY!

  17. What we’ve got in Linux to control camera • In Kernel • V4L2 (AKA Video For Linux 2) • The second version of the Video For Linux API • Kernel interface for analog radio and video capture and output drivers • In user space • Multimedia middleware • Gstreamer • Pipeline based multimedia framework written in the C programming language with the type system based on GObject • OpenMAX • Cross-platform set of C-language programming interfaces that provides abstractions for routines especially useful for audio, video, and still images

  18. Mobile camera module devices Is V4L2 enough for mobile digital camera modules? No way

  19. Framework for digital camera

  20. Video4Linux2 and camera device • Camera subsystem in Linux kernel aspect videobuf videodev V4L2 V4l2-dev V4l2-ioctl V4l2-int-device (old) V4l2-subdev (new) videobuf-core videobuf-sg-dma V4l2-device Camera interface V4L2 driver Command interface Resolution handling Data format handling (YUV/RAW/JPEG…) Buffer handling Camera module V4L2 driver Exposure control Lens control White balance control Effect control Face detect Zoom control Strobe control . . .

  21. Abstract view of V4L2 working with video capture device • How does it work with V4L2? Application or Middleware Open device Copy memory to userspace VIDIOC_QUERYCAP VIDIOC_S_INPUT VIDIOC_REQBUF VIDIOC_QBUF VIDIOC_QBUF VIDIOC_DQBUF VIDIOC_QUERYBUF VIDIOC_STREAMON

  22. New V4L2 APIs for digital camera Exposure control

  23. New V4L2 APIs for exposure control • Need for detailed control in exposure • We’ve got camera devices with mechanical shutter • Iris could be handled also • ISO controlled output is somehow useful Why not using enhanced exposure control like a regular digital camera?

  24. New V4L2 APIs for exposure control • What we’ve got in mainline V4L2 right now? #define V4L2_CID_EXPOSURE_AUTO (V4L2_CID_CAMERA_CLASS_BASE+1) enum  v4l2_exposure_auto_type {         V4L2_EXPOSURE_AUTO = 0,         V4L2_EXPOSURE_MANUAL = 1,         V4L2_EXPOSURE_SHUTTER_PRIORITY = 2,         V4L2_EXPOSURE_APERTURE_PRIORITY = 3 }; #define V4L2_CID_EXPOSURE_ABSOLUTE  (V4L2_CID_CAMERA_CLASS_BASE+2) #define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3) • No way to control iris • No way to control exposure metering method

  25. New V4L2 APIs for exposure control • Additional exposure control enum v4l2_exposure_auto_type { V4L2_EXPOSURE_AUTO = 0, V4L2_EXPOSURE_MANUAL = 1, V4L2_EXPOSURE_SHUTTER_PRIORITY = 2, V4L2_EXPOSURE_APERTURE_PRIORITY = 3, + /* Additional features for digital camera */ + V4L2_EXPOSURE_ISO_PRIORITY =4, }; #define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2) #define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3) . . . +#define V4L2_CID_EXPOSURE_LOCK (V4L2_CID_CAMERA_CLASS_BASE+18)

  26. New V4L2 APIs for exposure control • Supported aperture stages, shutter speed, and ISO speed could be different between every different camera module products +#define V4L2_CID_CAM_APERTURE(V4L2_CID_CAMERA_CLASS_BASE+19) +#define V4L2_CID_CAM_SHUTTER (V4L2_CID_CAMERA_CLASS_BASE+20) +#define V4L2_CID_CAM_ISO (V4L2_CID_CAMERA_CLASS_BASE+21) Supported values for aperture, shutter and ISO should be made in V4L2_CTRL_TYPE_MENU {     .id = V4L2_CID_CAM_APERTURE,     .type = V4L2_CTRL_TYPE_MENU,     .name = "Aperture",     .minimum = 0,     .maximum = 4,     .step = 1,     .default_value = 0, }, static const char *camera_iris_stages[] = { /* This module supports 5 Iris stages  * on it's own but F number depends  * how you package Lens module */ "2.8“, "4“, "5.6“, "8“, "11“, NULL };

  27. New V4L2 APIs for exposure control • New exposure metering control +/* Exposure Methods */ +#define V4L2_CID_PHOTOMETRY (V4L2_CID_CAMERA_CLASS_BASE+17) +enum v4l2_photometry_mode { + V4L2_PHOTOMETRY_MULTISEG = 0, + V4L2_PHOTOMETRY_CWA = 1, + V4L2_PHOTOMETRY_SPOT = 2, + V4L2_PHOTOMETRY_AFSPOT = 3, +};

  28. New V4L2 APIs for digital camera Lens control

  29. New V4L2 APIs for lens control • Need for enhanced Focus mode API • Current V4L2 framework just slightly covers MANUAL & AUTO FOCUS mode for usb webcam • High-end camera modules support for various focus mode presets like • AF Macro • AF-S / AF-C For compatibility Focus control API should be defined

  30. New V4L2 APIs for lens control • Focus Mode & Enumeration values #define V4L2_CID_FOCUS_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+10) #define V4L2_CID_FOCUS_RELATIVE (V4L2_CID_CAMERA_CLASS_BASE+11) +#define V4L2_CID_FOCUS_MODE (V4L2_CID_CAMERA_CLASS_BASE+13) +/* Focus Methods */ +enum v4l2_focus_mode { + V4L2_FOCUS_AUTO = 0, + V4L2_FOCUS_MANUAL = 1, + V4L2_FOCUS_MACRO = 2, + V4L2_FOCUS_CONTINUOUS = 3, +};

  31. New V4L2 APIs for digital camera Object recognition

  32. What is object detection? • Object detection Gee! It recognized my face! Hello everyone

  33. What should we consider? • What do we set? • Detect mode • Type of object • How many objects to detect • Triggered action • What do we get? • How many objects are detected • Detected object’s type • Detected object’s coordinates

  34. New V4L2 APIs for object recognition • New API for object recognition • New capability #define V4L2_CAP_ASYNCIO 0x02000000 /* async I/O */ #define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */ +#define V4L2_CAP_OBJ_RECOGNITION 0x10000000 • New IOCTL +#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect) +#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)

  35. New V4L2 APIs for object recognition • New API for object detection +/* Object detection and triggered actions */ +enum v4l2_recog_mode { + V4L2_RECOGNITION_MODE_OFF = 0, + V4L2_RECOGNITION_MODE_ON = 1, + V4L2_RECOGNITION_MODE_LOCK = 2, +}; + +enum v4l2_recog_action { + V4L2_RECOG_ACTION_NONE = 0,/* only detection */ + V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */ + V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */ +}; + +enum v4l2_recog_pattern { + V4L2_RECOG_PATTERN_FACE = 0; /* Face */ + V4L2_RECOG_PATTERN_HUMAN = 1; /* Human */ + V4L2_RECOG_PATTERN_CHAR = 2; /* Character */ +}

  36. New V4L2 APIs for object recognition • New API for object detection +struct v4l2_recog_rect { + enum v4l2_recog_pattern p; /* What pattern detected */ + struct v4l2_rect o; /* detected area */ + __u32 reserved[4]; +} + +struct v4l2_recognition_data { + __u32 detect_cnt; /* detected object counter */ + struct v4l2_recog_rect obj; /* detected area */ + __u32 reserved[4]; +}; + +structv4l2_recognition { + enum v4l2_recog_mode mode; + enum v4l2_recog_pattern pattern; /* What pattern to detect */ + __u32 obj_num; /* How many object to detect */ + __u32 detect_idx; /* select detected object */ + struct v4l2_recog_data data; /* read only */ + enum v4l2_recognition_action action; + __u32 reserved[4]; +};

  37. What should we consider? • What do we set? • Detect mode • How many objects to detect • Triggered action • What do we get? • How many objects are detected • Detected object’s coordinates

  38. New V4L2 APIs for object recognition • New API for object recognition • New capability #define V4L2_CAP_ASYNCIO 0x02000000 /* async I/O */ #define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */ +#define V4L2_CAP_OBJ_RECOGNITION 0x10000000 • New IOCTL +#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect) +#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)

  39. New V4L2 APIs for object recognition • New API for object detection +/* Object detection and triggered actions */ +enum v4l2_recognition_mode { + V4L2_RECOGNITION_MODE_OFF = 0, + V4L2_RECOGNITION_MODE_ON = 1, + V4L2_RECOGNITION_MODE_LOCK = 2, +}; + +enum v4l2_recognition_action { + V4L2_RECOG_ACTION_NONE = 0,/* only detection */ + V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */ + V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */ +}; + +struct v4l2_recognition_data { + __u8 detect_cnt; /* detected object counter */ + struct v4l2_rect o; /* detected area */ +}; + +structv4l2_recognition { + enum v4l2_recognition_mode mode; + __u8 obj_num; /* How many object to detect */ + __u8 detect_idx; /* select detected object */ + struct v4l2_recognition_data data; /* read only */ + enum v4l2_recognition_action action; +};

  40. New V4L2 APIs for digital camera Dual camera

  41. Dual camera for 3G handset • What is dual camera? • Dual camera applications • Video telephony • Camera applications • Taking self portrait shots Front side camera : for special purpose Flip side camera : for still shots

  42. Dual camera for 3G handset • H/W restrictions in Application Processor’s camera peripheral interface Mega camera VGA camera Data path

  43. Why should we make single device node? • H/W restrictions in Application Processor’s camera peripheral interface “SINGLE” camera interface on Application Processor ! Only one camera could be handled at a time Only one camera could be using data pins at a time Each camera is different in Working MCLK Working resolution (SYNC) Working PCLK

  44. What do we have in V4L2? • We already have API for dual camera • Enumerate video inputs •  Query or select the current video input VIDIOC_ENUMINPUT VIDIOC_G_INPUT VIDIOC_S_INPUT

  45. What do we need to consider for dual camera? • What do we need to consider? • Let user space application get noticed about camera input devices • Don’t forget VIDIOC_ENUMINPUT • Check out supported pixel format using VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES • Context of each camera module • Capability : Supported color spaces, resolutions, additional functions… • Latest working status : resolution, color space, effect and so on • Context of camera interface • Working information of the camera module getting switched : Clock information and so on

  46. Example of switching camera Megacamera VGAcamera • How to switch active camera Mega Enable VGA Enable • Condition 1 • How to enable camera module: • Give power source • Enable expecting camera • Give MCLK Mega Reset VGA Reset CAM POWER MCLK DATA PINS CPU • Condition 2 • Shared things • Camera power enable pin • Camera data pins • MCLK • Condition 3 • Dedicated things • Camera enable pins • Camera reset pins

  47. Abstract work flow of switching camera Cam power off Camera (A) enable pin off Camera (B) Enable pin on Cam power on Switch to camera (B) STREAMON STREAMOFF Camera (A) initialize Start preview Stop preview G_INPUT/ S_INPUT Camera (B) initialize Start preview Camera interface reset & setting Stop preview Stop DMA Reset camera interface’s current setting Make a proper MCLK for camera (B) Setup resolution, color-space, DMA

  48. Porting issues in SoC camera subsystem

  49. I2C issue • I2C protocol has start and stop condition • Some camera modules generate unexpected noise until their power up sequence • Makes I2C bus in bus busy state

  50. I2C issue • How to prevent this malfunctioning issue Use a “noise free on power up” camera module OR Use a level shifter to block and ignore noise from camera module OR Change functionality of I2C pins while turning on camera module

More Related