Camera Enabling on MeeGo from Cheese to Mobility camera

1. Hello every one, welcome to this session We will talk about camera enabling on different MeeGo platform. It covers: the generic camera of cheese on Netbook, camera connecting with HW encoder on tablet/handset platform Qt Mobility camera interface is also covered, it hides technical details to accelerate application development.Hello every one, welcome to this session We will talk about camera enabling on different MeeGo platform. It covers: the generic camera of cheese on Netbook, camera connecting with HW encoder on tablet/handset platform Qt Mobility camera interface is also covered, it hides technical details to accelerate application development.

2. Agenda Camera background overview Cheese Camera (SHMEM but always copy) Moorestown Camera (SHMEM with HW encoder) Medfield Camera (USERPTR with HW encoder) Camera of Qt Mobility Here is the Agenda: we will give an overview first then camera with different technology on Netbook/Tablet/Handset is covered. finally, we also give short introduction for camera interface in Qt Mobility, it hides technology details to accelerate application development.Here is the Agenda: we will give an overview first then camera with different technology on Netbook/Tablet/Handset is covered. finally, we also give short introduction for camera interface in Qt Mobility, it hides technology details to accelerate application development.

3. Agenda Camera(v4l2) overview Cheese Camera (SHMEM but always copy) Moorestown Camera (SHMEM with HW encoder) Medfield Camera (USERPTR with HW encoder) Camera of Qt Mobility

4. V4l2: data exchanging modes between driver and application Video for Linux (V4L) defines camera subsystem on Linux. Basing on data exchanging modes between driver and app, there are three modes of V4L2 READ_WRITE mode: it explicitly copy video data from camera driver to application. It is time consumption. MMAP mode: Video data is shared between driver and app USERPTR mode: application alloc the memory for video, driver write data to this memory directly.Video for Linux (V4L) defines camera subsystem on Linux. Basing on data exchanging modes between driver and app, there are three modes of V4L2 READ_WRITE mode: it explicitly copy video data from camera driver to application. It is time consumption. MMAP mode: Video data is shared between driver and app USERPTR mode: application alloc the memory for video, driver write data to this memory directly.

5. MeeGo Camera Stack Here is a general camera stack in MeeGo Gstreamer is used to wrap driver functionality camera service in Qt Mobility manage camera pipeline and operations. camera app is created on the API from Qt Mobility, the interface is pretty simple & straightforward.Here is a general camera stack in MeeGo Gstreamer is used to wrap driver functionality camera service in Qt Mobility manage camera pipeline and operations. camera app is created on the API from Qt Mobility, the interface is pretty simple & straightforward.

6. Evolution of camera in Linux V4lsrc: 2004 (gst-good) V4l2src: 2006 (gst-good) Cheese: 2007 Camerabin: 2009 Camerabin2: 2010 Qt Mobility Camera: 2010 It tries to hide technical details to accelerate camera app development I want to use this page to share with you that: Camera changes a lot on Linux in recent year, we are experiencing the evolutions. Cheese is default camera in many Linux distribution, it is created in 2007. Camerabin2 and Camera of Qt Mobility are created in 2010, they are pretty young.I want to use this page to share with you that: Camera changes a lot on Linux in recent year, we are experiencing the evolutions. Cheese is default camera in many Linux distribution, it is created in 2007. Camerabin2 and Camera of Qt Mobility are created in 2010, they are pretty young.

7. Agenda Camera(v4l2) overview Cheese Camera (SHMEM but always copy) Moorestown Camera (SHMEM with HW encoder) Medfield Camera (USERPTR with HW encoder) Camera of Qt Mobility Now, let�s look at camera on NetBook, it is Cheese. Though most netbook camera driver supports SHMEM, however, v4l2src always copies video data to user space after memory map. Camera pipeline works similar to READ_WRITE mode. I think v4l2src copies video data to ease the buffer management, because when you are using SHMEM you have to manage the buffer�s life cycle in the whole camera pipelineNow, let�s look at camera on NetBook, it is Cheese. Though most netbook camera driver supports SHMEM, however, v4l2src always copies video data to user space after memory map. Camera pipeline works similar to READ_WRITE mode. I think v4l2src copies video data to ease the buffer management, because when you are using SHMEM you have to manage the buffer�s life cycle in the whole camera pipeline

8. Cheese http://projects.gnome.org/cheese/ http://projects.gnome.org/cheese/ uses your webcam to take photos and videos, applies fancy special effects and lets you share the fun with others. http://projects.gnome.org/cheese/ uses your webcam to take photos and videos, applies fancy special effects and lets you share the fun with others.

9. Cheese optimization on Netbook Reasonable resolution and frame rate(hw capability and videorate) alsasrc vs pulsesrc Audio encoder (Vorbis/FLAC, Float/Integer) Image effect and capability query tuning for latency When we try to take care of cheese on Netbook, it almost doesn�t work.� almost 2 third frames are lost, audio and video are not sync. We try to tune it balance quality and performance. we have to use reasonable resolution and framerate. Cheese tried to used the highest resolution and fps before, it is not correct, some camera only support a fix fps of 30. we can use video rate to low it down alsasrc also performs much better than pulsesrc, about 8% vs 20%CPU, pulsesrc try to do some conversion and re-sampling. Which is not important for camera. different audio encoder type also make difference, when there is no FPU, FLAV performs much better than vorbis we also did other tuning to reduce some switch latency, to avoid unnecesary image effect filters, try to cache camera capability since there is few chance that it changes.When we try to take care of cheese on Netbook, it almost doesn�t work.� almost 2 third frames are lost, audio and video are not sync. We try to tune it balance quality and performance. we have to use reasonable resolution and framerate. Cheese tried to used the highest resolution and fps before, it is not correct, some camera only support a fix fps of 30. we can use video rate to low it down alsasrc also performs much better than pulsesrc, about 8% vs 20%CPU, pulsesrc try to do some conversion and re-sampling. Which is not important for camera. different audio encoder type also make difference, when there is no FPU, FLAV performs much better than vorbis we also did other tuning to reduce some switch latency, to avoid unnecesary image effect filters, try to cache camera capability since there is few chance that it changes.

10. Cheese performance after optimization on NetBook After tuning, cheese works pretty well on Nextbook with VGA resolution of 15 fps. Here is a comparison before and after tuningAfter tuning, cheese works pretty well on Nextbook with VGA resolution of 15 fps. Here is a comparison before and after tuning

11. Agenda Camera(v4l2) overview Cheese Camera (SHMEM but always copy) Moorestown Camera (SHMEM with HW encoder) Medfield Camera (USERPTR with HW encoder) Camera of Qt Mobility

12. Camera with HW encoder (Moorestown SHMEM) On MRST, HW video encoder is introduced to improve performance. SHMEM is really used between camera driver and encoder (video driver). It is also used between video driver and gfx/displayer driver for view-finding. The encoder also takes some responsibility of filters to make the pipeline tight, and make the buffer management easy.On MRST, HW video encoder is introduced to improve performance. SHMEM is really used between camera driver and encoder (video driver). It is also used between video driver and gfx/displayer driver for view-finding. The encoder also takes some responsibility of filters to make the pipeline tight, and make the buffer management easy.

13. Camera with HW encoder (Moorestown SHMEM) Here is the camera pipeline for MRST Camera src(driver) and video encoder (driver) share the video frame by a frame ID. Video frame is also shared between video and gfx driverHere is the camera pipeline for MRST Camera src(driver) and video encoder (driver) share the video frame by a frame ID. Video frame is also shared between video and gfx driver

14. Moorestown Camera Stack Here is the full stack of mrst camera. Here is the full stack of mrst camera.

15. Agenda Camera(v4l2) overview Cheese Camera (SHMEM but always copy) Moorestown Camera (SHMEM with HW encoder) Medfield Camera (USERPTR with HW encoder) Camera of Qt Mobility

16. Camera with HW encoder (Medfield with USERPTR)

17. Camera with HW encoder (Medfield with USERPTR) Basing on �User pointer� mode, Medfield camera pipeline works similar to web camera Camerabin2 brings more improvements for shutter/switch latency. (more detail is in camerabin2 part GStreamer Wiki: CameraBin2)

18. Agenda Camera(v4l2) overview Cheese Camera (SHMEM but always copy) Moorestown Camera (SHMEM with HW encoder) Medfield Camera (USERPTR with HW encoder) Camera of Qt Mobility

19. Camera of Qt Mobility Overview

20. Camera of Qt Mobility View Finding API camera = new QCamera; viewFinder = new QVideoWidget(); viewFinder->show(); camera->setViewfinder(viewFinder); camera->start();

21. Camera of Qt Mobility Image Capture API camera = new QCamera; imageCapture = new QCameraImageCapture(camera); camera->setCaptureMode(QCamera::CaptureStillImage); camera->start(); imageCapture->capture();

22. Camera of Qt Mobility Video Recording API camera = new QCamera; mediaRecorder = new QMediaRecorder(camera); camera->setCaptureMode(QCamera::CaptureVideo); camera->start(); mediaRecorder->record();

23. Questions?