User:Jhlin/Camera2

This page documents my findings for bug 1069246 [B2G] Evaluate new Android camera HAL & stagefright ProCamera API for getUserMedia()

Background

In Android 5 Lollipop (API version 21), Google introduced a new camera Java API (android.hardware.camera2) and deprecated the one it had been provided since the beginning of Android history (Camera). Along with this API, new Android camera HAL interface (HAL v3) were created and the camera service built on top of that extended. Some (internal/WIP) native API (camera2) is also found in current Android code base already.

The old HAL was designed as a black box with high-level controls targeting conventional camera apps (preview, video recording, and still capture) and makes it difficult, if not impossible, to implement new type of features such as burst mode without extending HAL by vendors.

HAL v3, on the other hand, remodels a camera device as a pipeline that receive capturing requests and produce one result per request[1]. One advantage, among many others, of this design is each request carries its own capture parameters. This way multiple series of requests can be served concurrently (from users' point of view) and each will have its own output (image + metadata) stream.

Since Gonk also uses the camera HAL & service, in theory it's possible to implement mozCamera and getUserMedia()(gUM) with HAL v3 on supporting devices.

[2] describes the architecture of Android camera subsystem.

[3] shows the components involved in camera2 API.

Current gUM Implementation

The following diagram briefly shows components used to implement gUM:

<graphviz> digraph current {

 node [shape=box];
 gUM [label="getUserMedia()"];
 mgr [label="mozilla::MediaManager"];
 eng [label="mozilla::MediaEngine"];
 msg [label="mozilla::MediaStream"];
 mozCam [label="mozilla::ICameraControl"];
 gonkCam [label="GonkCamera*"];
 andCam [label="android::Camera"];
 camSrv [label="android::ICameraService"];

 gUM -> mgr -> eng -> mozCam -> gonkCam -> andCam -> camSrv;
 gUM -> msg;
 eng -> msg;

} </graphviz>

MediaEngine, acting as a camera preview frame event listener, will receive images from camera preview stream and send them to MediaStream for its video outputs and listeners to consume. While this works, on B2G the buffer starvation problem is quite serious [4]. 2 ways to avoid the problem are:

1. Bypass MediaStreamGraph[5].
2. Make a copy of image[6].

Possible Implementation on Top of camera2

It might look like this if using camera2: <graphviz> digraph cam2 {

 node [shape=box];
 gUM [label="getUserMedia()"];
 mgr [label="mozilla::MediaManager"];
 eng [label="mozilla::MediaEngine"];
 msg [label="mozilla::MediaStream"];
 cam2 [label="android::ICameraDeviceUser"];
 camSrv [label="android::ICameraService"];

 gUM -> mgr -> eng;
 gUM -> msg;
 eng -> msg;
 eng -> camSrv;
 eng -> cam2;
 eng -> GonkNativeWindow;

} </graphviz>

MediaEngine connects to camera service and use camera2 API to receive camera frames through GonkNativeWindow (as image buffers).

camera2 Benefits

• Android has a standard tunneling mechanism that almost all hardware accelerated APIs support. It looks like:

<graphviz> digraph tunnel {

 graph [rankdir="LR", splines=curved];
 node [shape=box];
 BufferQueue -> IGraphicBufferProducer [label="dequeue"];
 IGraphicBufferProducer -> BufferQueue [label="queue"];
 BufferQueue -> IGraphicBufferConsumer [label="aquire"];
 IGraphicBufferConsumer -> BufferQueue [label="release/cancel"];

} </graphviz>

For example, MediaRecorder implementation on B2G could use MediaCodec::createInputSurface() to set up a buffer queue whose consumer is the encoder and pass the created 'surface' to camera as a image buffers source. Since both camera service and encoder run in the media server process, content process doesn't have to transfer data at all.

• Multiple concurrent requests and output buffers support. This is can be used to build camera apps with advanced features such as HDR viewfinder or raw image capture.

Challenges of implementing gUM using camera2

• Few supporting devices
• camera2 API is still in development and is likely to change in the future
• To fully support dropping frames according to CPU load or network condition (in WebRTC), frame rate control must be done in media engine or media stream graph. Asynchronous pulling perhaps?

Checking Camera HAL Version

$ adb shell dumpsys media.camera
...
Camera module API version: 0x203
...
Camera 0 static information:
 ...
 Device version: 0x302

Module version should be 0x2?? and device version 0x3?? when HAL v3 is supported.