Getting a Mask with ML Kit's Selfie Segmentation API in an Android Camera2 Image Reader App

Getting Masks with Selfie Segmentation API on Android: An ImageReader Approach

In this article, we will discuss how to use the Selfie Segmentation API on Android to get masks. Specifically, we will focus on the ImageReader approach instead of the ImageAnalysis use-case. We will cover the key concepts and provide detailed context on this topic, including subtitles and code blocks.

What is Selfie Segmentation API?

Selfie Segmentation API is a machine learning model that can be used to segment a person from a background in a photo. It is part of the Google's ML Kit, a suite of pre-trained models for common mobile vision use-cases. The API can detect and segment people in real-time, making it useful for a variety of applications, such as social media apps, games, and photography apps.

Why Use ImageReader instead of ImageAnalysis?

The ImageAnalysis use-case is the recommended way of using the Selfie Segmentation API on Android. However, there are situations where using the ImageReader approach may be more suitable. For instance, if you need full control over the image capture process, including frame rate and resolution, using the ImageReader approach is a good option.

Setting Up the ImageReader

To use the ImageReader approach, you need to create an ImageReader object. The ImageReader is responsible for reading images from the camera. You can specify the image format, size, and number of buffers. For example:

mImageReader = ImageReader.newInstance(mPreviewSize.getWidth(), mPreviewSize.getHeight(), ImageFormat.YUV_420_888, 2);

Linking the ImageReader to the Camera

Once you have created the ImageReader object, you can link it to the camera. You can use the CameraDevice.createCaptureSession method to do this. Here is an example:

CameraDevice cameraDevice = ...;
CaptureRequest.Builder previewBuilder = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
previewBuilder.addTarget(mImageReader.getSurface());
cameraDevice.createCaptureSession(Arrays.asList(mImageReader.getSurface()), new CameraCaptureSession.StateCallback() {
...
}, null);

Processing the Images

Once the CameraCaptureSession is created, the ImageReader will start receiving images. You can use the ImageReader.OnImageAvailableListener to process the images. Here is an example:

mImageReader.setOnImageAvailableListener(new ImageReader.OnImageAvailableListener() {
@Override
public void onImageAvailable(ImageReader reader) {
Image image = reader.acquireLatestImage();
if (image == null) {
return;
}
// process the image
image.close();
}
}, null);

Using the Selfie Segmentation API

To use the Selfie Segmentation API, you need to create a Segmentation mask request. Here is an example:

FirebaseVisionImage image = FirebaseVisionImage.fromMediaImage(image, rotation);
SegmentationMaskOption option = new SegmentationMaskOption.Builder()
.setConfidenceThreshold(0.5f)
.build();
SegmentationMaskRequest request = new SegmentationMaskRequest.Builder(image)
.setSegmentationMaskOption(option)
.build();

You can then use the FirebaseVision.getVisionImageDetector to process the request and get the mask.

• Selfie Segmentation API is a machine learning model that can be used to segment a person from a background in a photo
• To use the ImageReader approach, create an ImageReader object and link it to the camera
• Use the ImageReader.OnImageAvailableListener to process the images
• Create a Segmentation mask request and process it using FirebaseVision.getVisionImageDetector

References

• Firebase ML Kit