npm install @mediapipe/tasks-vision

The segmenter plugin uses MediaPipe's Image Segmenter and exposes ShaderPad mask textures and GLSL helper functions.

import segmenter from 'shaderpad/plugins/segmenter'

const shader = new ShaderPad(fragmentShaderSrc, {
  plugins: [
    segmenter({
      textureName: 'u_webcam',
      options: { outputConfidenceMasks: true },
    }),
  ],
})

The plugin reads from the texture named by textureName. Initialize and update that exact ShaderPad texture name, or the detector will have no live source to read from.

Options

Events

Subscribe with shader.on(name, callback).

shader.on('segmenter:result', result => {
  console.log(result.categoryMask)
})

Uniforms

u_segmentMask stores:

• R: confidence for the winning category at that pixel
• G: normalized category index in the range 0.0 to 1.0

If you need the integer category index in GLSL, recover it with:

int categoryIndex = int(floor(segment.y * float(u_numCategories - 1) + 0.5));

Helper Functions

If history is enabled, the helper below also has an overload with a trailing int framesAgo argument. 0 means the current analyzed frame, 1 means the previous stored frame, and so on.

segmentAt

vec2 segmentAt(vec2 pos)
vec2 segmentAt(vec2 pos, int framesAgo)

Returns vec2(confidence, normalizedCategoryIndex).

• x: confidence for the winning category at pos
• y: normalized category index in the range 0.0 to 1.0

When outputConfidenceMasks is false, the confidence component is always 1.0.

vec2 segment = segmentAt(v_uv);
float confidence = segment.x;
int categoryIndex = int(floor(segment.y * float(u_numCategories - 1) + 0.5));
float isForeground = float(categoryIndex != 0) * confidence;
color.rgb = mix(color.rgb, vec3(1.0, 0.0, 1.0), isForeground);

This page covers the ShaderPad-facing API surface. For MediaPipe result object structure and model changes, use the upstream MediaPipe docs.

Related

• History
• Webcam input