npm install @mediapipe/tasks-vision

The pose plugin uses MediaPipe's Pose Landmarker and exposes landmark and mask data in ShaderPad-friendly GLSL form.

import pose from 'shaderpad/plugins/pose'

const shader = new ShaderPad(fragmentShaderSrc, {
  plugins: [pose({ textureName: 'u_webcam', options: { maxPoses: 2 } })],
})

The plugin reads from the texture named by textureName. Initialize and update that exact ShaderPad texture name, or the detector will have no live source to read from.

Options

Events

Subscribe with shader.on(name, callback).

shader.on('pose:result', result => {
  console.log(result.landmarks.length)
})

Uniforms

Most shaders should use the helper functions below instead of sampling u_poseLandmarksTex or u_poseMask directly.

Helper Functions

If history is enabled, every helper below also has an overload with a trailing int framesAgo argument. 0 means the current analyzed frame, 1 means the previous stored frame, and so on.

nPosesAt

int nPosesAt()
int nPosesAt(int framesAgo)

Returns the number of poses stored for the current or historical frame.

poseLandmark

vec4 poseLandmark(int poseIndex, int landmarkIndex)
vec4 poseLandmark(int poseIndex, int landmarkIndex, int framesAgo)

Returns vec4(x, y, z, visibility).

• x, y: normalized landmark position in ShaderPad UV space
• z: MediaPipe landmark depth value
• w: landmark visibility / confidence

vec2 leftHip = vec2(poseLandmark(0, POSE_LANDMARK_LEFT_HIP));
vec2 rightHip = vec2(poseLandmark(0, POSE_LANDMARK_RIGHT_HIP));

poseAt

vec2 poseAt(vec2 pos)
vec2 poseAt(vec2 pos, int framesAgo)

Returns vec2(confidence, poseIndex).

• x: segmentation confidence for the sampled pixel
• y: the matching poseIndex, or -1.0 when no pose matched

This is the fastest way to ask “which pose owns this pixel?” without manually decoding u_poseMask.

vec2 hit = poseAt(v_uv);
if (hit.y >= 0.0) {
  float confidence = hit.x;
  int poseIndex = int(hit.y);
  vec2 torso = vec2(poseLandmark(poseIndex, POSE_LANDMARK_TORSO_CENTER));
  color.rgb = mix(color.rgb, vec3(torso, confidence), confidence);
}

inPose

float inPose(vec2 pos)
float inPose(vec2 pos, int framesAgo)

Returns the confidence component from poseAt(). This is 0.0 when no pose matched, otherwise the pose-mask confidence for that pixel.

Landmark Layout

The plugin exposes MediaPipe’s standard 33 pose landmarks plus six derived landmarks:

• POSE_LANDMARK_BODY_CENTER
• POSE_LANDMARK_LEFT_HAND_CENTER
• POSE_LANDMARK_RIGHT_HAND_CENTER
• POSE_LANDMARK_LEFT_FOOT_CENTER
• POSE_LANDMARK_RIGHT_FOOT_CENTER
• POSE_LANDMARK_TORSO_CENTER

Commonly useful named constants include:

• POSE_LANDMARK_LEFT_EYE
• POSE_LANDMARK_RIGHT_EYE
• POSE_LANDMARK_LEFT_SHOULDER
• POSE_LANDMARK_RIGHT_SHOULDER
• POSE_LANDMARK_LEFT_ELBOW
• POSE_LANDMARK_RIGHT_ELBOW
• POSE_LANDMARK_LEFT_HIP
• POSE_LANDMARK_RIGHT_HIP
• POSE_LANDMARK_LEFT_KNEE
• POSE_LANDMARK_RIGHT_KNEE

For the full MediaPipe landmark index map, use the upstream Pose Landmarker model reference.

This page covers the ShaderPad-facing API surface. For MediaPipe result object structure and model changes, use the upstream MediaPipe docs.

Related

• History
• Webcam input