npm install @mediapipe/tasks-vision

The hands plugin uses MediaPipe's Hand Landmarker and exposes ShaderPad textures and GLSL helper functions for hand-driven effects.

import hands from 'shaderpad/plugins/hands'

const shader = new ShaderPad(fragmentShaderSrc, {
  plugins: [hands({ textureName: 'u_webcam', options: { maxHands: 2 } })],
})

The plugin reads from the texture named by textureName. Initialize and update that exact ShaderPad texture name, or the detector will have no live source to read from.

Options

Events

Subscribe with shader.on(name, callback).

shader.on('hands:result', result => {
  console.log(result.landmarks.length)
})

Uniforms

Most shaders should use the helper functions below instead of sampling u_handLandmarksTex directly.

Helper Functions

If history is enabled, every helper below also has an overload with a trailing int framesAgo argument. 0 means the current analyzed frame, 1 means the previous stored frame, and so on.

nHandsAt

int nHandsAt()
int nHandsAt(int framesAgo)

Returns the number of hands stored for the current or historical frame.

handLandmark

vec4 handLandmark(int handIndex, int landmarkIndex)
vec4 handLandmark(int handIndex, int landmarkIndex, int framesAgo)

Returns vec4(x, y, z, handedness).

• x, y: normalized landmark position in ShaderPad UV space
• z: MediaPipe landmark depth value
• w: handedness, where 0.0 means left and 1.0 means right

Use vec2(handLandmark(...)) when you only need the screen position.

#define WRIST 0
#define THUMB_TIP 4
#define INDEX_TIP 8
#define HAND_CENTER 21

for (int i = 0; i < u_nHands; ++i) {
  vec4 center = handLandmark(i, HAND_CENTER);
  vec2 wrist = vec2(handLandmark(i, WRIST));
  vec2 pinchMid = 0.5 * (
    vec2(handLandmark(i, THUMB_TIP)) +
    vec2(handLandmark(i, INDEX_TIP))
  );
  float isRight = center.w;
  color.rgb += mix(vec3(0.0, 0.6, 1.0), vec3(1.0, 0.5, 0.0), isRight)
    * smoothstep(0.12, 0.0, distance(v_uv, pinchMid));
}

isRightHand

float isRightHand(int handIndex)
float isRightHand(int handIndex, int framesAgo)

Returns 1.0 for right hands and 0.0 for left hands. This is equivalent to handLandmark(handIndex, 0).w, but clearer to read when you only care about handedness.

isLeftHand

float isLeftHand(int handIndex)
float isLeftHand(int handIndex, int framesAgo)

Returns 1.0 for left hands and 0.0 for right hands.

Landmark Layout

The plugin exposes MediaPipe’s standard 21 hand landmarks plus one derived HAND_CENTER point.

HAND_CENTER is a ShaderPad convenience landmark derived from the wrist and MCP joints. It is useful when you want a stable center point for sprites, particles, or gesture indicators without averaging landmarks yourself.

This page covers the ShaderPad-facing API surface. For MediaPipe result object structure and model changes, use the upstream MediaPipe docs.

Related

• Webcam input
• History