Retargeting Sessions: Keypoints to Joint Angles

SDK Practice

Overview

Feed in a set of MediaPipe-format hand keypoints, get back a joint command you can send straight to a dexterous hand. This step connects to no device at all. This step performs computation only.

The keypoints can come from anywhere. A camera, a Wuji Glove, a recorded replay, VR headset hand tracking — as long as the array has the same shape and units, everything downstream stays identical. Run this example first to confirm retargeting works and to see the input and output shapes. When you then attach real hardware, there’s far less ground to search.

Matching examples: examples/python/retargeting/0.retarget_session.py and examples/c/retargeting/0_retarget_session.c. The C interface maps one to one onto Python.

For API and field definitions, see Retargeting.

Prerequisites

Hardware

None. This example uses synthetic keypoints, so it neither scans for nor connects to a device.

Software

Item	Requirement
Operating system	Ubuntu 20.04 LTS or later, or macOS 11.0 or later
Architecture	Retargeting ships prebuilt wheels for Linux x86_64, Linux aarch64, and macOS arm64
Wuji SDK	v2026.7.21 or later
Python	3.8 or later, plus `numpy` to hold the keypoint and joint-angle arrays
C toolchain	CMake 3.16 or later, plus the extracted SDK package, which supplies `include/` and the shared library (`.so` on Linux or `.dylib` on macOS)

Python
C

pip install wuji-sdk numpy

LIB="libwuji_sdk_c.so"
if [ "$(uname -s)" = "Darwin" ]; then LIB="libwuji_sdk_c.dylib"; fi
cmake -S examples/c/retargeting -B build/retargeting \
  -DWUJI_SDK_INCLUDE_DIR="${SDK}/include" \
  -DWUJI_SDK_LIB="${SDK}/lib/${LIB}"
cmake --build build/retargeting --target 0_retarget_session

Before You Start

Nothing. This example is the smallest starting point on the retargeting path. Get it running, then move on to real hardware.

Data Flow

A single frame travels the path below. The session object keeps state across frames, so create it once and reuse it in your loop.

Keypoints are in meters, joint commands in radians. The output is always float32 with a fixed shape that doesn’t vary with the number of online joints. float64 input is accepted and converted internally.

Integration Steps

Create a session, solve each frame, then end the tracking segment.

make_open_hand_keypoints generates synthetic keypoints in Python example. Replace it with the application’s keypoint source. C excerpts require wuji_sdk.h, stdio.h, and stdlib.h.

1. Create a Session

Select the target hand model and handedness. Reuse one session for each continuous tracking segment to retain state across frames.

Python
C

from wuji_sdk import HandModel, Handedness, RetargetSession

session = RetargetSession.for_hand(HandModel.WujiHand2, side=Handedness.Right)

struct WujiRetargetSession *session = NULL;
WujiStatus st = wuji_retarget_session_create(WUJI_HAND_MODEL_WUJI_HAND2, WUJI_HANDEDNESS_RIGHT, &session);
if (st != WUJI_STATUS_OK) {
    fprintf(stderr, "%s\n", wuji_last_error());
    return EXIT_FAILURE;
}

2. Solve Each Frame

Pass 21 MediaPipe keypoints per frame in meters. Python uses a (21, 3) float32 array. C uses a 63-element array of x, y, and z coordinates. The result contains 20 joint angles in radians, ordered by firmware joint index.

Python
C

keypoints = make_open_hand_keypoints()
qpos = session.step(keypoints)

float keypoints[63];
make_open_hand_keypoints(keypoints);
float qpos[20];
st = wuji_retarget_session_step(session, keypoints, qpos);
if (st != WUJI_STATUS_OK) {
    fprintf(stderr, "%s\n", wuji_last_error());
    wuji_retarget_session_free(session);
    return EXIT_FAILURE;
}

Pass subsequent frames to step. This page only calculates joint angles. The application handles device connection and command publishing separately.

3. Reset and Release

Call reset after tracking is interrupted, then start the next segment. Free the C session when the program ends.

Python
C

session.reset()

wuji_retarget_session_reset(session);
wuji_retarget_session_free(session);

Running It

Python
C

python examples/python/retargeting/0.retarget_session.py

./build/retargeting/0_retarget_session

Both sides print the same structure: the joint command’s shape and dtype, then the 20 joint angles grouped by finger.

qpos shape : (20,), dtype float32
joint command (radians, finger-major thumb/index/middle/ring/pinky):
  thumb : ['+1.291', '+0.206', '+0.345', '+0.039']
  index : ['+0.219', '+0.654', '+0.202', '+0.087']
  middle: ['+0.290', '+0.477', '+0.004', '+0.001']
  ring  : ['+0.166', '+0.274', '-0.002', '-0.001']
  pinky : ['+0.061', '+0.130', '+0.029', '+0.039']

C skips the shape line and starts at the joint command:

joint command (radians, finger-major thumb/index/middle/ring/pinky):
  thumb : [+1.291, +0.206, +0.345, +0.039]
  index : [+0.219, +0.654, +0.202, +0.087]
  middle: [+0.290, +0.477, +0.004, +0.001]
  ring  : [+0.166, +0.274, -0.002, -0.001]
  pinky : [+0.061, +0.130, +0.029, +0.039]

These are actual results, measured on Wuji SDK v2026.8.31, Python 3.12, the C SDK for x86_64-linux-gnu, and Linux x86_64, with the example’s built-in synthetic open right-hand pose as input.

For the same input, both interfaces produce bit-identical joint angles across all 20 values, because they share one solver implementation. Change the keypoints or the hand model and the numbers change, but the shape and grouping don’t.

Notes

Branch point	Value	Effect
Hand model	`HandModel.WujiHand2` or `HandModel.WujiHand`	Selects the internal tuning profile. You never name a config file path
Handedness	`Handedness.Right` or `Handedness.Left`	The example defaults to the right hand. Keypoints and target hand must be the same side, and mixing them gives mirrored results
Tracking gap	Whether you call `reset()`	Skip the reset and the previous segment’s warm-start and smoothing state carry into the new one, so the first few frames jump
Degenerate keypoints	All zeros, or containing NaN or Inf	`step()` raises `WujiException: retargeting: invalid keypoints: contains NaN or Inf`. Confirm the keypoint source is valid before solving, and never pass an uninitialized array as a placeholder
Wrong shape	Element count other than 63	Python raises `ValueError: cannot reshape array of size N into shape (63,)`. Python works with the same length-63 layout as C internally
C-side error	create or step returns something other than `WUJI_STATUS_OK`	Call `wuji_last_error()` for the reason. A session that was created still needs `free`, or it leaks

The session is stateful. Reuse one session across a continuous tracking segment, reset between segments, and never create one per frame.

Adapting It

Swap In a Real Keypoint Source

Replace the example’s make_open_hand_keypoints() with your source. As long as it produces (21, 3) keypoints in meters in MediaPipe landmark order, nothing downstream changes. The Wuji Glove’s hand_skeleton() stream is already in this format, and Retargeting shows how to connect it.

Change Hand Model and Handedness

Targeting a Wuji Hand instead, change HandModel.WujiHand2 to HandModel.WujiHand. For a left-hand assembly, change Handedness.Right to Handedness.Left and confirm the keypoints come from a left hand too.

Move to a Real-Time Loop

This example solves one frame. In a real-time setting, call step() on a fixed period in a loop and handle the case where no new frame arrived. For the full loop, see Retargeting.

Input Validity

Confirm the keypoints aren’t degenerate before solving. An all-zero array, or one holding NaN or Inf, makes step() raise invalid keypoints. When tracking is lost, repeating the last valid frame is safer than sending zeros.

Handling Tracking Gaps

The hand leaves the frame, the device disconnects, a replay cuts to a new segment — call reset() once after any of these. Skip it and the first few frames after recovery ease out of the old state, which looks like lag.

Retargeting: interface description and parameter meanings
Common Data Types: the structures behind keypoints and joint commands