Hand–Eye Calibration: Connecting Robot Vision to Robot Motion

Camera calibration makes vision measurable. Hand–eye calibration makes that vision usable by the robot.

What this post covers

1. From calibrated vision to robot-aware vision

The previous ChArUco camera calibration step produced camera_matrix.npy and dist_coeffs.npy. Those files made the wrist camera geometrically reliable; this step connects that calibrated camera to the robot so detections can become motion targets.

In other words, camera calibration answered: how does this camera measure the world? Hand–eye calibration answers the next question: where is this camera relative to the robot hand?

2. What hand–eye calibration is and why it matters

In robotics, the hand is the robot end-effector, wrist, flange, or gripper. The eye is the camera. Hand–eye calibration estimates the fixed 3D transform between those two coordinate frames.

For a wrist-mounted camera, the key result is usually:

Simple definition: hand–eye calibration teaches the robot where its camera is. After calibration, a point detected in the camera frame can be converted into the robot frame.

The reason this matters is practical: the camera reports positions in the camera coordinate frame, while the robot plans motion in the robot base or end-effector frame.

That conversion is useful for pick-and-place, bin picking, tabletop grasping, visual servoing, inspection, tool positioning, marker-based alignment, and sim-to-real workflows where camera geometry has to line up with robot geometry.

3. Where we used it: the SO-101 pipeline

For our implementation, we performed eye-in-hand calibration on an SO-101 arm with a camera mounted on the wrist. The ChArUco board stayed fixed on the table, while the wrist camera moved through different poses.

The important transform chain for eye-in-hand calibration is:

Visuals from the hand–eye calibration setup

These cropped visuals show the real SO-101 wrist-camera setup used during calibration: the board view from the camera, the robot arm, and the fixed target used for multiple robot poses.

Figure 1 — Wrist-camera view: ChArUco corners, marker IDs, and axes are visible before saving a sample.

Figure 2 — A second wrist pose shows the same fixed board detected from a different angle.

Figure 3 — SO-101 setup: the wrist-mounted camera faces the fixed ChArUco target during capture.

Figure 4 — The target stays fixed while the wrist camera moves through different poses for hand–eye solving.

4. Solving and choosing a method

OpenCV supports several hand–eye solvers through cv.calibrateHandEye(). We treated the methods as candidates, not as universal rankings. Method performance is data-dependent: pose diversity, rotation noise, translation noise, robot accuracy, and target detection quality can all change which method behaves best. Published comparisons also show that noise type and motion range matter, so the safest workflow is to try multiple methods and compare consistency rather than trusting one result blindly.

Capturing the two inputs the solver needs

The solver needs synchronized robot poses and camera target poses. The robot side is T_base_gripper. The camera side is T_camera_charuco. For SO-101, the joint readings can be pulled through LeRobot’s SO101 follower interface, then converted to T_base_gripper with the same FK model or URDF used by the robot stack. The important contract is simple: each saved image must be paired with the 4×4 wrist pose from the same physical moment.

This is the concrete capture pattern we used conceptually. The exact FK helper is project-specific, but the robot-state read is not just a placeholder: it follows the SO-101 follower API style, where the robot is configured with its serial port and id, connected, and read through get_observation().

# 1) Robot side: read SO-101 follower joint positions with LeRobot.
from lerobot.common.robots.so101_follower import (
    SO101Follower,
    SO101FollowerConfig,
)
robot_cfg = SO101FollowerConfig(
    port="/dev/ttyACM0",    # your MotorBus port
    id="my_so101_follower", # same id used during calibration
    use_degrees=True,
)
robot = SO101Follower(robot_cfg)
robot.connect(calibrate=False)
obs = robot.get_observation()
joint_positions = {
    "shoulder_pan": obs["shoulder_pan.pos"],
    "shoulder_lift": obs["shoulder_lift.pos"],
    "elbow_flex": obs["elbow_flex.pos"],
    "wrist_flex": obs["wrist_flex.pos"],
    "wrist_roll": obs["wrist_roll.pos"],
    "gripper": obs["gripper.pos"],
}
# Convert calibrated joint positions to the wrist pose used by hand-eye.
# This function comes from your SO-101 URDF/DH model or robot stack.
T_base_gripper = so101_fk_to_matrix(joint_positions)
# If your control stack already exposes an end-effector pose, use that
# instead, but keep the output as a 4 x 4 T_base_gripper matrix.

# 2) Camera side: estimate the fixed board pose from the wrist camera image.
# K and D are camera_matrix.npy and dist_coeffs.npy from ChArUco camera calibration.
marker_corners, marker_ids, _ = cv.aruco.detectMarkers(gray, aruco_dict)
_, charuco_corners, charuco_ids = cv.aruco.interpolateCornersCharuco(
    marker_corners, marker_ids, gray, board, K, D
)
ok, rvec, tvec = cv.aruco.estimatePoseCharucoBoard(
    charuco_corners, charuco_ids, board, K, D, None, None
)
R, _ = cv.Rodrigues(rvec)
T_camera_charuco = make_T(R, tvec)

If FK is not integrated during live capture, log the obs dictionary beside each saved image and convert those joint positions to T_base_gripper offline before running the solver. What matters is synchronization and using the same calibrated robot id, units, and wrist frame every time.

Solving the hand–eye transform

After comparing methods, the calibration call itself is small. The example below shows the baseline call pattern using TSAI; in practice we ran the same block for all five methods and selected the result that validated best.

import cv2 as cv
import numpy as np

R_gripper2base = []
t_gripper2base = []
R_target2cam = []
t_target2cam = []

for T_bg, T_ct in zip(T_base_gripper_all, T_camera_charuco_all):
    R_gripper2base.append(T_bg[:3, :3])
    t_gripper2base.append(T_bg[:3, 3].reshape(3, 1))
    R_target2cam.append(T_ct[:3, :3])
    t_target2cam.append(T_ct[:3, 3].reshape(3, 1))

R_cam2gripper, t_cam2gripper = cv.calibrateHandEye(
    R_gripper2base,
    t_gripper2base,
    R_target2cam,
    t_target2cam,
    method=cv.CALIB_HAND_EYE_TSAI,
)

T_gripper_camera = np.eye(4)
T_gripper_camera[:3, :3] = R_cam2gripper
T_gripper_camera[:3, 3] = t_cam2gripper.reshape(3)
np.save("outputs/T_gripper_camera.npy", T_gripper_camera)

Once this file exists, detections from the wrist camera can be converted into robot coordinates:

tmp = np.matmul(T_base_gripper, T_gripper_camera)
T_base_object = np.matmul(tmp, T_camera_object)

Files produced by the pipeline

5. Checking the result

There are two different checks that often get mixed together.

projected, _ = cv.projectPoints(
    object_points,
    rvec,
    tvec,
    camera_matrix,
    dist_coeffs,
)
error_px = cv.norm(
    image_points,
    projected,
    cv.NORM_L2,
) / len(projected)
print(f"reprojection error: {error_px:.4f} px")

T_gripper_camera = np.load("outputs/T_gripper_camera.npy")
data = np.load("data/handeye_samples.npz")

base_positions = []
for T_bg, T_ct in zip(data["T_base_gripper"], data["T_camera_charuco"]):
    tmp = np.matmul(T_bg, T_gripper_camera)
    T_base_charuco = np.matmul(tmp, T_ct)
    base_positions.append(T_base_charuco[:3, 3])

base_positions = np.asarray(base_positions)
mean_position = base_positions.mean(axis=0)
errors_mm = np.linalg.norm(
    base_positions - mean_position,
    axis=1,
) * 1000.0
print("mean error mm:", errors_mm.mean())
print("max error mm:", errors_mm.max())

Optional weak-frame filtering before the final solve

Outlier rejection should not be used to hide a bad calibration. We used it only after inspecting the saved images and robot logs, then removed samples that were visibly blurred, partially detected, or inconsistent with the fixed-board assumption.

# After a first solve, reject samples whose fixed-board position
# is an outlier. Then re-run calibrateHandEye on the filtered set.
median = np.median(errors_mm)
mad = np.median(np.abs(errors_mm - median)) + 1e-9
robust_sigma = 1.4826 * mad
keep = errors_mm < (median + 3.0 * robust_sigma)
print(f"keeping {keep.sum()} of {len(keep)} samples")

filtered = {
    "T_base_gripper": data["T_base_gripper"][keep],
    "T_camera_charuco": data["T_camera_charuco"][keep],
}
np.savez("data/handeye_samples_filtered.npz", **filtered)
# Re-run the solver on handeye_samples_filtered.npz and compare:
# - mean/max fixed-board error
# - camera mount plausibility
# - physical touch-test accuracy

A good filtering rule is conservative: remove obvious bad samples, then compare the filtered and unfiltered transforms. If filtering changes the transform dramatically, the dataset likely needs to be recaptured with better pose diversity or cleaner detections.

Making error targets task-specific

Important framing: the targets below are practical prototype / hobbyist-arm rules of thumb for a low-cost robot setup like our SO-101 workflow. Industrial systems with rigid robots and metrology-grade cameras often expect much tighter residuals: sub-millimeter for smaller robots and millimeter-level for larger robots. Treat the table as task guidance for this class of prototype system, not an industry-wide accuracy standard.

Another reason to treat the number carefully is error compounding. A small transform error near the wrist can become a larger miss at the fingertips, especially when the tool is long or the robot approaches at an angle. That is why the final question is not just “what was the residual?” but “did the robot touch or grasp the thing it was supposed to?”

6. Validating beyond the numbers

A good calibration should pass more than one test. We used numerical validation, but a task-level check is equally important.

Physical validation is important because a calibration can look acceptable numerically while still failing at the tool tip if the TCP, gripper offset, robot kinematics, or transform direction is wrong.

7. Challenges and what we did about them

Hand-eye calibration looks simple on paper, but the practical details matter. The table below summarizes the issues we faced and the fixes that made the result more reliable.