SO-101

ft_6341c5_d13da9 — MolmoAct-2 fine-tuned on the SO-101 single-arm embodiment. 6D joint state, two RGB cameras in any order, 30-step action chunks. Norm tag so100_so101_molmoact2. [source: https://huggingface.co/allenai/MolmoAct2-SO100_101]

Quickstart

import os
import newt
import numpy as np

def read_state():
    return {
        "state": np.zeros(6, dtype=np.float32),
        "images": {
            "top":  np.zeros((3, 224, 224), dtype=np.uint8),
            "side": np.zeros((3, 224, 224), dtype=np.uint8),
        },
    }

def execute(chunk):
    for joints in chunk:
        pass  # send 6D joint targets to your controller

robot = newt.Robot(
    api_key=os.environ["NT_API_KEY"],
    model="so101",
    read_state=read_state,
    execute=execute,
)
result = robot.run("place the block in the bin", max_duration=30.0)

Replace np.zeros(...) with your actual sensor reads. state is 6 floats (single-arm joint positions); each image is channels-first uint8 at 224×224. Neither camera key is individually required — a missing key zero-fills with a DegradationWarning, and camera order is not fixed. The SO-101 starter ships both callbacks in embodiment.py already wired to real lerobot SO101Follower hardware — what you fill in is nt.toml (serial port, camera indices), not the callback bodies.

Inputs

read_state must return a dict with these keys:

The server validates the state shape and image dimensions on the first observation frame. A state shape mismatch closes with code 4422. Camera mismatches are permissive — neither key is hard-required; see Camera roles below. See the API reference for error envelope details.

Outputs

execute receives a chunk of shape (30, 6) — a 30-step action horizon, 6 action dims per step. Each row is a target joint configuration in the same 6D format as state.

The 6 action dims correspond to ["shoulder_pan", "shoulder_lift", "elbow_flex", "wrist_flex", "wrist_roll", "gripper"] in that order (see State layout).

Clients typically execute a prefix of the chunk, then read fresh state and send the next observation. See the API reference for the receding-horizon pattern.

Provenance

State layout

The 6D state vector maps to the SO-101's joint axes, gripper-last:

Send these in the same layout to read_state. The SO-101 is not factory-calibrated — run lerobot's calibration flow before the first run.

Camera roles

The two expected cameras serve fixed viewing angles in the training distribution:

Camera order is not fixed; the server does not consume them positionally. A missing key zero-fills with a DegradationWarning rather than closing with 4422.

Notes

• max_action_horizon (30) vs flow_matching_num_steps (10) — max_action_horizon is the action chunk length you receive (30 steps); flow_matching_num_steps is the internal diffusion solver step count (10). HuggingFace inference examples pass num_steps=10 referring to the latter; that value does not describe the action chunk shape.
• Gripper axis (axis 6): negative-signed on synthetic observations. Physical open/close semantics are unverified without hardware — the sign convention may need to be flipped on a real SO-101. Verify against your arm before deploying.
• Eval-parity: task-success rate against the Allen AI baseline has not been benchmarked with physical hardware. A success-rate gap should be treated as a potential ingestion issue rather than a model quality ceiling until verified on a real arm.
• To add a customer fine-tune on top of this checkpoint, declare a registry entry with base: ft_6341c5_d13da9 — the contract is inherited automatically via recursive resolution.

References

• Model card: allenai/MolmoAct2-SO100_101
• Paper: arXiv:2605.02881
• Upstream code: github.com/allenai/molmoact2