SO-101 starter

The SO-101 starter is a project you clone to run MolmoAct-2 SO-101 inference on a Hugging Face SO-101 arm with two cameras. The model produces (30, 6) action chunks representing 30 target joint configurations across 6 degrees of freedom; the starter supplies the SO101 embodiment class that reads the arm's joint state and applies those chunks.

Repository: new-theory-research/newt-starter-so101.

Not to be confused with the Hackathon Starter Kit, the recording/teleop kit used at The Embodied Metal Hackathon — this page is the SO-101 deploy client.

git clone git@github.com:new-theory-research/newt-starter-so101.git

What's in it

The repo is small and flat. The pieces you interact with:

Setup uses uv, not pip. The canonical first command is:

uv sync

This resolves the newt dependency. newt-python is public — no collaborator access needed. If uv sync fetches it over a pinned git+ssh:// URL, you still need an SSH key registered with GitHub; see SSH-only machines.

Callbacks

SO101.read_state() reads one observation frame from the hardware. It queries the arm's 6 Feetech servos for their current joint positions via lerobot's SO101Follower driver, and captures RGB frames from the top and side cameras, resized to 224×224. It returns a dict with state (6D joint positions in normalized lerobot units) and images (one entry per camera present).

The model was trained with top and side as camera names. If one camera is missing at runtime, the model server zero-fills it with DegradationWarning; quality degrades but the run does not close. Both cameras are optional from the server's perspective — present both for best results.

SO101.execute(chunk) applies one (30, 6) action chunk to the arm. For each of the first MAX_ACTIONS_PER_CHUNK rows (default 15), it sends the target joint positions to the servos via sync_write. The first chunk uses a 1.5-second settle sleep before streaming remaining actions. The servos travel at their own speed — there is no goal-time interpolation.

Gripper (axis 6). The gripper axis sign convention is unverified on physical hardware. Synthetic observations suggest negative-signed gripper commands, but physical open/close semantics have not been confirmed with a real arm. Verify the sign direction against your arm before deploying; the starter README carries the same caveat.

Both methods live in embodiment.py. If you wire a different arm or camera configuration, replace those bodies — the from_config() constructor and config-loading helpers can stay.

What you fill in

Run the setup command after uv sync:

uv run python3 run.py setup

This writes ~/.config/nt/nt.toml from the template, auto-detects your serial port if exactly one SO-101 candidate is present, and reports what still needs your attention (camera indices, arm id). It accepts --force, --report-only, --non-interactive, and --json flags.

Manual fallback. If you prefer to configure by hand, copy the template and edit it directly:

mkdir -p ~/.config/nt
cp conf/nt.toml.example ~/.config/nt/nt.toml

Then fill in:

• [[robot_config.arms]] — id is the lerobot robot ID for your calibrated arm (it also names the calibration file), and port is the serial port for your SO-101 arm (e.g., /dev/ttyACM0 on Linux, /dev/tty.usbmodem* on macOS). Calibration files are stored at ~/.cache/huggingface/lerobot/calibration/robots/so101_follower/<id>.json (override with HF_LEROBOT_CALIBRATION).
• [[camera_config.cameras]] — one entry each for id = "top" and id = "side", with index_or_path (integer camera index, or a device path like /dev/video0), plus width, height, and fps. uv run python3 run.py --check reports the arm and cameras your config declares — it does not probe hardware.

You also export your NT_API_KEY. The entry script checks for it at launch and exits immediately if it's missing.

Camera placement. The camera names (top, side) match the labels used during training. Their physical mounting positions affect model performance: cameras mounted at angles that differ significantly from training geometry will push observations out-of-distribution. Match mounting geometry to how the training data was collected.

Calibration. The SO-101 is not factory-calibrated. Run lerobot's calibration flow before the first run:

lerobot-calibrate --robot.type=so101_follower --robot.port=<port> --robot.id=<arm_id>

This generates the calibration file the starter reads via from_config().

Running without the rig

You can run uv run python3 run.py --check on a Mac or any machine before the arm is connected. Without the [hardware] extra installed, the check validates your config file and API key against the model server without touching servo hardware. It exits 0 and names the arm and cameras it would use, or explains what's missing.

The actual closed-loop trial requires the arm connected and calibrated — there is no mock run mode.

Tasks

The SO-101 model is open-vocabulary and language-conditioned. --task is free-form text, not a fixed key:

uv run python3 run.py --task "pick up the block and place it in the bin"

To see example task descriptions:

uv run python3 run.py --list-tasks

This prints clearly-illustrative example prompts and notes what the model was trained on. There is no fixed task-key registry — any reasonable natural-language task description is valid input.

Agent-driven rigs

If you're running run.py from an agent harness rather than an interactive terminal:

SSH install approval. Agent harnesses may require human approval for git+ssh:// dependency installs. This is expected — approve the newt and lerobot install steps and continue.

Cleanup in non-TTY environments. Ctrl+H triggers emergency_home() — moves the arm to its rest pose, then disconnects — but it's a stdin listener that only arms when stdin is a TTY, so it's unavailable in a non-TTY agent session. SIGINT/Ctrl+C triggers the weaker path, teardown(): it disconnects without homing, so the arm goes limp wherever it stopped. In a non-TTY agent session, Ctrl+C may not reach the process — use timeout --signal=INT to send SIGINT explicitly:

timeout --signal=INT 60 uv run python3 run.py --task "pick up the block and place it in the bin"

This disconnects the arm after 60 seconds; it does not home it first.

When to use it

Use this starter when you have an SO-101 arm and want MolmoAct-2 SO-101 driving it. The starter runs open-vocabulary tasks — you supply a natural-language description with --task. uv run python3 run.py --reset moves the arm to a safe rest pose without running inference.

Requires Python 3.11–3.13. Python ≥ 3.14 is not supported; it hits a torch compatibility wall.

Setup

For the wire contract — state shape, action shape, cameras, and caveats — see the SO-101 model page. The repo's own README is the other reference, covering the full install sequence, calibration, camera index enumeration, and troubleshooting.