# SO-101 starter URL: /docs/starters/so101 What's in the newt-starter-so101 repo — the project you clone to run MolmoAct-2 SO-101 inference on a Hugging Face SO-101 arm. The SO-101 starter is a project you clone to run [MolmoAct-2 SO-101](/docs/models/molmoact2/so101) inference on a Hugging Face SO-101 arm with two cameras. The model produces `(30, 6)` action chunks representing 30 target joint configurations across 6 degrees of freedom; the starter supplies the `SO101` embodiment class that reads the arm's joint state and applies those chunks. Repository: [`new-theory-research/newt-starter-so101`](https://github.com/new-theory-research/newt-starter-so101). Not to be confused with the [Hackathon Starter Kit](https://github.com/mission-robotics-ai/so100-hackathon), the recording/teleop kit used at The Embodied Metal Hackathon — this page is the SO-101 deploy client. ```bash git clone git@github.com:new-theory-research/newt-starter-so101.git ``` ## What's in it [#whats-in-it] The repo is small and flat. The pieces you interact with: | File | What it is | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `embodiment.py` | The `SO101` class — the embodiment you own. Implements `read_state()` and `execute()` for the SO-101 arm and cameras; `SO101.from_config()` loads hardware addresses from `~/.config/nt/nt.toml`. Rename it, edit the wiring, add your own logic. | | `run.py` | The thin entry script. Constructs `SO101.from_config()`, passes it to `newt.Robot(embodiment=rig, model="so101")`, and runs one closed-loop trial. Accepts `--task`, `--list-tasks`, `--check`, `--reset`, and `--arm` flags. | | `conf/nt.toml.example` | The config template. You copy it to `~/.config/nt/nt.toml` and fill in your hardware specifics. | | `pyproject.toml` | Dependencies, resolved with `uv`. Pulls in `newt` and `lerobot`. Install `--extra hardware` on the rig machine to add the Feetech servo driver (`lerobot[feetech]`). | Setup uses `uv`, not pip. The canonical first command is: ```bash uv sync ``` This resolves the `newt` dependency. `newt-python` is public — no collaborator access needed. If `uv sync` fetches it over a pinned `git+ssh://` URL, you still need an SSH key registered with GitHub; see [SSH-only machines](/docs/install#ssh-only-machines). ## Callbacks [#callbacks] `SO101.read_state()` reads one observation frame from the hardware. It queries the arm's 6 Feetech servos for their current joint positions via lerobot's `SO101Follower` driver, and captures RGB frames from the `top` and `side` cameras, resized to 224×224. It returns a dict with `state` (6D joint positions in normalized lerobot units) and `images` (one entry per camera present). The model was trained with `top` and `side` as camera names. If one camera is missing at runtime, the model server zero-fills it with `DegradationWarning`; quality degrades but the run does not close. Both cameras are optional from the server's perspective — present both for best results. `SO101.execute(chunk)` applies one `(30, 6)` action chunk to the arm. For each of the first `MAX_ACTIONS_PER_CHUNK` rows (default 15), it sends the target joint positions to the servos via `sync_write`. The first chunk uses a 1.5-second settle sleep before streaming remaining actions. The servos travel at their own speed — there is no goal-time interpolation. **Gripper (axis 6).** The gripper axis sign convention is unverified on physical hardware. Synthetic observations suggest negative-signed gripper commands, but physical open/close semantics have not been confirmed with a real arm. Verify the sign direction against your arm before deploying; the starter README carries the same caveat. Both methods live in `embodiment.py`. If you wire a different arm or camera configuration, replace those bodies — the `from_config()` constructor and config-loading helpers can stay. ### What you fill in [#what-you-fill-in] Run the setup command after `uv sync`: ```bash uv run python3 run.py setup ``` This writes `~/.config/nt/nt.toml` from the template, auto-detects your serial port if exactly one SO-101 candidate is present, and reports what still needs your attention (camera indices, arm id). It accepts `--force`, `--report-only`, `--non-interactive`, and `--json` flags. **Manual fallback.** If you prefer to configure by hand, copy the template and edit it directly: ```bash mkdir -p ~/.config/nt cp conf/nt.toml.example ~/.config/nt/nt.toml ``` Then fill in: * `[[robot_config.arms]]` — `id` is the lerobot robot ID for your calibrated arm (it also names the calibration file), and `port` is the serial port for your SO-101 arm (e.g., `/dev/ttyACM0` on Linux, `/dev/tty.usbmodem*` on macOS). Calibration files are stored at `~/.cache/huggingface/lerobot/calibration/robots/so101_follower/.json` (override with `HF_LEROBOT_CALIBRATION`). * `[[camera_config.cameras]]` — one entry each for `id = "top"` and `id = "side"`, with `index_or_path` (integer camera index, or a device path like `/dev/video0`), plus `width`, `height`, and `fps`. `uv run python3 run.py --check` reports the arm and cameras your config declares — it does not probe hardware. You also export your `NT_API_KEY`. The entry script checks for it at launch and exits immediately if it's missing. **Camera placement.** The camera names (`top`, `side`) match the labels used during training. Their physical mounting positions affect model performance: cameras mounted at angles that differ significantly from training geometry will push observations out-of-distribution. Match mounting geometry to how the training data was collected. **Calibration.** The SO-101 is not factory-calibrated. Run lerobot's calibration flow before the first run: ```bash lerobot-calibrate --robot.type=so101_follower --robot.port= --robot.id= ``` This generates the calibration file the starter reads via `from_config()`. ### Running without the rig [#running-without-the-rig] You can run `uv run python3 run.py --check` on a Mac or any machine before the arm is connected. Without the `[hardware]` extra installed, the check validates your config file and API key against the model server without touching servo hardware. It exits 0 and names the arm and cameras it would use, or explains what's missing. The actual closed-loop trial requires the arm connected and calibrated — there is no mock run mode. ## Tasks [#tasks] The SO-101 model is open-vocabulary and language-conditioned. `--task` is free-form text, not a fixed key: ```bash uv run python3 run.py --task "pick up the block and place it in the bin" ``` To see example task descriptions: ```bash uv run python3 run.py --list-tasks ``` This prints clearly-illustrative example prompts and notes what the model was trained on. There is no fixed task-key registry — any reasonable natural-language task description is valid input. ## Agent-driven rigs [#agent-driven-rigs] If you're running `run.py` from an agent harness rather than an interactive terminal: **SSH install approval.** Agent harnesses may require human approval for `git+ssh://` dependency installs. This is expected — approve the `newt` and `lerobot` install steps and continue. **Cleanup in non-TTY environments.** Ctrl+H triggers `emergency_home()` — moves the arm to its rest pose, then disconnects — but it's a stdin listener that only arms when stdin is a TTY, so it's unavailable in a non-TTY agent session. SIGINT/Ctrl+C triggers the weaker path, `teardown()`: it disconnects without homing, so the arm goes limp wherever it stopped. In a non-TTY agent session, Ctrl+C may not reach the process — use `timeout --signal=INT` to send SIGINT explicitly: ```bash timeout --signal=INT 60 uv run python3 run.py --task "pick up the block and place it in the bin" ``` This disconnects the arm after 60 seconds; it does not home it first. ## When to use it [#when-to-use-it] Use this starter when you have an SO-101 arm and want [MolmoAct-2 SO-101](/docs/models/molmoact2/so101) driving it. The starter runs open-vocabulary tasks — you supply a natural-language description with `--task`. `uv run python3 run.py --reset` moves the arm to a safe rest pose without running inference. Requires Python 3.11–3.13. Python ≥ 3.14 is not supported; it hits a torch compatibility wall. ## Setup [#setup] For the wire contract — state shape, action shape, cameras, and caveats — see the [SO-101 model page](/docs/models/molmoact2/so101). The repo's own README is the other reference, covering the full install sequence, calibration, camera index enumeration, and troubleshooting.