# Calling the model
URL: /docs/calling-the-model
The robot.infer() call and the InferenceResponse it returns — action chunk, axis labels, latency.
`robot.infer(obs)` sends one observation to the model and returns an `InferenceResponse` — the action chunk the model predicts for that observation, plus the labels and timing you need to interpret it. For a walkthrough of making this call against a recorded snapshot, see [Getting started](/docs/getting-started#make-a-test-call).
## The call [#the-call]
```python
from newt import Robot, snapshots
robot = Robot()
obs = snapshots.load("red_cube")
response = robot.infer(obs)
```
`infer()` takes one observation — robot state plus camera frames in the model's contract shape. A snapshot from `snapshots.load()` satisfies the contract out of the box; a real integration passes reads from its own hardware. One observation in, one response back; no loop runs and no robot is involved.
## `InferenceResponse` [#inferenceresponse]
```python
print(response)
# action_chunk (30, 6): shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper | latency 261ms
```
The printed form shows the three fields in one row: chunk shape, axis labels, round-trip latency.
| Field | What it is |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `action_chunk` | `(30, 6)` float32 ndarray — 30 target joint configurations, one per row. |
| `axes` | `['shoulder_pan', 'shoulder_lift', 'elbow_flex', 'wrist_flex', 'wrist_roll', 'gripper']` — semantic label for each action dimension. |
| `latency_ms` | Round-trip time for the final attempt's send and receive, in milliseconds. Excludes connect time and any cold-start retry. |
| `model` | Resolved model identifier (UID or tag), or `None` when it isn't known. |
| `total_ms` | Wall-clock time for the whole `infer()` call, in milliseconds — connect (including any cold-start retry), every failed attempt and its backoff, and the final send/recv. Equals `latency_ms` when nothing but the final attempt happened. |
| `retries` | Count of retries the call needed — a cold-start connect retry and/or transient verifier retries. `0` on the common warm path. |
## Action chunk [#action-chunk]
Each row of the chunk is one target joint configuration: `shoulder_pan`, `shoulder_lift`, `elbow_flex`, `wrist_flex`, `wrist_roll` in radians, and a normalized `gripper` value. Rows are ordered in time — the model's plan for the next 30 steps.
## Axis labels [#axis-labels]
`axes` comes from the model's registry contract (`action_axes`), so you read what each dimension means instead of guessing from float indices. The labels arrive with the contract at `Robot()` construction — see [model discovery](/docs/api#model-discovery) in the API reference.
## Latency [#latency]
`latency_ms` measures one round trip. The first call after an idle period wakes the model's container; that cold start can take a few minutes. If the first connect attempt times out, the SDK retries once and surfaces a `ColdStartRetry` warning while it waits. `total_ms`, not `latency_ms`, is the field that includes this cold-start time — check it when a call takes longer than the round-trip figure suggests. Warm calls return in a few seconds.
## Other models [#other-models]
The chunk shape and axis labels vary by model. [MA2-SO-101](/docs/models/molmoact2/so101) returns `(30, 6)` joint configurations. The registry contract is the source of truth for any model's shapes — see [model discovery](/docs/api#model-discovery) and the per-model pages under [Models](/docs/models).
---
# Getting started
URL: /docs/getting-started
Install newt, authenticate, and confirm you can reach the API. No robot required.
Your setup path is [missionrobotics.ai/hackers](https://missionrobotics.ai/hackers) — the hackathon kit installs everything in one pass, and its guide is written for the event. The steps below describe the general New Theory setup.
**Agents:** append `.md` to any page URL, or fetch [`/llms-full.txt`](/llms-full.txt), for machine-readable docs.
**Using Claude Code? Install with our setup skill.** After installing newt (step 1), run `newt skill install` — it equips your agent with a `newt-onboarding` guide skill. Ask it *get me set up* and it walks this page step by step. Starter kits ship the same skill, so it's already there if you clone one later.
New Theory serves action policies for robots: you send what the robot is sensing, and a hosted model returns the poses it should move to next.
In this guide you install the SDK, authenticate with an API key, confirm the API answers, and — optionally — make a test inference call against a recorded observation. It takes about ten minutes, most of which is the install. The finish line is "I reached the API and it answered." No robot is required.
Already have newt installed and authenticated? Skip to [Set up your embodiment](/docs/set-up-your-embodiment) to wire a real arm with a starter kit.
## Three ways to use New Theory [#three-ways-to-use-new-theory]
New Theory exposes hosted models through three surfaces.
* **[API](/docs/api).** A WebSocket endpoint that takes observation frames and returns action chunks. Use it when you write a custom client or call from a non-Python language. See the [API reference](/docs/api).
* **[SDK](/docs/newt-sdk).** The `newt` Python library wraps the WebSocket, the msgpack framing, and the inference loop behind two callbacks and a `robot.run(prompt)` call. Most developers start here. See the [SDK reference](/docs/newt-sdk).
* **[Starter kits](/docs/starters).** Hardware-specific bundles that wire a real arm and cameras to the SDK. Use one when you move from recorded observations to a physical robot. See [Set up your embodiment](/docs/set-up-your-embodiment).
## 1. Install newt [#1-install-newt]
Install `newt` as a global CLI tool from the public `newt-python` repo. The HTTPS command below clones anonymously — no GitHub account or access setup needed.
```bash
uv tool install "git+https://github.com/new-theory-research/newt-python.git"
```
On a machine set up for SSH-based GitHub auth instead, use the SSH URL:
```bash
uv tool install "git+ssh://git@github.com/new-theory-research/newt-python.git"
```
Requires Python 3.11 or newer.
**Agent-driven install:** `git+ssh://` installs may require a human-approval step in your agent's harness — that's expected behavior, not a hang; approve and the install continues.
No `uv` yet? `curl -LsSf https://astral.sh/uv/install.sh | sh`, then `source $HOME/.local/bin/env` to put it on PATH — details in the [full install guide](/docs/install).
This installs `newt` as a global command available in every shell — no environment to activate. If `newt` isn't on PATH after install, run `uv tool update-shell` to add `~/.local/bin`.
## 2. Authenticate the SDK with a New Theory API key [#2-authenticate-the-sdk-with-a-new-theory-api-key]
```bash
newt login
```
Running `newt login` creates an API key and asks you to confirm you're pairing with this session. If you don't have an account yet, you'll need to make one. The command prints a URL and a pairing code: open the URL in a browser, confirm the code matches the one in your terminal, and approve. The CLI stores the key at `~/.nt/credentials`, and the SDK and CLI read it from there automatically on subsequent runs — no shell export needed.
In environments without a browser (CI, an agent, a bare SSH session), set `NT_API_KEY` in the environment instead. The SDK and CLI both read it, and `NT_API_KEY` takes precedence over the stored credentials file when both are present.
To generate an API key manually, do so from the [New Theory Console](https://newtheory-console.vercel.app) — see [Key management](/docs/authentication/key-management) for the full flow.
## 3. Check the model registry [#3-check-the-model-registry]
```bash
newt models
```
```
Key nt_••••••••5bde16a3 (environment)
ft-64cad948c4e9f09c-red-cube-bowl [base: ft_base_so101_ft]
ft-64cad948c4e9f09c-svla-so101-pickplace-20260714213146 [base: ft_base_so101_ft]
ft-64cad948c4e9f09c-my-first-pickplace [base: ft_base_so101_ft]
molmoact2
so101
```
The registry answers with a `Key` line naming which credential answered (masked — never the full key), then the available models grouped by family — base models with their action axes, fine-tunes beneath them. Raw uids are left out by default; pass `--ids` if you need them. This is your first contact with the API. If `newt models` returns a list, you're connected.
`newt status` shows your key source, identity, and whether the live registry is reachable — useful for diagnosing auth or connectivity issues.
## 4. Connect from Python [#4-connect-from-python]
The New Theory Python library installs into a project — you add it when you have Python code to write. Create one if you don't have it yet:
```bash
uv init my-robot
cd my-robot
uv add "newt @ git+https://github.com/new-theory-research/newt-python.git"
```
On a machine set up for SSH-based GitHub auth instead, use `uv add "newt @ git+ssh://git@github.com/new-theory-research/newt-python.git"`.
Then confirm the API answers:
```bash
uv run python -c "from newt import Robot; print(Robot())"
# expected output:
# so101 · contract received · (30,6) · 6 labeled axes
```
`Robot()` resolves the default model and fetches its contract from the registry — output shape, axis labels, everything you need to know about what the model returns. It reads the same credentials `newt login` created — no second login, no environment variable needed.
**You've successfully communicated with the API.** The model contract is in your hands. Some developers are done at this step.
## Make a test call [#make-a-test-call]
This step is optional and needs no hardware. Run this section if you want to see what the model returns before wiring anything physical.
This is a test call against a recorded observation. `snapshots.load()` replays a saved camera-and-state snapshot; the model returns what it would do given that snapshot. No robot is connected. Nothing moves anywhere. Run the code below in the project you set up in step 4, using `uv run python`.
### Load a snapshot [#load-a-snapshot]
```python
from newt import Robot, snapshots
robot = Robot()
obs = snapshots.load("red_cube")
```
A snapshot is one recorded observation — robot state, camera frames, and the task prompt the episode was collected with. `snapshots.available()` lists all bundled recordings, including `cup_stacking` and `pour_coffee_beans`.
### Run inference [#run-inference]
```python
response = robot.infer(obs)
print(response)
# action_chunk (30, 6): shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper | latency 261ms
```
The first inference call wakes the model's container — it can take a few minutes. Subsequent calls return in a few seconds.
That printed line is the model's response in one row: the action chunk shape, the labeled axes, and the round-trip latency. Nothing moved, because there's no robot in this call — you confirmed the request and response shapes a real integration builds on. For the full anatomy of the response — every field, what each axis means, latency behavior — see [Calling the model](/docs/calling-the-model).
### If the call fails [#if-the-call-fails]
* **`newt.AuthError`** — your key is wrong, revoked, or not visible in this shell. The registry validates the key during `Robot()` construction, so a bad key fails fast — before any connection opens — with a hint that valid keys start with `nt_`. Re-run `newt login` or reissue the key.
* **A `ColdStartRetry` warning that lingers** — the GPU container is warming up; first call after idle can take a few minutes. The call completes on its own, and later calls in the same session skip the wait.
* **`TimeoutError`** — the container did not become ready within the SDK's 180-second retry window. This is not a routine cold start. Retry once; if it persists, contact support.
## Next: set up your embodiment [#next-set-up-your-embodiment]
In the next guide you set up an embodiment: wire the SDK to a real arm and cameras and drive the model on hardware. Continue to [Set up your embodiment](/docs/set-up-your-embodiment) — choose your hardware and follow its starter-kit walkthrough.
---
# Welcome to New Theory
URL: /docs
Build with the New Theory SDK and inference API.
**Agents:** append `.md` to any page URL, or fetch [`/llms-full.txt`](/llms-full.txt), for machine-readable docs.
New Theory hosts robotics models behind an API. Your code sends what the robot is sensing — joint state, camera images, a plain-English task. The model returns a plan of poses for the robot to follow. Pick a starting point below.
## Start building [#start-building]
Install the SDK, issue a key, and run your first inference call.
The Python library reference: `Robot()`, the `Embodiment` protocol, and the `run()` loop.
Wire-level protocol. For custom clients or non-Python integrations.
Create a key in the console and start making authenticated calls.
## Learn more [#learn-more]
Push a LeRobot dataset to New Theory and train with `newt finetune`.
---
# Install newt
URL: /docs/install
Install the newt CLI as a global tool from the public newt-python repo. Takes about five minutes.
New Theory ships from a public GitHub repo today, not PyPI. The HTTPS install command below clones `new-theory-research/newt-python` anonymously — no GitHub account or access setup needed. A public PyPI release is eventually planned.
## Install the CLI with uv tool [#install-the-cli-with-uv-tool]
`uv tool install` puts `newt` on your PATH as a global command — no virtual environment to create or activate:
```bash
uv tool install "git+https://github.com/new-theory-research/newt-python.git"
```
On a machine set up for SSH-based GitHub auth instead, use the SSH URL:
```bash
uv tool install "git+ssh://git@github.com/new-theory-research/newt-python.git"
```
If `uv` is not installed: `curl -LsSf https://astral.sh/uv/install.sh | sh`, then `source $HOME/.local/bin/env`.
If `newt` isn't on PATH after install, run `uv tool update-shell` — this adds `~/.local/bin` to your shell profile. uv's own installer handles it automatically for new installs; a manual shell may need it once.
Python 3.11 or newer is required. `uv python install 3.12` if you need it.
## Add the library to a Python project [#add-the-library-to-a-python-project]
The `newt` Python library installs into a project, not globally. You add it when you have Python code to write:
```bash
uv init my-robot
cd my-robot
uv add "newt @ git+https://github.com/new-theory-research/newt-python.git"
```
On a machine set up for SSH-based GitHub auth instead, use `uv add "newt @ git+ssh://git@github.com/new-theory-research/newt-python.git"`.
`uv run python` resolves the project environment regardless of shell state — no activation needed, agent-safe:
```bash
uv run python -c "from newt import Robot"
```
Starter kits run `uv sync` on first setup and handle the library install automatically.
## SSH-only machines [#ssh-only-machines]
If your machine routes all GitHub traffic over SSH, add the insteadOf rewrite before any install step that fetches `git+https://` URLs:
```bash
git config --global url."git@github.com:".insteadOf "https://github.com/"
```
## HTTPS-only machines [#https-only-machines]
If your machine authenticates to GitHub over HTTPS (the `gh` CLI credential helper, for example) and has no SSH key registered, add the reverse rewrite before any install step that fetches `git+ssh://` URLs:
```bash
git config --global url."https://github.com/".insteadOf "git@github.com:"
```
## Verify [#verify]
```bash
newt --version
```
Or continue to [Getting started](/docs/getting-started) — `newt login` in step 2 confirms the tool is on PATH and can reach the API.
## Next [#next]
[Getting started →](/docs/getting-started) — authenticate with an API key and confirm the API answers. No hardware needed.
---
# New Theory SDK
URL: /docs/newt-sdk
Python SDK for calling the New Theory API.
The New Theory SDK is a Python library for the New Theory inference API. You write two callbacks — one that returns what your robot is sensing, one that moves your robot — and call `robot.run(prompt)`. The SDK handles the network connection, the message format, and the inference loop.
Source repo: [`new-theory-research/newt-python`](https://github.com/new-theory-research/newt-python). Import name: `newt`.
Install today is from the `newt-python` repo, not PyPI. A public PyPI release is eventually planned; this page will update when it lands.
## What it does [#what-it-does]
* **The loop.** Calls your `read_state()` to get an observation, sends it to the model, receives the plan, calls your `execute(chunk)` with it. Repeats until the model emits a terminal frame or `max_duration` elapses.
* **Connection.** Opens one authenticated WebSocket per `robot.run()` call, holds it for the duration of the task, closes it cleanly when the run ends.
* **Framing.** Encodes observation dicts as msgpack (with the msgpack-numpy ndarray extension) on the way out; decodes action chunks back into NumPy arrays on the way in. You never touch the wire format.
* **Bootstrap.** On `Robot()` construction, the SDK contacts the always-on registry to resolve the model tag and fetch its contract. A bad or revoked key fails here, before any WebSocket is opened, with `AuthError` and a hint that valid keys start with `nt_`. If the registry itself is unreachable, `RegistryUnavailable` is raised (check `NT_BOOTSTRAP_URL` or retry). Two environment overrides exist for infra testing: `NT_BOOTSTRAP_URL` replaces the registry URL and `NT_INFERENCE_URL` pins the inference endpoint. Leave both unset for production runs.
* **Auth.** Resolves your API key — `NT_API_KEY` env var first, then `~/.nt/credentials` — and presents it as a Bearer token on the WebSocket handshake. A key revoked mid-session raises `AuthError` at the next connection.
* **Error mapping.** Translates WebSocket close codes into typed Python exceptions, all subclasses of `newt.NewTheoryError`: `AuthError` (4001), `ForbiddenError` (4403), `ProtocolError` (4400), `ModelNotFoundError` (4404), `ContractMismatchError` (4422), `ServerError` (4500), and `VerifierError` (4503) — plus a codeless `ConnectionDroppedError` for abnormal drops. The full close-code table lives in the [API reference](/docs/api).
## What it doesn't do [#what-it-doesnt-do]
The SDK does not run inference — the model lives on New Theory's hosted endpoint, and the SDK is the client that calls it. It does not drive the robot; the SDK hands action chunks to your `execute` callback, which decides how to apply them. It does not bundle a simulator, a viewer, or a training stack.
## Call shape [#call-shape]
```python
import os
import newt
robot = newt.Robot(
api_key=os.environ["NT_API_KEY"],
read_state=read_my_robot, # you write this
execute=move_my_robot, # you write this too
model="so101",
)
robot.run("pick up the cup", max_duration=30.0)
```
`newt.Robot(...)` takes your API key, two callbacks, and the model name. `robot.run(prompt, max_duration=...)` opens the connection, drives the loop, and returns when the model signals the task is complete or `max_duration` elapses. The return value names how the run ended.
See [Set up your embodiment](/docs/set-up-your-embodiment) for wiring the callbacks into real hardware.
## Embodiment [#embodiment]
An embodiment is any object that drives your hardware. It implements two methods, and you pass it to `Robot(embodiment=obj)` instead of the two callbacks.
```python
class Embodiment(Protocol):
def read_state(self) -> dict: ...
def execute(self, action_chunk: np.ndarray) -> None: ...
```
`read_state()` returns one observation dict — the same shape `read_state=` returns. `execute(action_chunk)` applies one action chunk, an ndarray of shape `(action_horizon, action_dim)`. The protocol is `runtime_checkable`, so any object with both methods qualifies; no inheritance or registration is required.
`Robot(embodiment=obj)` is equivalent to passing `read_state=obj.read_state` and `execute=obj.execute` separately. The two paths produce the same run. Passing both `embodiment=` and either callback raises `EmbodimentError` with `type` `embodiment.conflict` — they are mutually exclusive. Passing a string raises `EmbodimentError` with `type` `embodiment.string_not_object`, because the SDK consumes objects and never resolves names. See [Embodiment errors](/docs/api/errors#embodiment) for all three failure types.
Bare callbacks remain supported and undeprecated. `embodiment=` is shorthand for the object you already build; if your style is two closures, two closures still work.
### Construction [#construction]
Starter kits stamp an embodiment class into your repo as `embodiment.py`. You construct it and pass the instance:
```python
from newt import Robot
from embodiment import SO101
robot = Robot(embodiment=SO101.from_config())
robot.run("Pick up the red cup")
```
`from embodiment import SO101` resolves because the file lives in your repo — the [starter kit](/docs/starters/so101) put it there. `from_config()` reads the hardware addresses from `~/.config/nt/nt.toml`. The class is yours to rename and edit. To write your own for any hardware, see [Set up your embodiment](/docs/set-up-your-embodiment).
## Install [#install]
The `newt` SDK lives in the public [`new-theory-research/newt-python`](https://github.com/new-theory-research/newt-python) repo. The HTTPS command clones anonymously — no GitHub account or access setup needed:
```bash
pip install git+https://github.com/new-theory-research/newt-python.git
```
On a machine set up for SSH-based GitHub auth instead, use the SSH URL:
```bash
pip install git+ssh://git@github.com/new-theory-research/newt-python.git
```
Then `import newt` works.
PyPI publication is on the roadmap; when it lands, this becomes a single `pip install newt` line.
## Try it [#try-it]
For the full walk-through — issuing a key, running the script, reading what comes back — see [Getting started](/docs/getting-started).
---
# Set up your embodiment
URL: /docs/set-up-your-embodiment
Take your first inference call from a mock smoke test to a real robot using an embodiment starter kit.
You've made an inference call against the API and watched action chunks come back ([Getting started](/docs/getting-started)). Those chunks ran against mock callbacks. This page is the fast route to running them on real hardware.
## How embodiments connect [#how-embodiments-connect]
An embodiment is a class you own that implements `read_state()` and `execute()`. For the embodiments New Theory supports, you get that class from a **starter kit** — a project you clone and own. The kit's centerpiece is `embodiment.py`, where the class lives; the kit also ships the project structure, dependencies, config, and a thin entry script that constructs the class and passes it to `Robot(embodiment=...)`. You fill in your hardware specifics and run it. For the full picture, see [How to use embodiment starter kits](/docs/starters).
## The fast path [#the-fast-path]
1. **Pick your embodiment.** Each starter pairs one robot with the model that drives it. The list lives on the [starters page](/docs/starters).
2. **Clone its starter repo.** Each starter is a standalone Git repository you clone and modify. You own the result after clone, including `embodiment.py` and its class.
3. **Fill in your hardware config.** Copy the starter's config template to `~/.config/nt/nt.toml` and edit it with your specifics — the arm's serial port and your two camera indices. The class reads this file in `from_config()`; the starter's README names the exact fields.
4. **Run the entry script.** With your `NT_API_KEY` exported and the config filled in, run the starter's entry script. It constructs the embodiment class, passes it to `Robot(embodiment=...)`, opens a WebSocket to the model, streams observations from your hardware, and executes the returned chunks on the arm.
## SO-101 [#so-101]
The SO-101 is a low-cost arm driven by MolmoAct-2 SO-101. Start with the [SO-101 starter](/docs/starters/so101) for what's in the kit — clone URL, config fields, camera setup, calibration flow, and task flags. The model is open-vocabulary: you supply a natural-language `--task` description at run time.
## No starter for your embodiment yet? [#no-starter-for-your-embodiment-yet]
Any class with `read_state()` and `execute()` is an embodiment. Write one for any hardware: `read_state()` returns the robot's current state and camera frames, and `execute()` applies a `(30, 6)` action chunk to the arm. Pass an instance to `Robot(embodiment=your_rig)`. No inheritance or registration — the two methods are the whole contract. See the [SDK reference](/docs/newt-sdk#embodiment) for the protocol.
You can also pass the two methods as bare callbacks — `Robot(read_state=..., execute=...)` — which produces the same run and stays supported. The [SDK reference](/docs/newt-sdk#call-shape) shows both as mocks; a real integration replaces them with reads and writes against your hardware. More starters are coming as we add embodiments.
## Safety is client-side [#safety-is-client-side]
The API has no emergency-stop or recovery primitive — stopping the robot is your code's job. The starter kits wire their own stop paths: a keyboard abort during the trial and a blocking move to a safe rest pose afterwards. A custom integration needs the same, in addition to the hardware e-stop your arm ships with.
---
# Errors
URL: /docs/api/errors
Error types the New Theory inference API emits, grouped by WebSocket close code.
Every abnormal WebSocket close includes a JSON payload (the error envelope) with six fields.
```json
{
"code": 4001,
"type": "auth.invalid_key",
"message": "API key rejected. The key was revoked, never issued, or malformed. Generate a new key in the New Theory console.",
"context": { "key_prefix": "nt_a1b2c3d4" },
"docs": "https://docs.newtheory.ai/api/errors",
"trace_id": "tr_a1b2c3d4"
}
```
| Field | Type | Notes |
| ---------- | -------------- | ---------------------------------------------------------------------------------------- |
| `code` | `integer` | WebSocket close code. One of `4001`, `4403`, `4400`, `4404`, `4422`, `4500`, `4503`. |
| `type` | `string` | Dot-namespaced subtype (`domain.specific`). Stable identifier for programmatic handling. |
| `message` | `string` | Human-readable description. Do not parse; use `type` for branching. |
| `context` | `object` | Structured diagnostics. Fields vary by `type` — see each entry below. |
| `docs` | `string` (URI) | Link to this page's anchor for the error type. |
| `trace_id` | `string` | Opaque support token (`tr_xxxxxxxx`). Include in support requests. |
In the Python SDK, every close triggers an exception that inherits from `newt.NewTheoryError`. Catch by class for domain-level handling, branch on `exc.type` for specific cases.
```python
import newt
try:
robot.run("pick up the cup")
except newt.AuthError as e:
print(e.type, e.trace_id) # "auth.invalid_key", "tr_a1b2c3d4"
except newt.NewTheoryError as e:
print(e.code, e.type, e.message)
```
Every error the server can emit is registered in a single-source catalog, and server startup validates coverage — a new error type can't ship without an entry.
***
## Raised before connection [#raised-before-connection]
These conditions are detected by the SDK during `Robot()` construction, before any WebSocket is opened.
### `RegistryUnavailable` [#registryunavailable]
The registry service (`nt-registry-production.up.railway.app`) was unreachable when `Robot()` tried to resolve the model and contract. This is the only condition that raises `RegistryUnavailable` — it means the registry itself is down or the host cannot reach it, not a model or auth problem.
**Next step:** check your network, verify `NT_BOOTSTRAP_URL` is not set to a bad value (leave it unset to use the production registry), and retry. If the registry is down, wait and retry — it is an always-on service and outages are brief.
### `AuthError` at construction [#autherror-at-construction]
Your API key failed validation at the registry during `Robot()` construction — the registry returned a 401 with a hint that valid keys start with `nt_`. The connection closes before any WebSocket is opened.
**Next step:** re-export `NT_API_KEY` with the correct key (`nt_` prefix + 40 hex chars), or generate a new key in the [console](/docs/authentication).
### `ColdStartRetry` warning [#coldstartretry-warning]
Not an error — a `logging.WARNING` the SDK emits when the GPU container did not respond within the initial connection timeout and the SDK is opening an extended retry (180s). This is the normal cold-start signal. Wait it out; a cold container can take a few minutes to come up. If the extended retry also times out, a `TimeoutError` is raised to your code.
***
## 4001 — Authentication [#4001--authentication]
Sent when the API key fails verification at the WebSocket handshake. The connection closes before any inference runs.
### auth.invalid\_key [#authinvalid_key]
**SDK exception:** `newt.AuthError`
The submitted key was revoked, never issued, or malformed.
```json
{
"code": 4001,
"type": "auth.invalid_key",
"message": "API key rejected. The key was revoked, never issued, or malformed. Generate a new key in the New Theory console.",
"context": {
"key_prefix": "nt_a1b2c3d4"
},
"trace_id": "tr_a1b2c3d4"
}
```
`context.key_prefix` is the first eight characters of the submitted key — enough for diagnostics without leaking the full value.
**Next step:** generate a new key in the [console](/docs/authentication). If the key was just created, confirm it has not been revoked.
***
## 4403 — Authorization [#4403--authorization]
Sent when the API key is valid but does not own the requested model. A fine-tune belongs to the team that made it — another team's key can't serve it. The connection closes before any inference runs.
### auth.forbidden [#authforbidden]
**SDK exception:** `newt.ForbiddenError`
```json
{
"code": 4403,
"type": "auth.forbidden",
"message": "That model belongs to another team. Your key can only serve models your team owns.",
"context": {
"model": "ft_6341c5_d13da9"
},
"trace_id": "tr_a1b2c3d4"
}
```
`context.model` is the requested model identifier, as submitted.
**Next step:** use one of your own team's models, or ask the owning team to share. A model's owner is the team that fine-tuned it.
***
## 4400 — Protocol [#4400--protocol]
Sent when a received frame cannot be decoded or is structurally invalid. The connection closes on the first bad frame. The Python SDK raises `newt.ProtocolError` for all 4400 subtypes.
### protocol.malformed\_msgpack [#protocolmalformed_msgpack]
**SDK exception:** `newt.ProtocolError`
A binary WebSocket frame could not be decoded as msgpack.
```json
{
"code": 4400,
"type": "protocol.malformed_msgpack",
"message": "The server received a WebSocket frame that could not be decoded as msgpack. Ensure every frame is msgpack-encoded binary.",
"context": {
"frame_bytes": 42
},
"trace_id": "tr_a1b2c3d4"
}
```
`context.frame_bytes` is the size of the frame that failed to decode.
**Next step:** use `msgpack.packb(obs, use_bin_type=True)` to encode each frame. Text WebSocket frames are not accepted.
### protocol.missing\_type [#protocolmissing_type]
**SDK exception:** `newt.ProtocolError`
A decoded msgpack frame did not contain the required `type` key.
```json
{
"code": 4400,
"type": "protocol.missing_type",
"message": "The server received a msgpack frame with no 'type' key. Every frame must include a 'type' key set to 'obs' or 'stop'.",
"context": {
"keys_present": ["state", "images"]
},
"trace_id": "tr_a1b2c3d4"
}
```
`context.keys_present` lists the keys the server found in the frame.
**Next step:** ensure every frame dict includes a `"type"` key before packing.
### protocol.unknown\_type [#protocolunknown_type]
**SDK exception:** `newt.ProtocolError`
A decoded msgpack frame had a `type` value the server does not recognize.
```json
{
"code": 4400,
"type": "protocol.unknown_type",
"message": "The server received a frame with an unrecognized type. Accepted frame types are: 'obs', 'stop'.",
"context": {
"received_type": "observation"
},
"trace_id": "tr_a1b2c3d4"
}
```
`context.received_type` is the value the server received.
**Next step:** `type` must be exactly `"obs"` (to send an observation) or `"stop"` (to end the session).
***
## 4404 — Model not found [#4404--model-not-found]
Sent when the model identifier in the first `obs` frame does not match any UID or tag in the server's registry.
### model\_not\_found.unknown\_identifier [#model_not_foundunknown_identifier]
**SDK exception:** `newt.ModelNotFoundError`
```json
{
"code": 4404,
"type": "model_not_found.unknown_identifier",
"message": "Model not found. Check spelling or call newt.list_models() to see available models.",
"context": {
"requested": "so101-v2",
"known_uids": ["ft_base_molmoact2", "ft_6341c5_d13da9"],
"known_tags": ["so101"]
},
"trace_id": "tr_a1b2c3d4"
}
```
`context.requested` is the identifier from the `obs` frame. `context.known_uids` and `context.known_tags` list what the registry contained at the time.
**Next step:** model identifiers are case-sensitive. Run `newt.list_models()` to see current UIDs and tags.
***
## 4422 — Contract mismatch [#4422--contract-mismatch]
Sent when an `obs` frame's data does not match the model's trained input contract. Each subtype names the specific field that failed validation. The Python SDK raises `newt.ContractMismatchError` for all 4422 subtypes.
### contract\_mismatch.state\_shape [#contract_mismatchstate_shape]
**SDK exception:** `newt.ContractMismatchError`
The state array shape does not match the shape the model was trained on.
```json
{
"code": 4422,
"type": "contract_mismatch.state_shape",
"message": "State shape mismatch. Adjust read_state() to return the expected shape, or switch to a model that matches your robot.",
"context": {
"model": "so101",
"expected_shape": [6],
"got_shape": [14]
},
"trace_id": "tr_a1b2c3d4"
}
```
**Next step:** check the model's expected state shape via `robot.contract`. The contract is fixed at training time.
### contract\_mismatch.state\_dtype [#contract_mismatchstate_dtype]
**SDK exception:** `newt.ContractMismatchError`
The state array dtype does not match the dtype the model was trained on.
```json
{
"code": 4422,
"type": "contract_mismatch.state_dtype",
"message": "State dtype mismatch. Cast your state array to the expected dtype in read_state().",
"context": {
"model": "so101",
"expected_dtype": "float32",
"got_dtype": "float64"
},
"trace_id": "tr_a1b2c3d4"
}
```
**Next step:** most models expect `float32`. Use `state.astype(np.float32)` in `read_state()`.
### contract\_mismatch.camera\_missing [#contract_mismatchcamera_missing]
**SDK exception:** `newt.ContractMismatchError`
A camera the model requires was absent from the `obs` frame's `images` dict.
```json
{
"code": 4422,
"type": "contract_mismatch.camera_missing",
"message": "Required camera missing. Add the missing camera to the images dict in read_state(), or switch to a model that matches your hardware.",
"context": {
"model": "so101",
"missing_required_camera": "side",
"cameras_required": ["top", "side"],
"got_cameras": ["top"]
},
"trace_id": "tr_a1b2c3d4"
}
```
`context.missing_required_camera` names the absent key. `context.cameras_required` lists all keys the model requires.
**Next step:** check `cameras.required` via `robot.contract`. Each required camera key must appear in the `images` dict.
### contract\_mismatch.image\_shape [#contract_mismatchimage_shape]
**SDK exception:** `newt.ContractMismatchError`
A camera image's shape does not match the shape the model expects.
```json
{
"code": 4422,
"type": "contract_mismatch.image_shape",
"message": "Image shape mismatch. Resize the camera frame to the expected shape (CHW) in read_state().",
"context": {
"model": "so101",
"camera": "top",
"expected_shape": [3, 224, 224],
"got_shape": [3, 480, 640]
},
"trace_id": "tr_a1b2c3d4"
}
```
**Next step:** images must be in CHW layout. Check the expected shape via `robot.contract` and resize on the client.
***
## 4500 — Server error [#4500--server-error]
Sent when the server encounters an unhandled exception, either inside the model call or in the WebSocket handler itself. The Python SDK raises `newt.ServerError` for all 4500 subtypes. Include `exc.trace_id` in any support request.
### server.inference\_error [#serverinference_error]
**SDK exception:** `newt.ServerError`
The model raised an unhandled exception during `policy.infer()`.
```json
{
"code": 4500,
"type": "server.inference_error",
"message": "The model raised an error during inference. Retry the request. If this persists, contact support with the trace_id.",
"context": {
"error_type": "RuntimeError"
},
"trace_id": "tr_a1b2c3d4"
}
```
`context.error_type` is the Python exception class name from the inference call.
**Next step:** retry the request. If this persists, contact support with the `trace_id`. This error may indicate an edge case in the input data.
In an earlier protocol revision, in-model errors emitted a terminal frame with stop\_reason="error" and closed with code 1000. They now close 4500 with this envelope, so ServerError is raised rather than returning a RunResult with stop\_reason="error".
### server.internal [#serverinternal]
**SDK exception:** `newt.ServerError`
An unhandled exception occurred in the WebSocket handler outside the inference path.
```json
{
"code": 4500,
"type": "server.internal",
"message": "The server encountered an internal error and closed the session. Retry the request. If this persists, contact support with the trace_id.",
"context": {
"error_type": "ValueError"
},
"trace_id": "tr_a1b2c3d4"
}
```
**Next step:** retry the request. If this persists, contact support with the `trace_id`.
***
## Embodiment [#embodiment]
Raised by the SDK during `Robot()` construction when the value passed to `embodiment=` is invalid. This is a client-side check — no WebSocket is opened, and no server is involved. The SDK raises `newt.EmbodimentError` for all three subtypes; the `code` is `4422` by HTTP convention. See the [SDK reference](/docs/newt-sdk#embodiment) for the protocol an embodiment object must satisfy.
### embodiment.string\_not\_object [#embodimentstring_not_object]
**SDK exception:** `newt.EmbodimentError`
Raised when `embodiment=` receives a string. `embodiment=` takes your embodiment object, not a name string — the SDK never resolves names. This commonly happens when a developer expects a name-based API such as `Robot(embodiment="so101")`.
```json
{
"code": 4422,
"type": "embodiment.string_not_object",
"message": "Robot(embodiment=) takes your embodiment object, not a name string (got 'so101'). Generate one with a starter kit, or implement read_state() and execute() on any class.",
"context": { "got": "so101" }
}
```
`context.got` is the string the SDK received.
**Next step:** pass an object that implements `read_state()` and `execute()`. Generate one with a [starter kit](/docs/starters), or write your own class — see [Set up your embodiment](/docs/set-up-your-embodiment).
### embodiment.conflict [#embodimentconflict]
**SDK exception:** `newt.EmbodimentError`
Raised when `embodiment=` is combined with `read_state=` or `execute=`. The two paths are equivalent and mutually exclusive: `embodiment=` is convenience shorthand for passing the two callbacks separately.
```json
{
"code": 4422,
"type": "embodiment.conflict",
"message": "Robot() received both embodiment= and read_state=. Pick one path: pass embodiment= (an object with read_state() and execute()), or pass read_state= and execute= as separate callbacks. The two paths are equivalent; embodiment= is convenience shorthand.",
"context": { "conflict_kwargs": ["read_state="] }
}
```
`context.conflict_kwargs` lists the conflicting callback arguments the SDK saw.
**Next step:** pass `embodiment=` alone, or pass `read_state=` and `execute=` alone. Not both.
### embodiment.missing\_method [#embodimentmissing_method]
**SDK exception:** `newt.EmbodimentError`
Raised when the object passed as `embodiment=` is missing `read_state()`, `execute()`, or both. The message names every missing method.
```json
{
"code": 4422,
"type": "embodiment.missing_method",
"message": "The object passed as embodiment= is missing: execute(). An embodiment must implement both read_state() -> dict and execute(action_chunk) -> None.",
"context": { "missing": ["execute()"], "got_type": "PartialRig" }
}
```
`context.missing` lists the absent methods; `context.got_type` is the class name of the object that was passed.
**Next step:** implement both `read_state() -> dict` and `execute(action_chunk) -> None` on the class. Any object with those two methods is an embodiment — no inheritance or registration required.
***
## 4503 — Verifier unavailable [#4503--verifier-unavailable]
Sent when the key verification service is unreachable or returns an error during the WebSocket handshake. The connection closes before any inference runs.
### verifier.unavailable [#verifierunavailable]
**SDK exception:** `newt.VerifierError`
```json
{
"code": 4503,
"type": "verifier.unavailable",
"message": "The API key verification service is temporarily unavailable. Retry the request in a few seconds.",
"context": {
"error_type": "httpx.ConnectTimeout"
},
"trace_id": "tr_a1b2c3d4"
}
```
`context.error_type` is the Python exception class from the verification call.
**Next step:** this is a server-side infrastructure issue, not a problem with your API key. Retry with exponential backoff.
---
# API
URL: /docs/api
Reference for New Theory's Inference API
The New Theory API is how your code calls New Theory's hosted robotics models. You send the robot's current state and camera images; the API returns a chunk of target poses. Tasks and observations stream over WebSocket. When the task ends, the connection closes.
The wire schema below is illustrated with a single-arm joint-space contract as a worked example. Shapes and camera keys vary by model — always resolve the contract from the registry (see [Model discovery](#model-discovery)) rather than hardcoding them.
## How the API Works [#how-the-api-works]
Most developers don't write this protocol by hand. The New Theory SDK wraps the WebSocket, framing, and loop. See the [New Theory SDK](/docs/newt-sdk). Use this page if you're writing a custom client, hitting the endpoint from a non-Python language, or debugging the SDK.
A persistent network connection (WebSocket) carries binary messages encoded as msgpack-numpy — a compact binary format for NumPy arrays.
```python
import asyncio
import os
import msgpack
import msgpack_numpy
import numpy as np
import websockets
msgpack_numpy.patch()
# The inference endpoint for this model, resolved from the registry —
# see Model discovery below.
URL = "wss:///stream"
async def main():
# Authenticate with your API key (loaded from a shell env var).
headers = {"Authorization": f"Bearer {os.environ['NT_API_KEY']}"}
# Open one WebSocket per task. The handshake auth happens here.
async with websockets.connect(URL, extra_headers=headers) as ws:
# An observation describes the robot's current state for the model.
obs = {
"type": "obs",
# 6 joint values: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper.
"state": np.zeros(6, dtype=np.float32),
# Both cameras are expected: two RGB views of the workspace.
"images": {
"top": np.zeros((3, 224, 224), dtype=np.uint8),
"side": np.zeros((3, 224, 224), dtype=np.uint8),
},
"prompt": "pick up the cup", # plain-English task description
"max_duration": 5.0, # wall-clock cap in seconds; first frame only
}
# Encode the dict as a binary msgpack message and send it.
await ws.send(msgpack.packb(obs, use_bin_type=True))
# Receive one binary message back, decode it, pull the action chunk.
frame = msgpack.unpackb(await ws.recv(), raw=False)
chunk = frame["chunk"] # (30, 6) float32 — 30 target joint positions
asyncio.run(main())
```
The full control loop reads fresh state and sends the next `obs` after executing some prefix of the chunk. See [What you get back](#what-you-get-back).
## Model discovery [#model-discovery]
Inference endpoints are resolved from the model registry, not hardcoded. The first call a custom client makes is `GET /v1/models` against the always-on registry — the same call the SDK makes at `Robot()` construction:
```bash
curl -H "Authorization: Bearer $NT_API_KEY" \
https://nt-registry-production.up.railway.app/v1/models
```
The response is a JSON list with one object per model: its immutable `uid`, the `tags` you pass to `Robot(model=...)`, the `endpoint` to open the WebSocket against, and the `contract` the server enforces on your frames. Trimmed to one model:
```json
[
{
"uid": "ft_base_example",
"tags": [""],
"type": "base",
"base": null,
"endpoint": "wss:///stream",
"contract": {
"state_shape": [6],
"state_dtype": "float32",
"cameras": {
"required": [],
"expected": ["top", "side"]
},
"image_shape": [3, 224, 224],
"action_shape": [30, 6],
"action_axes": ["shoulder_pan", "shoulder_lift", "elbow_flex", "wrist_flex", "wrist_roll", "gripper"]
}
}
]
```
`GET /v1/models/{uid_or_tag}` returns a single entry. Fine-tune entries inherit `endpoint` and `contract` from their base; the registry returns them resolved. The registry runs on an always-on service, so discovery answers in under a second even when the GPU container serving inference is cold.
## Endpoint and auth [#endpoint-and-auth]
```
wss:///stream
```
This is the `endpoint` value the registry returns for the model you resolve. Resolve it from the registry rather than hardcoding it — the value changes as serving infrastructure moves.
Authenticate once, at the handshake:
```
Authorization: Bearer nt_a1b2c3d4...
```
The server verifies against the [console](/docs/authentication). On success, the key is trusted for the connection lifetime — no mid-stream re-auth. On failure, the server closes with code `4001`.
## What you send [#what-you-send]
The client drives inference by sending `obs` frames. Each one carries the robot's current state, camera images, and the task prompt.
```python
{
"type": "obs",
"state": np.ndarray, # float32 (6,) — joint positions, radians
"images": {
"top": np.ndarray, # uint8 (3, 224, 224) — CHW, RGB
"side": np.ndarray, # uint8 (3, 224, 224)
},
"prompt": "pick up the cup",
"max_duration": 30.0, # seconds; first frame only
}
```
`state` is the robot's joint positions, in the order the model's `action_axes` declares — for this worked example, shoulder\_pan, shoulder\_lift, elbow\_flex, wrist\_flex, wrist\_roll, then a normalised gripper value.
`images` are CHW uint8 at 224×224. Resize on the client to match — the server's transform pipeline expects this size, and full-resolution camera payloads exceed the connection's message size cap.
Send only the cameras the model expects. An entry for a camera the model doesn't expect — for example, a second arm's wrist camera left in your config file — fails inference server-side. Filter your config down to the model's camera keys before sending.
`max_duration` is read from the first `obs` only; ignored on later frames.
To stop a run early, send:
```python
{"type": "stop"}
```
## What you get back [#what-you-get-back]
The server emits one `action` frame per inference cycle:
```python
{
"type": "action",
"chunk": np.ndarray, # float32 (30, 6)
}
```
Each row of the chunk is a target joint position in the same format as `state`: `[shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper]`. The action horizon is 30 steps.
Clients typically don't execute all 30 steps before requerying. The common pattern is receding-horizon: take a prefix — `max_actions_per_chunk=15` is typical — at a steady cadence, often 15 Hz with `goal_time_s` smoothing between consecutive targets, then read fresh state and send the next `obs`. The model is trained for chunk overlap; tune the cut-off to your control loop.
## How runs end [#how-runs-end]
When the run ends, the server emits one `terminal` frame and then closes the connection:
```python
{
"type": "terminal",
"stop_reason": "max_duration",
}
```
| Value | Meaning |
| -------------- | ------------------------------------------------------------------------------------------------------ |
| `max_duration` | Wall-clock time since the first `obs` exceeded `max_duration`. The expected outcome on current models. |
| `interrupted` | Client sent a `stop` frame. |
Missing or wrong-shape fields produce degraded actions, not a dropped connection. The only hard reject at the frame level is an undecodable msgpack payload.
## Debugging connection issues [#debugging-connection-issues]
When something is wrong at the transport layer — auth, malformed frames, server errors — the WebSocket closes with one of these codes:
| Code | Meaning | SDK exception |
| ------ | ----------------------------------------------------------- | ----------------------- |
| `1000` | Normal closure after a `terminal` frame. | — |
| `4001` | Auth failed — missing, malformed, or revoked key. | `AuthError` |
| `4400` | Protocol error — undecodable frame or missing `type`. | `ProtocolError` |
| `4403` | Key is valid but doesn't own the requested model. | `ForbiddenError` |
| `4404` | Model identifier not found in the registry. | `ModelNotFoundError` |
| `4422` | An `obs` frame doesn't match the model's declared contract. | `ContractMismatchError` |
| `4500` | Server error outside the model call. | `ServerError` |
| `4503` | Verifier unavailable at handshake. | `VerifierError` |
Errors inside the model close abnormally too: the server sends a `server.inference_error` envelope and closes `4500`. See [Errors](/docs/api/errors) for the envelope shape.
To confirm the URL is reachable and your key works without writing client code, hit the handshake with `websocat`:
```bash
websocat \
-H "Authorization: Bearer $NT_API_KEY" \
--binary \
wss:///stream
```
A valid key holds the connection open; a bad one closes `4001`. `websocat` doesn't encode msgpack, so this verifies the handshake but won't drive inference.
## What might change [#what-might-change]
* **Future single-arm joint-space models** will share this wire shape — 6D joint state, two named cameras, `(30, 6)` action chunks. A client written against this page should keep working as the New Theory model family advances.
* **Other models during early development.** We serve MolmoAct2 and potentially other models to prototype the API with real developers; some may use different wire shapes. The long-term direction is one consistent wire across models; in the short term, expect shifts.
---
# Get an API key
URL: /docs/authentication
What an NT API key is, how it gates inference, and what happens on revocation.
NT uses bearer-token authentication. You issue a key — with `newt login` or in the [console](https://newtheory-console.vercel.app) — and the `newt` library presents it on every connection. Revoke a key in the console and the connections it backs start refusing new calls within seconds.
## What a key looks like [#what-a-key-looks-like]
NT keys carry a recognisable prefix:
```
nt_a1b2c3d4e5f60718293a4b5c6d7e8f9011223344
```
The `nt_` prefix identifies the key as an NT API key. The 40-character hex suffix is the secret. Treat the whole string as a password: it grants the holder the right to spend inference time against the account that issued it.
## How a key is presented [#how-a-key-is-presented]
The `newt` library reads `NT_API_KEY` from the environment and presents it on every connection. In code:
```python
from newt import Robot
robot = Robot() # uses NT_API_KEY from env
```
Or, explicitly:
```python
robot = Robot(api_key="nt_...")
```
On the wire, the key travels as a standard bearer token on the WebSocket handshake:
```
Authorization: Bearer nt_a1b2c3d4...
```
If you're hitting the endpoint without the SDK, that's the header to set.
## What happens on revocation [#what-happens-on-revocation]
When you revoke a key in the console, NT's inference endpoint stops honouring it on the next handshake.
`AuthError` can fire in two places:
* **At `Robot()` construction** — the SDK validates your key against the registry before opening any WebSocket. A bad, revoked, or malformed key fails here in about a second, with a hint that valid keys start with `nt_`. This is the earliest and most common failure point.
* **At the WebSocket handshake** — if a key is revoked mid-session, the next connection attempt fails at the handshake. The key is verified once, at handshake, and never re-checked after that, so an already-open connection keeps running until it closes on its own.
```python
from newt import Robot, AuthError
try:
robot = Robot()
for chunk in robot.run("pick up the cup", stream=True):
...
except AuthError:
# Key is invalid, revoked, or absent.
...
```
`AuthError` is the only auth-related exception the SDK raises. It covers wrong key, revoked key, and missing key — the error message distinguishes them; the type does not.
## Where to go next [#where-to-go-next]
* **Issue, list, or revoke a key.** See [Key management](/docs/authentication/key-management).
* **Get a key from the console.** Open the [console](https://newtheory-console.vercel.app) and click **Create key**.
---
# Key management
URL: /docs/authentication/key-management
Issue keys with newt login, and manage them — list, revoke, logout — from the CLI and the console.
`newt login` is the primary way to issue a key. The console is the management surface: it issues keys for contexts where the CLI isn't available, and it's where you list and revoke keys.
## Issue a key with `newt login` [#issue-a-key-with-newt-login]
```bash
newt login
```
The command prints a URL and a pairing code. Open the URL in a browser, confirm the code shown there matches the one in your terminal, and approve. The CLI receives a key and stores it at `~/.nt/credentials`. The SDK and CLI read it from there automatically on subsequent runs.
For scripted or agent contexts, `newt login --print` runs the same pairing flow but prints the key to stdout and persists nothing. Capture the output yourself — into a secret store, an environment variable, or a `.env` file.
```bash
newt login --print
```
## Issue a key in the console [#issue-a-key-in-the-console]
Use the console when the CLI isn't an option — a teammate provisioning a key, a deploy target without `newt` installed, or any environment where the browser pairing flow doesn't fit.
1. Open the [console](https://newtheory-console.vercel.app) and sign in.
2. Open the **Keys** page from the sidebar.
3. Click **Create API key**. The console reveals the key value once, in a dialog.
4. **Copy the value now.** It looks like `nt_<40 hex>`. Save it to your password manager or export it directly into the shell you'll run from.
**One-time reveal.** The console only shows the key plaintext at the moment of creation. After you close the dialog you can list the key (prefix only) and revoke it, but you can't see the full value again. If you lose the plaintext, revoke and reissue.
## List your keys [#list-your-keys]
The **Keys** page in the console shows every key issued under your account: name, masked key, creation date, and a status of **In use** or **Never used**. The full key value is never shown — the suffix hint (last 8 chars) is enough to identify which one is which when you cross-reference against the value you saved.
## Revoke a key [#revoke-a-key]
1. On the **Keys** page, find the key you want to revoke.
2. Click **Revoke** on its row.
3. Confirm. The key is removed from the active set.
Revocation propagates to the inference endpoint within seconds; new connections fail with `AuthError`. The key is checked once, at connection handshake — an already-open connection isn't re-checked, so it keeps running on a revoked key until it closes on its own.
## Remove local credentials with `newt logout` [#remove-local-credentials-with-newt-logout]
```bash
newt logout
```
`newt logout` deletes `~/.nt/credentials`. The key itself stays valid — `logout` only removes it from this machine. To make the key stop working, revoke it in the console.
## Practices that pay off [#practices-that-pay-off]
* **One key per project or robot.** When something leaks or a workstation gets re-imaged, you revoke one key, not all of them.
* **Name keys after where they live.** `laptop-mattie`, `demo-rig-1`, `ci-runner`. Future-you grepping a list of prefixes will thank you.
* **Rotate by reissue, not by edit.** Create the new key, swap your environment over, then revoke the old one. There is no "rotate" operation — issuing plus revoking is the rotation.
* **Don't commit keys to source.** Use `NT_API_KEY` from the shell, a `.env` file gitignored at the repo root, or your platform's secret store. The `newt` library reads `NT_API_KEY` first and falls back to `~/.nt/credentials` — the env var wins when both are present, so you don't have to thread plaintext through code.
---
# Push a dataset to New Theory
URL: /docs/integrations
How an external tool uploads a LeRobot dataset into New Theory so newt finetune can train on it — the one-line CLI hand-off, and the raw HTTP sequence for a native integration.
This page is for engineers integrating an external tool — a recording kit, an exporter, a data pipeline — that produces a LeRobot dataset and needs to get it into New Theory so `newt finetune` can train on it. The developer authenticates with their own `nt_` API key; no infrastructure credential ever leaves New Theory.
There are two doors, and they end in the same place — a dataset staged under the developer's account, referenced by name:
* **The CLI door.** Shell out to `newt finetune --dataset ./folder`. One call uploads the local export and launches the run. Reach for this first.
* **The HTTP door.** Sign, `PUT`, verify, launch — the raw request sequence, for a native integration that doesn't want to depend on the CLI.
## The CLI door [#the-cli-door]
If your tool has the exported dataset folder on disk, the whole hand-off is one subprocess call. `newt finetune` detects that the argument is a local folder (it contains a path separator, or resolves to an existing directory), uploads it under the developer's account, and launches training against the staged name:
```bash
newt finetune --dataset ./datasets/my_task
```
The CLI validates the export locally before a byte moves, uploads every file in one signing round-trip, then launches and watches the run:
```
Detected a local folder — uploading 214 MB… staged as my_task
… 342/342 files (214 MB / 214 MB) 100%
uploaded 214 MB, staged as my_task
Launched fine-tune on dataset 'my_task'.
job handle:
watch page: https://newtheory-console.vercel.app/runs/
```
The staged name is the folder's basename (`my_task` above), so it must be a valid name: letters, digits, `.`, `_`, `-`, up to 128 characters. Rename the folder if it isn't.
The developer's key is read from `NT_API_KEY`, or from `~/.nt/credentials` after `newt login`. Install the CLI with `pip install "git+https://github.com/new-theory-research/newt-python.git"`.
If your integration is already Python and you want to upload without launching — for example, to stage a dataset now and let the developer launch later — call the upload primitive directly:
```python
from newt.recording import NTCloudSink
sink = NTCloudSink("my_task", api_key="nt_…")
namespace = sink.upload_directory("./datasets/my_task")
```
`upload_directory` validates the export, signs every file in one round-trip, `PUT`s each one, and writes a completeness marker last. It raises on any failure rather than leaving a partial upload that looks complete. This needs the recording extra: `pip install "newt[recording]"`.
## What the dataset must contain [#what-the-dataset-must-contain]
Both doors upload a **LeRobot v3 dataset directory**. Training reads the dataset's schema from `meta/info.json`; the intake gate rejects a dataset that doesn't meet the contract before spending any GPU time.
The directory layout:
```
my_task/
meta/
info.json # schema: features, shapes, robot_type
tasks.parquet # the task strings, one per task index
stats.json # per-feature normalization stats (quantiles)
data/
chunk-000/
file-000.parquet # per-frame robot state + action
videos/
chunk-000/
observation.images.cam0/… # camera video, when features declare dtype "video"
```
This is a standard LeRobot v3 export — the same layout `lerobot` writes and the Rerun kit produces. Training reads all of it: the frame data from `data/`, the camera video from `videos/`, the instructions from `meta/tasks.parquet`, and the normalization stats from `meta/stats.json`. Export it complete; upload it as-is.
`meta/info.json` is the contract. Its `features` map is what the intake gate checks:
```json
{
"codebase_version": "v3.0",
"robot_type": "so101",
"features": {
"observation.images.cam0": {"dtype": "video", "shape": [480, 640, 3], "names": ["height", "width", "channels"]},
"observation.images.cam1": {"dtype": "video", "shape": [480, 640, 3], "names": ["height", "width", "channels"]},
"observation.state": {"dtype": "float32", "shape": [6], "names": ["state"]},
"action": {"dtype": "float32", "shape": [6], "names": ["action"]}
}
}
```
Before it spends any GPU time, the intake gate reads `meta/info.json` and requires:
* `features` contains `action`, `observation.state`, and **at least one** `observation.images.*` camera.
* `action` and `observation.state` both have shape `[6]` — the SO-101 joint dimension.
* Cameras it can map to `cam0`/`cam1` — it derives the rename from the dataset; a non-standard name is remapped, not rejected.
A dataset that misses the first two fails the run at `gate: intake`, with a message naming what tripped, before any GPU is spent. The gate also reads `meta/tasks.parquet` and rejects a dataset whose every episode's task string is empty, whitespace, or the exporter's placeholder.
The **task strings must be real instructions**, not the Rerun exporter's literal `"task"` placeholder — a policy trained on the placeholder learns nothing. Export with a real instruction (`--task ` when you export) and the right normalization stats (`meta/stats.json` with quantiles); the New Theory intake gate refuses a taskless dataset outright, so this is enforced twice.
## The HTTP door [#the-http-door]
For a native integration, the sequence is four requests. Every request authenticates with the developer's key:
```
Authorization: Bearer nt_a1b2c3d4…
```
A real key is `nt_` followed by 40 characters. A missing or malformed key is `401`. All requests are against the console at `https://newtheory-console.vercel.app`.
### 1. Sign [#1-sign]
`POST /api/uploads/sign` mints one short-lived upload URL per file, in a single round-trip. Send the dataset name and the list of file paths, each **relative to the dataset directory**:
```
POST /api/uploads/sign
Authorization: Bearer nt_…
Content-Type: application/json
{
"dataset": "my_task",
"paths": [
"meta/info.json",
"meta/tasks.parquet",
"data/chunk-000/file-000.parquet",
"videos/chunk-000/observation.images.cam0/file-000.mp4"
]
}
```
Response:
```json
{
"namespace": "u_9f3c…",
"dataset": "my_task",
"count": 4,
"urls": [
{
"path": "meta/info.json",
"url": "https://storage.example/upload-url-issued-by-new-theory…",
"objectPath": "…",
"expiresAt": "2026-07-15T18:42:00.000Z"
}
]
}
```
Each entry pairs the `path` you sent with the `url` to `PUT` it to. `expiresAt` is when the URL stops working — minutes, not hours, so sign and upload in the same pass. `objectPath` is New Theory's internal storage path for the object; you don't need it — the top-level `namespace` is the durable handle for the upload.
Constraints:
* **At most 1000 paths per request.** More than that is `400 {"error": "bad_request", "max_batch_paths": 1000}` — split into multiple sign calls. The list is never silently truncated.
* Each path is a `/`-joined set of safe segments (letters, digits, `.`, `_`, `-`), up to 512 characters. No `..`, no leading `/`. A malformed path is `400 {"error": "bad_request"}`.
* The `dataset` name is one safe segment, up to 128 characters.
* The upload lands under the key owner's namespace, derived server-side. You cannot name it, and one developer's key can never write into another's.
* **The dataset name must not already exist in your namespace.** Uploads are create-only — signing against a name you've already staged (complete or partial) is `409 {"error": "dataset_name_exists", ...}`, and no URLs are minted. Re-upload under a new name, e.g. `my_task_v2`.
There is also a single-file shape — `{"dataset": "…", "path": "…"}` → `{"url", "objectPath", "expiresAt"}` — but the batch shape above is what you want for a dataset directory: one round-trip instead of one per file.
### 2. Upload [#2-upload]
`PUT` each file's bytes to its signed `url`. The content type is fixed:
```
PUT
Content-Type: application/octet-stream
```
**`Content-Type: application/octet-stream` is required.** It is bound into the URL's signature — a `PUT` with any other content type fails the signature check. No `Authorization` header goes on the `PUT`; the signed URL carries its own authorization. A `2xx` means the object landed. Bytes never pass through the console — they go straight to storage over the signed URL.
### 3. Verify [#3-verify]
`GET /api/uploads/list?dataset=` lists what landed under the developer's account, so you can confirm every file arrived before launching:
```
GET /api/uploads/list?dataset=my_task
Authorization: Bearer nt_…
```
```json
{
"namespace": "u_9f3c…",
"count": 4,
"objects": [
{
"objectPath": "…",
"path": "my_task/meta/info.json",
"size": 4360,
"updated": "2026-07-15T18:41:12.000Z"
}
]
}
```
Here `path` includes the dataset name (`my_task/meta/info.json`) — it is the object's path within the developer's namespace. Match `count` against the number of files you uploaded. The list is scoped to the key owner's namespace; a developer only ever sees their own uploads.
### 4. Launch [#4-launch]
`POST /api/finetune` launches training against the staged dataset name and returns a job handle. Training runs on New Theory's GPUs under the developer's key:
```
POST /api/finetune
Authorization: Bearer nt_…
Content-Type: application/json
{ "dataset": "my_task" }
```
```json
{
"job_handle": "",
"dataset": "my_task",
"uid": "",
"status": "launched"
}
```
Optional body fields:
| Field | Type | Meaning |
| ------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `steps` | integer, 2000–100000 | Total training steps. Out of range is `400` with a `detail` naming the bounds — never clamped. Omit for the server default. |
| `name` | slug `[a-z0-9-]`, 3–40 chars | Name for the model this run produces. A name already used under the account is `409` with a `detail`, and the run is **not** launched (no GPU spent). Omit to name the model after the dataset. |
### 5. Poll (optional) [#5-poll-optional]
`GET /api/finetune/status?job_handle=` reports the run's state. Poll it until `status` is terminal:
```
GET /api/finetune/status?job_handle=
Authorization: Bearer nt_…
```
```json
{ "status": "succeeded", "gate": null, "detail": null, "tag": "", "report_card": null, "model_status": "pending" }
```
`status` is `launched` / `running` while the run is going, then `succeeded` or `failed`. On `failed`, `gate` names the step that stopped it (`intake`, `train`, `frame-check`, …) and `detail` carries the pipeline's human-readable failure cause. `model_status` reports the trained model's admission state — `pending`, `probed`, `live`, or `dead`. A `succeeded` run is not servable until admission reaches `live`; the tag resolves for inference only then. Training runs for hours — poll on a slow interval (every 15 seconds is what the CLI uses).
### Errors [#errors]
| Status | Where | Meaning |
| ----------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `401` | any | Key missing, malformed, or rejected. |
| `400` | sign | Bad `dataset`/`path`, or over 1000 paths (`max_batch_paths` in the body). |
| `409` | sign | `dataset_name_exists` — a dataset already exists under this name in your namespace; no URLs are minted (see `detail`). Upload under a new name instead. |
| `400` | finetune | `dataset` name invalid, or `steps`/`name` out of bounds (see `detail`). |
| `409` | finetune | `name` already used under this account — the run was not launched (see `detail`). |
| signature failure | `PUT` | Content type wasn't `application/octet-stream`, or the URL expired. |
| `503` | sign / finetune | A New Theory service is briefly unavailable — retry. |
## End-to-end example [#end-to-end-example]
A complete native integration in Python, standard library plus `requests`. It uploads a dataset directory, verifies the upload, launches training, and polls to completion. Everything it needs is an `nt_` key and a folder path.
```python
import os
import sys
import time
from pathlib import Path
import requests
CONSOLE = "https://newtheory-console.vercel.app"
API_KEY = os.environ["NT_API_KEY"] # nt_ + 40 chars
DATASET = "my_task" # staged name (folder basename)
EXPORT_DIR = Path("./datasets/my_task") # LeRobot v3 dataset directory
AUTH = {"Authorization": f"Bearer {API_KEY}"}
def push_and_launch() -> str:
# 1. Collect every file, as paths relative to the dataset directory.
files = sorted(
p.relative_to(EXPORT_DIR).as_posix()
for p in EXPORT_DIR.rglob("*")
if p.is_file()
)
if not files:
sys.exit(f"no files under {EXPORT_DIR}")
# 2. Sign every file in one round-trip (up to 1000 per call).
signed = requests.post(
f"{CONSOLE}/api/uploads/sign",
headers={**AUTH, "Content-Type": "application/json"},
json={"dataset": DATASET, "paths": files},
)
signed.raise_for_status()
urls = {entry["path"]: entry["url"] for entry in signed.json()["urls"]}
# 3. PUT each file to its signed URL — content type MUST be octet-stream.
for rel in files:
with open(EXPORT_DIR / rel, "rb") as fh:
put = requests.put(
urls[rel],
data=fh.read(),
headers={"Content-Type": "application/octet-stream"},
)
put.raise_for_status()
print(f"uploaded {rel}")
# 4. Verify the whole set landed before launching.
listed = requests.get(
f"{CONSOLE}/api/uploads/list",
headers=AUTH,
params={"dataset": DATASET},
)
listed.raise_for_status()
landed = listed.json()["count"]
if landed < len(files):
sys.exit(f"only {landed}/{len(files)} files landed — not launching")
# 5. Launch training against the staged name.
launched = requests.post(
f"{CONSOLE}/api/finetune",
headers={**AUTH, "Content-Type": "application/json"},
json={"dataset": DATASET}, # add "steps"/"name" here to override defaults
)
launched.raise_for_status()
return launched.json()["job_handle"]
def watch(handle: str) -> None:
while True:
status = requests.get(
f"{CONSOLE}/api/finetune/status",
headers=AUTH,
params={"job_handle": handle},
).json()
state = status["status"]
if state == "succeeded":
model_status = status.get("model_status")
print(f"trained — model tag: {status.get('tag')} (admission: {model_status})")
if model_status != "live":
print("not servable yet — the tag resolves once admission reaches 'live'")
return
if state == "failed":
sys.exit(f"failed at gate: {status.get('gate')} — {status.get('detail')}")
print(f"… {state}")
time.sleep(15)
if __name__ == "__main__":
handle = push_and_launch()
print(f"launched: {handle}")
watch(handle)
```
## Where to go next [#where-to-go-next]
* [CLI reference](/docs/reference/cli) — every `newt finetune` flag, the `--json` output shape, and exit codes.
* [Rerun SO-101 hackathon kit](https://github.com/mission-robotics-ai/so100-hackathon) — the record, calibrate, and export reference the dataset comes from.
---
# Models
URL: /docs/models
Browse models available through the NT inference API — base models and their fine-tunes.
The NT inference API serves the MolmoAct-2 family. The base entry defines the wire contract for a robot class. Fine-tunes specialize behavior for a specific task; each inherits or declares its own contract.
## Contract quick reference [#contract-quick-reference]
Every model exposes a contract — the wire shapes the API enforces on observation and action frames. The server validates your first `obs` frame against the contract and closes with `4422` on mismatch.
| Field | What it means |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `state_shape` | Dimensions of the joint-state vector your `read_state()` callback must return |
| `state_dtype` | Expected dtype — `float32` for all current models |
| `cameras` | Nested object — `required` camera keys must be present in your `images` dict (missing one closes `4422`); `expected` camera keys zero-fill if missing |
| `image_shape` | Expected shape of each camera array, channels-first (`[C, H, W]`) |
| `action_shape` | Shape of the action chunk the server returns — `[horizon, action_dim]` |
| `tags` | Docker-style identifiers for the model — the string you pass to `Robot(model=...)` |
The full contract for any model is available from the always-on registry — no GPU warmup needed:
```bash
curl -H "Authorization: Bearer $NT_API_KEY" \
https://nt-registry-production.up.railway.app/v1/models/ft_6341c5_d13da9
```
Fine-tune entries return the resolved contract inherited from their base. See [Reference → Models](/docs/reference/models) for per-model contract documentation.
## Model families [#model-families]
| Family | Base UID | Description | Fine-tunes |
| ------------------------------------ | ------------------- | ---------------------------------------------------- | -------------------------------------- |
| [MolmoAct-2](/docs/models/molmoact2) | `ft_base_molmoact2` | Bimanual, not directly deployable — pick a fine-tune | [SO-101](/docs/models/molmoact2/so101) |
---
# CLI Reference
URL: /docs/reference/cli
Every command, flag, and environment variable in the newt CLI.
The `newt` CLI is the primary tool for authenticating, inspecting your API key, listing models, and installing the onboarding skill. It is a global command — installed once, used in any directory.
```
newt [command] [flags]
```
***
## Global flags [#global-flags]
| Flag | Description |
| ----------------- | --------------------------------------------------------------------------------------------------- |
| `-h`, `--help` | Print usage and exit. Zero network calls, zero writes. |
| `-V`, `--version` | Print `newt ` and exit. Also available as `newt version`. Zero network calls, zero writes. |
**Help guard:** `newt --help` and `newt -h` are safe to call anywhere — they make no network calls and write nothing to disk.
***
## Commands [#commands]
### `newt login` [#newt-login]
Authenticate with the New Theory console and save your API key.
```
newt login [--print]
```
**What it does:** Opens an interactive browser pairing flow. Prints a URL and an 8-character code formatted `XXXX-XXXX`, then polls the console until you complete the authorization. On success, writes your API key to `~/.nt/credentials`.
**First-line output:** `Starting authentication…`
The pairing URL and code appear on subsequent lines (\~line 6). The load-bearing fact (the code itself) is not on line 1 — this is a known constraint of the interactive auth flow.
**Flags:**
| Flag | Description |
| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--print` | Route auth output to stderr; print only the bare API key to stdout. Does **not** write `~/.nt/credentials`. Use to pipe the key into another tool or capture it in a script. |
**Exit codes:**
| Code | Condition |
| ---- | ------------------------------------------------ |
| 0 | Authentication succeeded |
| 1 | Network failure, pairing expired, or other error |
**Network:** `POST` to `NT_CONSOLE_URL/api/cli/auth/start`; polls `NT_CONSOLE_URL/api/cli/auth/poll` every 2 s for up to 10 minutes.
**Writes:** `~/.nt/credentials` (mode `0600`) on success — unless `--print` is set.
**Help guard:** `newt login --help` and `newt login -h` are safe — they print usage and exit 0 with zero network calls and zero writes. The guard shipped 2026-06-11 (commit 50004519, 44 tests).
***
### `newt logout` [#newt-logout]
Remove your API key from disk.
```
newt logout [--json]
```
**What it does:** Deletes `~/.nt/credentials` and removes `~/.nt/` if it is now empty. Safe to call when already logged out.
**First-line output (human):** `Logged out.` or `Already logged out — no credentials file found.`
**Flags:**
| Flag | Description |
| -------- | ---------------------------------------------------------------------------- |
| `--json` | Emit a JSON object instead of human-readable output. See output shape below. |
**`--json` output shape:**
```json
{
"action": "removed" | "already_logged_out",
"credentials_path": "/Users//.nt/credentials",
"env_var_warning": true | false
}
```
`env_var_warning` is `true` when `NT_API_KEY` is set in the environment — your key remains active even after logout.
**Exit codes:**
| Code | Condition |
| ---- | ------------------- |
| 0 | Always (idempotent) |
**Network:** None.
**Writes:** Deletes `~/.nt/credentials`; removes `~/.nt/` if empty.
**Help guard:** `newt logout --help` and `newt logout -h` are safe — they print usage and exit 0 with zero network calls and zero writes. The guard shipped 2026-06-11 (commit 50004519, 44 tests).
***
### `newt models` [#newt-models]
List models available to your API key.
```
newt models [--json] [--ids]
```
**What it does:** Calls the New Theory registry and prints the model families your key has access to. Human output leads with a `Key` line naming which credential produced the listing (masked — never the full key) and its source (environment or credentials file), then the catalog grouped by base with its fine-tunes indented underneath. Raw uids are left out by default — the tag is the identity you scan by; `--ids` brings them back alongside each name.
**First-line output (human):** `Key nt_•••••••• ()` — the masked key identity and where it came from (`environment` or `credentials file`). The model catalog follows below it. If your key has no models, a message appears on stdout.
**Example (default):**
```
Key nt_••••••••5bde16a3 (environment)
ft-64cad948c4e9f09c-red-cube-bowl [base: ft_base_so101_ft]
ft-64cad948c4e9f09c-svla-so101-pickplace-20260714213146 [base: ft_base_so101_ft]
ft-64cad948c4e9f09c-my-first-pickplace [base: ft_base_so101_ft]
molmoact2
so101
```
**Example (`--ids`):**
```
Key nt_••••••••5bde16a3 (environment)
ft-64cad948c4e9f09c-red-cube-bowl ft_64cad948c4e9f09c_31ac23c75952 [base: ft_base_so101_ft]
ft-64cad948c4e9f09c-svla-so101-pickplace-20260714213146 ft_64cad948c4e9f09c_e4519e2a61d5 [base: ft_base_so101_ft]
ft-64cad948c4e9f09c-my-first-pickplace ft_64cad948c4e9f09c_4527340bb316 [base: ft_base_so101_ft]
molmoact2 ft_base_molmoact2
so101 ft_6341c5_d13da9
```
**Flags:**
| Flag | Description |
| -------- | --------------------------------------------------- |
| `--json` | Emit the raw model array from the registry as JSON. |
| `--ids` | Show each model's uid alongside its name. |
**`--json` output shape:** An array of model objects. Shape is determined by the registry API.
**Exit codes:**
| Code | Condition |
| ---- | ----------------------------------------------------------- |
| 0 | Models listed successfully |
| 1 | No API key, authentication failure, or registry unreachable |
**Network:** Registry call via `newt.list_models(api_key)`.
**Writes:** None.
**Help guard:** `newt models --help` and `newt models -h` are safe — they print usage and exit 0 with zero network calls and zero writes. The guard shipped 2026-06-11 (commit 50004519, 44 tests).
***
### `newt status` [#newt-status]
Show your current authentication state and registry connectivity.
```
newt status [--json]
```
**What it does:** Reports your key source (environment variable or credentials file), any active URL overrides, and whether the registry is reachable. Prints an amber warning block if `NT_BOOTSTRAP_URL` or `NT_INFERENCE_URL` is set.
**First-line output (human):** `Key source: env_var` / `Key source: credentials_file` / `Key source: none`. Registry reachability appears on a subsequent line (\~line 4).
**Flags:**
| Flag | Description |
| -------- | -------------------------- |
| `--json` | Emit a JSON status object. |
**`--json` output shape:**
```json
{
"key_source": "env_var" | "credentials_file" | "none",
"overrides": {
"NT_BOOTSTRAP_URL": null | "",
"NT_INFERENCE_URL": null | ""
},
"registry_reachable": true | false,
"latency_ms": | null
}
```
**Exit codes:**
| Code | Condition |
| ---- | ----------------------------------------------------- |
| 0 | Status check completed without error |
| 1 | No key, bad key, registry unreachable, or other error |
**Network:** Registry call via `newt.list_models(api_key)`. Skipped when no key is present.
**Writes:** None.
**Help guard:** `newt status --help` and `newt status -h` are safe — they print usage and exit 0 with zero network calls and zero writes. The guard shipped 2026-06-11 (commit 50004519, 44 tests).
***
### `newt run` [#newt-run]
Run one real inference against your model and print what came back.
```
newt run [--snapshot ] [--prompt ] [--json]
```
**What it does:** Loads a bundled recorded observation and calls your model once against prod via `Robot(model=).infer(obs)` — no hardware involved. Prints the resolved model, the round-trip latency, and the action-chunk shape. **No robot is connected and nothing moves** — this is a live inference against your model, not a robot demo, and the output says so plainly.
**First-line output (human):** The resolved model name.
**Flags:**
| Flag | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--snapshot ` | Bundled observation to send. Options are `red_cube` (6-axis SO-101), `cup_stacking` (8-axis rig), and `pour_coffee_beans` (8-axis rig). Defaults to the snapshot matching your model's contract. |
| `--prompt ` | Override the snapshot's recorded prompt. |
| `--json` | Emit machine-readable JSON instead of human output. |
**`--json` output shape:**
```json
{
"tag": "",
"model": "",
"snapshot": "",
"prompt": "",
"latency_ms": ,
"total_ms": ,
"retries": ,
"action_chunk": {
"shape": [, ...],
"axes": ["", ...]
}
}
```
**Exit codes:**
| Code | Condition |
| ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0 | Inference completed and result printed |
| 1 | No model tag given, unknown `--snapshot`, no API key, or the call failed (authentication, model not found, model not deployable, registry unreachable, contract mismatch, server error, verifier unavailable, or protocol error) |
**Network:** One inference call against prod via `Robot(model=).infer(obs)`.
**Writes:** None.
**Help guard:** `newt run --help` and `newt run -h` are safe — they print usage and exit 0 with zero network calls and zero writes.
***
### `newt skill` [#newt-skill]
Manage Claude Code skills provided by New Theory.
```
newt skill [-h | --help]
```
**Subcommands:**
| Subcommand | Description |
| ---------- | ------------------------------------------------------------------------------------ |
| `install` | Install the `newt-onboarding` skill into `.claude/skills/` in the current directory. |
**Help guard:** `newt skill --help` and `newt skill -h` are safe — they print usage and exit 0 with zero network calls and zero writes.
**Exit codes:**
| Code | Condition |
| ---- | -------------------------------------- |
| 0 | `--help` flag; or subcommand succeeded |
| 1 | Unknown subcommand |
**Network:** None.
**Writes:** None (for the `skill` dispatcher itself; `install` writes — see below).
***
#### `newt skill install` [#newt-skill-install]
Install the `newt-onboarding` skill into the current directory.
```
newt skill install [--json]
```
**What it does:** Writes `.claude/skills/newt-onboarding/SKILL.md` into the current working directory, creating the directory if needed. If the file already exists, it is updated in-place.
**First-line output (human):** `Skill installed — ` or `Skill updated — `
**Flags:**
| Flag | Description |
| -------- | -------------------------- |
| `--json` | Emit a JSON result object. |
**`--json` output shape:**
```json
{"ok": true, "path": "/path/to/.claude/skills/newt-onboarding/SKILL.md", "overwrite": false}
```
On error:
```json
{"ok": false, "error": ""}
```
**Exit codes:**
| Code | Condition |
| ---- | ------------------------------------- |
| 0 | Skill written or updated successfully |
| 1 | Read or write error |
**Network:** None. The skill file is read from package resources, not fetched over the network.
**Writes:** `.claude/skills/newt-onboarding/SKILL.md` relative to the current working directory.
**Help guard:** `newt skill install --help` and `newt skill install -h` are safe — they print usage and exit 0 with zero network calls and zero writes. The guard shipped 2026-06-11 (commit 50004519, 44 tests).
***
### `newt record` [#newt-record]
Record NT episodes from a robot (or a simulated joint stream) — the keyboard capture frontend on `newt.recording.Session`.
```
newt record --task [--dest ] [--simulate | --source ] [--target ] [--hz ] [--json]
```
**What it does:** Reads keystrokes and drives a recording session. SPACE starts and stops an episode; at the stop, ENTER keeps it, `D` discards, `R` redoes. A live frame counter and joint readout print during capture, and Ctrl+H is the kill — it torques off and leaves no partial episode directory. Every decision about episode format, atomicity, and the kill lives in `newt.recording.Session`; this command is only the skin.
**Requires the recording extra:** `pip install "newt[recording]"`. Without it, the command stands down with the exact install line.
**Flags:**
| Flag | Description |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `--task ` | Language task prompt recorded in every episode. Required. |
| `--dest ` | Episode output directory (default: `./episodes`). |
| `--simulate` | Record from a fake joint stream — no hardware. |
| `--source ` | Load a developer `RecordingSource`, `MODULE:FACTORY` (e.g. `mypkg.rig:make_source`). Mutually exclusive with `--simulate`. |
| `--target ` | Stop after N kept episodes. |
| `--hz ` | State sample rate (default: 30). |
| `--author ` / `--license ` | Provenance written to each `episode.json`. |
| `--json` | Agent mode: line-delimited JSON events on stdout, line-delimited commands on stdin. |
**Exit codes:**
| Code | Condition |
| ---- | ---------------------------------------------------------------------------------- |
| 0 | Session ran and closed cleanly |
| 130 | Ctrl+H kill (no partial episode left behind) |
| 1 | Missing `--task`, no writable destination, or the recording extra is not installed |
**Network:** None. **Writes:** `episode_` directories under `--dest`.
**Non-TTY without `--json`:** stands down loudly — there is no keyboard to read.
***
### `newt finetune` [#newt-finetune]
Launch a training run on New Theory's GPUs with your API key, then watch it to completion.
```
newt finetune (--dataset | --handle | --list) [--status] [--steps ] [--name ] [--fresh] [--json]
```
**What it does:** Sends your dataset to the New Theory console, which launches the training job server-side on New Theory's GPUs and returns a job handle. No infrastructure credential ever reaches your machine. The CLI then polls the run until it reaches a terminal state, printing the resulting model tag and report-card pointer on success, or the pipeline gate that failed. `--handle` re-attaches to a run you already launched (poll only, no new launch); `--status` prints that run's current state once and exits; `--list` prints your recent runs.
**First-line output (human):** `Launched fine-tune on dataset ''.` followed by the job handle. When re-attaching, the first line is `Watching …`.
**Flags:**
| Flag | Description |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--dataset ` | The staged dataset to fine-tune on. Launches a new run. |
| `--handle ` | Re-attach to a run already launched, and poll it. |
| `--status` | With `--handle`, print the run's current state once and exit instead of watching. |
| `--list` | List your recent runs — handle, dataset, last recorded state, and when. |
| `--steps ` | Advanced. Total training steps for this run, used with `--dataset`. A whole number between 2000 and 100000. Optional; omit it for the default. |
| `--name ` | Advanced. Name for the model this run produces, used with `--dataset`. A slug — lowercase letters, digits, and hyphens, 3–40 characters. Optional; omit it to name the model after the dataset. |
| `--fresh` | Advanced. Ignore any existing checkpoint and retrain from scratch, used with `--dataset`. Optional; omit it to resume a completed run's finishing steps. |
| `--json` | Emit machine-readable JSON instead of human output. |
**`--steps` bounds:** A value outside `2000`–`100000` is rejected with an error naming the bounds, never clamped into range. The value is recorded on the run. Leave `--steps` off for the default step count.
**`--name` validation and collision:** A name is validated as a slug — lowercase letters, digits, and hyphens, 3–40 characters — before any network call; anything else is rejected with an error, never reshaped to fit. If a model of that name already exists under your account, the launch is refused with a 409 naming the tag, before any training starts and before any GPU time is spent. Omit `--name` to name the model after its dataset (the default).
**`--fresh` behavior:** By default a re-run resumes a completed run's finishing steps instead of retraining. `--fresh` forces the full retrain from scratch, ignoring any checkpoint. Either way, the launch output names which path ran — resume or forced retrain — so the choice is never silent.
**`--json` output shape (launch, `--handle`, `--status`):**
```json
{
"job_handle": "",
"status": "succeeded" | "failed",
"gate": null | "",
"tag": null | "",
"report_card": null | "",
"steps": null | ,
"name": null | ""
}
```
`steps` is the value set with `--steps`, or `null` when none was set (the server default is in force). `name` is the value set with `--name`, or `null` when none was set (the model is named after its dataset).
**`--list --json` output shape:**
```json
[
{
"job_handle": "",
"dataset": "",
"status": "",
"created_at": "",
"model_status": null | "pending" | "probed" | "live" | "dead"
}
]
```
`model_status` is each run's model lifecycle state: `null` before the model registers, `pending`/`probed` while New Theory verifies the weights load, `live` once they do, and `dead` if they don't. A `dead` model is recoverable and stays listed.
Each run also carries a stable model identity (`uid`), assigned at launch and returned by the finetune status and jobs APIs. Bring it to the support table if you need help with a specific run.
**Exit codes:**
| Code | Condition |
| ---- | -------------------------------------------------------------- |
| 0 | Run reached `succeeded` (or `--status`/`--list` completed) |
| 1 | No API key, launch rejected, run `failed`, or the poll gave up |
| 130 | You stopped watching with Ctrl-C (the run keeps going) |
**Network:** `POST /api/finetune` to launch, `GET /api/finetune/status` to poll a handle, and `GET /api/finetune/jobs` for `--list`. The console holds the training credentials; the CLI only ever sees a job handle.
**Writes:** None.
***
### `newt promote` [#newt-promote]
Keep a fine-tune's checkpoint band and serve it — the CLI twin of the console's promote button.
```
newt promote --band [--json]
```
**What it does:** Registers one of a run's evaluated checkpoint bands as a served model, over the same route the console's promote button calls. The model is born `pending`; New Theory's admission chain runs a safety check and takes it live, usually within a few minutes. The output names the new model and points at `newt models` to watch it. Handle-only: list your runs with `newt finetune --list`.
**First-line output (human):** `Promoted — your checkpoint is registered as a model.` followed by the model tag and its `pending` status.
**Flags:**
| Flag | Description |
| ---------------- | ----------------------------------------------------------------------------------------------------- |
| `` | The run whose checkpoint to serve (required, first argument). |
| `--band ` | Which checkpoint band to serve — the evaluated step, passed verbatim (e.g. `010000`). Required. |
| `--json` | Emit the route's JSON response body on stdout (the registered model, or the server's refusal detail). |
**Refusals:** Every server refusal prints its plain reason verbatim — a band whose eval hasn't completed, a checkpoint whose location the training pipeline hasn't reported yet, or a run that already has a registered model (with that model's identity). A refusal is never collapsed into a generic failure.
**Exit codes:**
| Code | Condition |
| ---- | ------------------------------------------------------------- |
| 0 | The band was registered (model born `pending`) |
| 1 | No API key, a bad argument, or the server refused the promote |
**Network:** `POST /api/finetune/runs//promote`. **Writes:** None.
***
### `newt episodes` [#newt-episodes]
Inspect recorded episodes, and pull a staged dataset back down.
```
newt episodes validate [--json]
newt episodes pull [--dest DIR] [--json]
```
**`validate`** calls `newt.recording.validate` on an `episode_` directory and renders the verdict — a `PASS`/`FAIL` line plus one line per invariant check. Frontend only: the checks themselves are the library's. Requires the recording extra: `pip install "newt[recording]"`.
**`pull`** downloads a staged dataset from your NT namespace. It fetches the owner-scoped download manifest (authed with your `nt_` key), then downloads each object **straight from storage over signed links** — the bytes go GCS → your machine, never through the console. Files land under `--dest` (default `./`), recreating the dataset's relative layout. Progress is reported by **files completed**, never a percentage. Reruns are **resumable**: a file already present at the manifest's size is skipped, so an interrupted pull resumes cleanly.
**Subcommands:**
| Subcommand | Description |
| ---------------- | ---------------------------------------------------------------- |
| `validate ` | Validate an `episode_` directory. |
| `pull ` | Download a staged dataset into `--dest` (default `./`). |
**Flags:**
| Flag | Description |
| -------------- | -------------------------------------------------------------------------------------------------------------- |
| `--dest ` | (`pull`) Where to write the dataset. Default: `./`. |
| `--json` | Emit machine-readable JSON (`validate`: the verdict; `pull`: `total_files`, `downloaded`, `skipped`, `bytes`). |
**Exit codes:**
| Code | Condition |
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0 | Episode is valid / the pull completed |
| 1 | A check failed, no argument given, an unknown subcommand, the recording extra is missing (`validate`), or the download failed / no key / dataset not found (`pull`) |
**Network:** `validate` none; `pull` reads the console manifest + downloads from storage. **Writes:** `validate` none; `pull` writes files under `--dest`.
**Help guard:** `newt episodes --help` and `newt episodes -h` print usage and exit 0.
***
### `newt upgrade` [#newt-upgrade]
```
newt upgrade [--print]
```
Upgrade the CLI to the latest version. When `newt` was installed as a `uv tool` (the documented install), this runs `uv tool upgrade newt` for you and streams its output. When the install method can't be confirmed, it prints the command instead of running it — it never guess-runs a package manager against the wrong environment.
**Flags:**
| Flag | Description |
| --------- | --------------------------------------------------- |
| `--print` | Print the upgrade command and exit — never runs it. |
**Passive update notice:** After any command succeeds, the CLI may print one quiet line to **stderr** when a newer version is available — `newt available — run 'newt upgrade'`. It runs at most once a day, never blocks or slows a command, is silent on any network failure, and is skipped entirely under `--json`. Disable it with `NEWT_NO_UPDATE_CHECK=1`.
**Exit codes:**
| Code | Condition |
| ---- | ------------------------------------------------------------------------------- |
| 0 | The upgrade ran and succeeded, or the command was printed (unconfirmed install) |
| 1 | The upgrade command failed, or `uv` was not found |
**Network:** `GET /api/cli/version` (public, to source the command). **Writes:** `~/.nt/update-check.json` (the once-a-day timestamp cache).
***
### `newt version` [#newt-version]
```
newt version
```
Prints `newt ` and exits. Identical to `newt --version` and `newt -V`.
| Exit code | Meaning |
| --------- | ------- |
| 0 | Always |
**Network:** None. **Writes:** None.
***
## Environment and credentials [#environment-and-credentials]
### API key resolution [#api-key-resolution]
The CLI and SDK resolve your API key in this order, stopping at the first hit:
1. `NT_API_KEY` environment variable
2. `~/.nt/credentials` file
`NT_API_KEY` always wins over the credentials file.
### `~/.nt/credentials` [#ntcredentials]
Written by `newt login`. Deleted by `newt logout`.
| Property | Value |
| -------- | -------------------- |
| Path | `~/.nt/credentials` |
| Mode | `0600` |
| Format | `api_key = nt_` |
### Environment variables [#environment-variables]
| Variable | Scope | Behavior | Default |
| ---------------------- | ------------- | ---------------------------------------------------------------------------- | ----------------------------------------------- |
| `NT_API_KEY` | CLI + SDK | API key; takes precedence over `~/.nt/credentials` | — |
| `NT_CONSOLE_URL` | `login` | Console URL for the pairing flow | `https://newtheory-console.vercel.app` |
| `NT_BOOTSTRAP_URL` | `status`, SDK | Override the registry discovery base URL | `https://nt-registry-production.up.railway.app` |
| `NT_INFERENCE_URL` | `status`, SDK | Override the inference endpoint; bypasses per-model routing via `/v1/models` | — |
| `NT_CONSOLE_URL` | `upgrade` | Console URL the version check + `newt upgrade` read | `https://newtheory-console.vercel.app` |
| `NEWT_NO_UPDATE_CHECK` | CLI | Set to `1` to disable the passive once-a-day "update available" notice | — |
**Override warnings:**
* **CLI (`newt status`):** An amber `Overrides active:` block is printed when `NT_BOOTSTRAP_URL` or `NT_INFERENCE_URL` is set, showing the active values.
* **SDK (`import newt`):** `EnvOverrideWarning` is emitted via `warnings.warn()` when `NT_INFERENCE_URL` is set. Warning text: `NT_INFERENCE_URL is set to — this overrides dynamic /v1/models discovery for ALL models. Unset it to use per-model cross-app routing.`
Override variables are intended for development and testing against local or staging infrastructure. Unset them before running production workloads.
***
## Related [#related]
* [Getting started](/docs/getting-started) — install `newt` and run your first inference
* [Authentication](/docs/authentication) — credentials, key rotation, and team access
---
# Two clocks — action rate vs inference rate
URL: /docs/reference/two-clocks
How a 15 Hz arm and a ~0.4 Hz cloud inference call coexist. Why "streaming" doesn't mean we're inference-streaming, and what that actually means for robotics control.
A common confusion: the API is on a streaming connection, so it sounds like we're streaming inference at the arm's control rate. We aren't. The arm and the model run on two different clocks, decoupled by the model's prediction horizon.
## The two clocks [#the-two-clocks]
* **Arm clock:** \~15 Hz — every 67 ms, the arm gets a new target pose and moves toward it.
* **Inference clock:** \~0.4 Hz — every \~2.5 s, the SDK sends an observation and the cloud returns a plan.
Inference is roughly **40× slower** than the arm's natural cadence. The two have to be reconciled somehow.
## How they're reconciled — the model emits plans, not actions [#how-theyre-reconciled--the-model-emits-plans-not-actions]
The model doesn't return one action. It returns a **plan**: 30 future poses, covering roughly 2 s of intended motion (30 actions at the arm's \~67 ms step). The arm plays that plan back at 15 Hz over the same \~2 s of wall time.
While the arm is mid-plan, the SDK kicks off the next inference call. By the time the arm has played the first \~20 actions of the current plan (\~1.3 s of motion), the next plan is arriving from the cloud. The SDK swaps to it; the arm starts playing actions from the new plan.
```
T=0.00 s send obs A ────────┐
T=0.00 s start playing plan_prev[0..19] (~1.3 s of motion)
T=1.33 s discard plan_prev[20..30]; we want fresh
T=2.50 s plan A arrives ─────┘
T=2.50 s start playing plan A[0..19]
T=3.83 s discard A[20..30]; send obs B
...
```
The arm moves smoothly the whole time. The model only "sees" the robot every \~2.5 s — not every 67 ms.
## What this means practically [#what-this-means-practically]
**The model has to plan, not react.** A single-action API would be useless here: the action returned would be 2.5 s stale by the time it executed. So the model emits forward-looking plans — predicting what to do over the next 1–2 seconds based on the latest observation it has.
**Reactive correction has a floor of \~1 second.** Between observation round-trips, the arm is executing the model's plan blindly. If something unexpected happens (the object slips, you bump the arm), the model can't notice until the next observation arrives. Hardware-level reflexes (collision detection, joint velocity limits) catch the dangerous cases; the model isn't fast enough.
**Plans need to be *good*.** Since the arm is flying blind for \~1 second between observations, the model can't emit drunk-walk trajectories that need constant correction. It has to predict competently far enough ahead that the arm tracks something useful even with stale conditioning.
**Receding horizon discards work to stay fresh.** The model gives the SDK 30 actions; the demo applies \~20 and throws away the rest. Why discard? Because by action 20, the observation that produced the plan is already 1.3 s old. Asking "what should I do *now* given where I am *now*" with a fresh observation beats trusting the older plan's tail.
## What "streaming" actually buys us here [#what-streaming-actually-buys-us-here]
It's not that we're streaming frames at the arm's rate. What we get from the streaming connection:
1. **One handshake per session, not per inference call.** Even at 0.4 Hz, handshake-per-request would add \~150 ms of TCP + TLS overhead to every cycle. The persistent connection amortizes that to one cost per `run()`.
2. **A bidirectional channel for out-of-band signals.** The model can push a `terminal` frame when the task is complete; the SDK can push a `stop` frame when the operator aborts. No polling, no separate endpoints.
3. **A session shape that matches the work.** One robot trial = one WebSocket = one `run()` call. The conversation has a defined start and end built into the protocol.
## The fundamental trade [#the-fundamental-trade]
We get the *responsiveness* of high-frequency arm control (15 Hz, smooth motion, tight joint velocity limits) **and** the *smartness* of slow cloud inference (a big vision-language-action model, language conditioning, generalization across tasks). The cost is the \~1 second of open-loop drift between observations.
Streaming doesn't eliminate that — nothing eliminates that until inference gets much faster. What streaming does is minimize the protocol overhead so the inference clock can run as fast as the model itself allows.
So the precise mental model: **the SDK plan-streams, not inference-streams.** The model emits a plan; the arm consumes that plan at its own rhythm; the SDK re-plans periodically using a fresh observation. The streaming connection is the conduit that makes that re-plan cycle as cheap as possible.
## Related [#related]
* [WebSocket vs HTTP](/docs/reference/websocket-vs-http) — why we picked a streaming connection at all
---
# WebSocket vs HTTP
URL: /docs/reference/websocket-vs-http
Why the shipped inference API is a WebSocket stream instead of HTTP POST, what each transport gives up, and what the wire shape says about the product.
A reference for the choice, not an argument for it. The shipped API is WebSocket. Both transports would work at our current call rate. The reason we picked WebSocket is the wire shape, not the latency.
## The two transports, at the wire [#the-two-transports-at-the-wire]
HTTP is request/response. The client opens a connection, sends one request, reads one response, and the conversation is over. Keep-alive can reuse the TCP and TLS state for the next request — modern clients do this by default — but each call still has its own beginning, middle, and end. The unit is the call.
WebSocket is a persistent bidirectional stream. The client opens one connection, and from then on either side can send framed messages until somebody closes it. The unit is the connection.
Both sit on top of TCP and TLS. TCP isn't a separate axis for this choice.
## Why the wire shape matters for NT [#why-the-wire-shape-matters-for-nt]
The case for WebSocket isn't latency. At our actual call rate it isn't even close — HTTP with keep-alive amortizes the handshake to near zero, and the per-call gateway cost is dominated by inference. The case is about what the wire shape says.
A world model is a bidirectional stream of observation and inference. Sensors flow in, intent flows back, the loop runs until the task ends. That is what the product is. The wire shape developers first encounter is the first thing that trains their mental model of what NT is — before the SDK, before the docs, before the first `print(chunk)`. A POST endpoint trains "inference call": you ask, you get one answer, you're done. A persistent stream trains "ongoing control loop": you join the conversation, it runs, you leave. The two framings build different products on top.
Every API surface developers respect today — Stripe, Anthropic, OpenAI, Linear, Resend, GitHub — is HTTP request/response. That's the right shape when the unit of work is a discrete call. The argument for WebSocket here is not "more familiar." It's that we're not a request/response API. We're a streaming control primitive, and the wire shape should say so.
The catch: a streaming primitive is heavier to call directly than a POST. The async-lifecycle ceremony you see in any raw WebSocket example — connect, send, recv, close codes, cancellation, reconnect — isn't a WebSocket quirk. It's what bidirectional streams look like in async languages, whether the substrate is WebSocket, gRPC bidi, HTTP/2 streams, or WebTransport. The complexity belongs to the primitive, not to any one transport. We either expose it to every caller or we wrap it.
We wrap it. The whole pitch is one line:
```python
robot.run("pick up the cup")
```
The SDK hides the streaming primitive behind that call. The developer never opens a socket, never writes an async function, never sees a close code. WebSocket isn't what they pay. The SDK is where we get to design well — and the wire shape is what lets the SDK's surface be "ask the model to do a task" instead of "make an inference call and then make another one and another one."
## What WebSocket gives you [#what-websocket-gives-you]
* **A streaming primitive on the wire.** Observation goes up, action chunk comes back, both directions stay open. The transport matches the abstraction the SDK exposes and the framing we use for the product.
* **Connection-as-session lifecycle.** One `run()` is one socket. Open is "start of trial," close is "end of trial." The protocol carries the session boundary; you don't reinvent it on top.
* **A surface for out-of-band signals.** The server can push a `terminal` frame when the run hits its stop condition (`max_duration`, `interrupted`); the client can push a `stop` frame to abort. No polling, no second endpoint.
* **One handshake per session, not per request.** At today's \~0.4 Hz call rate this is marginal; with HTTP keep-alive the difference shrinks further. At a future world-model rate (10–30 Hz on the wire) the gateway and TLS overhead per call compounds, and amortizing it across one connection starts to matter.
## What WebSocket costs you [#what-websocket-costs-you]
* **The SDK is mandatory.** There is no useful single-line invocation in shell or in another language. `websocat` confirms the handshake but won't drive inference, since it doesn't encode msgpack. Any non-Python client has to bring a WebSocket library, a msgpack library, and an async runtime.
* **A learning surface most HTTP APIs don't have.** Close codes (`4001`, `4400`, `4503`), frame types, reconnect policy, cancellation semantics. The SDK hides all of this; a developer writing a custom client sees it all at once.
* **Standard tooling fits awkwardly.** No `curl`, no Postman recording, no browser dev-tools timeline. Logging and observability stacks built around request/response need translation.
* **The QUIC-era successor is not "WebSocket over QUIC."** HTTP/3 swaps the transport under the same semantics — a config change, not a code change. WebSocket doesn't have an equivalent migration. The streaming primitive on QUIC is [WebTransport](https://developer.mozilla.org/en-US/docs/Web/API/WebTransport_API), which is a different API. This is a 12–24 month concern, not today's, but it's worth knowing the direction.
## What HTTP would give you [#what-http-would-give-you]
* **`curl` works.** Auth, request body in, response body out, in one shell line. So does Postman, so do browser dev tools, so do every fixture-replay harness and request-recorder in the ecosystem.
* **The SDK is optional.** Any language with a standard HTTP client and a msgpack library can call the API. No async runtime, no callback wiring, no WebSocket dependency.
* **The auth pattern matches every other API developers respect.** Stripe, Anthropic, OpenAI, Linear, Resend, GitHub — all bearer-token HTTP. The vocabulary is already in the developer's head.
* **Standard status codes.** `200`, `401`, `429`, `5xx` instead of WebSocket close codes. Every HTTP tool already speaks this surface.
* **HTTP/3 is a config flag.** Same semantics, faster transport. No rewrite, no API change.
## What HTTP would cost you [#what-http-would-cost-you]
* **Request/response is not a streaming primitive.** The wire shape trains "inference call." Once the world-model story is the pitch — and it is — the wire shape and the pitch say different things. Closing that gap takes copy, examples, and SDK design that the WebSocket shape just doesn't have to do.
* **Per-call overhead compounds at higher rates.** At \~0.4 Hz the per-call TCP, TLS, and gateway costs are dominated by the \~1.6 s inference; at 30 Hz they are not. The mitigations are real (keep-alive, HTTP/2 multiplexing, chunked responses, SSE) and each one adds back complexity on the day it ships. WebSocket pays the complexity once.
* **Out-of-band server-to-client signals need a separate shape.** A `terminal {stop_reason: "max_duration"}` from the model has no obvious home in stateless POST. The options are embed it in the action response or add SSE later. Neither is broken; both are extra design.
* **Latency is implementation-dependent across the board.** WebSocket adds \~100 ms in some L7 load-balancer and proxy stacks. On New Theory's serving infrastructure, WebSocket and HTTP share routing, so the gap closes. The transport doesn't determine the number — the deployment does. Worth knowing before you trust a benchmark you read on Twitter.
## What stays the same either way [#what-stays-the-same-either-way]
* **Wire encoding.** msgpack with the msgpack-numpy ndarray extension.
* **Observation schema.** `state`, `images`, `prompt`. Same fields, same shapes, same units, whichever transport carries them.
* **Action shape.** A `(horizon, action_dim)` float32 chunk, per the model's contract. Same row layout.
* **`stop_reason` vocabulary.** `max_duration`, `interrupted`. The surface that carries it shifts; the vocabulary holds.
* **Auth.** Bearer tokens against the same console key lifecycle.
* **Firehose coercion after the handshake.** Missing or wrong-shape fields in steady-state frames are coerced server-side and produce degraded actions, not 4xx responses. The first observation frame is contract-gated: a wrong-shape state or a missing required camera closes 4422 with an error envelope.
The choice is the transport. The contract doesn't move.
## What we picked and why [#what-we-picked-and-why]
We picked WebSocket. The reason isn't latency or handshake overhead — at our current call rate, HTTP with keep-alive holds up fine on both. The reason is that a streaming connection is the wire shape that matches what we're trying to be. The SDK hides the cost of that choice behind `robot.run("pick up the cup")`. The developer never opens a socket; they ask the model to do a task.
HTTP isn't broken here. It would work — and it would give you `curl` and a smaller dependency footprint in exchange. The honest version is that both shapes serve the API contract. The one we shipped is the one we want first contact with the product to look like.
## Related [#related]
* [Two clocks — action rate vs inference rate](/docs/reference/two-clocks) — why "streaming" here isn't "inference at the arm's rate," and what the persistent connection actually buys us once you account for slow cloud inference.
---
# How to use embodiment starters
URL: /docs/starters
Project structures you clone and own for a specific embodiment + model pairing. Build config, deps, wiring, entry script — boilerplate ready to run after you fill in your specifics.
A starter kit is a project you clone and modify, not a library you install and use. It ships the project structure — build config, deps, wiring, entry script — for one specific use case. The developer clones it, fills in their specifics, runs it. They own the result; the starter has no ongoing version relationship to their project after clone.
If you're setting up hardware for the first time, start with [Set up your embodiment](/docs/set-up-your-embodiment) — it walks the onboarding path fast. This page and the per-starter guides below are the deeper reference for what each kit contains.
Each starter targets the Linux workstation that drives the robot. macOS install paths are documented per-starter for developers scaffolding on a laptop before moving to the deploy machine.
## Why starter kits [#why-starter-kits]
Some integrations need a project structure, not just a function call. Documenting "here's how to set up your project" in prose drifts and breaks. Shipping the same instructions as a working project — version-controlled, executable, copy-pasteable — is reliable.
Each starter encodes one opinionated path through the problem. It's not a framework with options; it's "here's how we'd do it." If you want a different shape later, throw the project away and start over from a different starter.
## Available starter kits [#available-starter-kits]
Each row pairs a robot (the embodiment) with the model that drives it. A model name like MolmoAct-2 SO-101 identifies a specific checkpoint; the starter's docs explain what the model expects and produces.
| Starter | Embodiment | Model | Status |
| ------------------------------ | ----------------------------------- | ----------------- | --------- |
| [SO-101](/docs/starters/so101) | Hugging Face SO-101 arm + 2 cameras | MolmoAct-2 SO-101 | Available |
More embodiments will land as we add them. The pattern is consistent: clone the repo, edit `~/.config/nt/nt.toml` with your hardware specifics, run the entry script.
---
# SO-101 starter
URL: /docs/starters/so101
What's in the newt-starter-so101 repo — the project you clone to run MolmoAct-2 SO-101 inference on a Hugging Face SO-101 arm.
The SO-101 starter is a project you clone to run [MolmoAct-2 SO-101](/docs/models/molmoact2/so101) inference on a Hugging Face SO-101 arm with two cameras. The model produces `(30, 6)` action chunks representing 30 target joint configurations across 6 degrees of freedom; the starter supplies the `SO101` embodiment class that reads the arm's joint state and applies those chunks.
Repository: [`new-theory-research/newt-starter-so101`](https://github.com/new-theory-research/newt-starter-so101).
Not to be confused with the [Hackathon Starter Kit](https://github.com/mission-robotics-ai/so100-hackathon), the recording/teleop kit used at The Embodied Metal Hackathon — this page is the SO-101 deploy client.
```bash
git clone git@github.com:new-theory-research/newt-starter-so101.git
```
## What's in it [#whats-in-it]
The repo is small and flat. The pieces you interact with:
| File | What it is |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `embodiment.py` | The `SO101` class — the embodiment you own. Implements `read_state()` and `execute()` for the SO-101 arm and cameras; `SO101.from_config()` loads hardware addresses from `~/.config/nt/nt.toml`. Rename it, edit the wiring, add your own logic. |
| `run.py` | The thin entry script. Constructs `SO101.from_config()`, passes it to `newt.Robot(embodiment=rig, model="so101")`, and runs one closed-loop trial. Accepts `--task`, `--list-tasks`, `--check`, `--reset`, and `--arm` flags. |
| `conf/nt.toml.example` | The config template. You copy it to `~/.config/nt/nt.toml` and fill in your hardware specifics. |
| `pyproject.toml` | Dependencies, resolved with `uv`. Pulls in `newt` and `lerobot`. Install `--extra hardware` on the rig machine to add the Feetech servo driver (`lerobot[feetech]`). |
Setup uses `uv`, not pip. The canonical first command is:
```bash
uv sync
```
This resolves the `newt` dependency. `newt-python` is public — no collaborator access needed. If `uv sync` fetches it over a pinned `git+ssh://` URL, you still need an SSH key registered with GitHub; see [SSH-only machines](/docs/install#ssh-only-machines).
## Callbacks [#callbacks]
`SO101.read_state()` reads one observation frame from the hardware. It queries the arm's 6 Feetech servos for their current joint positions via lerobot's `SO101Follower` driver, and captures RGB frames from the `top` and `side` cameras, resized to 224×224. It returns a dict with `state` (6D joint positions in normalized lerobot units) and `images` (one entry per camera present).
The model was trained with `top` and `side` as camera names. If one camera is missing at runtime, the model server zero-fills it with `DegradationWarning`; quality degrades but the run does not close. Both cameras are optional from the server's perspective — present both for best results.
`SO101.execute(chunk)` applies one `(30, 6)` action chunk to the arm. For each of the first `MAX_ACTIONS_PER_CHUNK` rows (default 15), it sends the target joint positions to the servos via `sync_write`. The first chunk uses a 1.5-second settle sleep before streaming remaining actions. The servos travel at their own speed — there is no goal-time interpolation.
**Gripper (axis 6).** The gripper axis sign convention is unverified on physical hardware. Synthetic observations suggest negative-signed gripper commands, but physical open/close semantics have not been confirmed with a real arm. Verify the sign direction against your arm before deploying; the starter README carries the same caveat.
Both methods live in `embodiment.py`. If you wire a different arm or camera configuration, replace those bodies — the `from_config()` constructor and config-loading helpers can stay.
### What you fill in [#what-you-fill-in]
Run the setup command after `uv sync`:
```bash
uv run python3 run.py setup
```
This writes `~/.config/nt/nt.toml` from the template, auto-detects your serial port if exactly one SO-101 candidate is present, and reports what still needs your attention (camera indices, arm id). It accepts `--force`, `--report-only`, `--non-interactive`, and `--json` flags.
**Manual fallback.** If you prefer to configure by hand, copy the template and edit it directly:
```bash
mkdir -p ~/.config/nt
cp conf/nt.toml.example ~/.config/nt/nt.toml
```
Then fill in:
* `[[robot_config.arms]]` — `id` is the lerobot robot ID for your calibrated arm (it also names the calibration file), and `port` is the serial port for your SO-101 arm (e.g., `/dev/ttyACM0` on Linux, `/dev/tty.usbmodem*` on macOS). Calibration files are stored at `~/.cache/huggingface/lerobot/calibration/robots/so101_follower/.json` (override with `HF_LEROBOT_CALIBRATION`).
* `[[camera_config.cameras]]` — one entry each for `id = "top"` and `id = "side"`, with `index_or_path` (integer camera index, or a device path like `/dev/video0`), plus `width`, `height`, and `fps`. `uv run python3 run.py --check` reports the arm and cameras your config declares — it does not probe hardware.
You also export your `NT_API_KEY`. The entry script checks for it at launch and exits immediately if it's missing.
**Camera placement.** The camera names (`top`, `side`) match the labels used during training. Their physical mounting positions affect model performance: cameras mounted at angles that differ significantly from training geometry will push observations out-of-distribution. Match mounting geometry to how the training data was collected.
**Calibration.** The SO-101 is not factory-calibrated. Run lerobot's calibration flow before the first run:
```bash
lerobot-calibrate --robot.type=so101_follower --robot.port= --robot.id=
```
This generates the calibration file the starter reads via `from_config()`.
### Running without the rig [#running-without-the-rig]
You can run `uv run python3 run.py --check` on a Mac or any machine before the arm is connected. Without the `[hardware]` extra installed, the check validates your config file and API key against the model server without touching servo hardware. It exits 0 and names the arm and cameras it would use, or explains what's missing.
The actual closed-loop trial requires the arm connected and calibrated — there is no mock run mode.
## Tasks [#tasks]
The SO-101 model is open-vocabulary and language-conditioned. `--task` is free-form text, not a fixed key:
```bash
uv run python3 run.py --task "pick up the block and place it in the bin"
```
To see example task descriptions:
```bash
uv run python3 run.py --list-tasks
```
This prints clearly-illustrative example prompts and notes what the model was trained on. There is no fixed task-key registry — any reasonable natural-language task description is valid input.
## Agent-driven rigs [#agent-driven-rigs]
If you're running `run.py` from an agent harness rather than an interactive terminal:
**SSH install approval.** Agent harnesses may require human approval for `git+ssh://` dependency installs. This is expected — approve the `newt` and `lerobot` install steps and continue.
**Cleanup in non-TTY environments.** Ctrl+H triggers `emergency_home()` — moves the arm to its rest pose, then disconnects — but it's a stdin listener that only arms when stdin is a TTY, so it's unavailable in a non-TTY agent session. SIGINT/Ctrl+C triggers the weaker path, `teardown()`: it disconnects without homing, so the arm goes limp wherever it stopped. In a non-TTY agent session, Ctrl+C may not reach the process — use `timeout --signal=INT` to send SIGINT explicitly:
```bash
timeout --signal=INT 60 uv run python3 run.py --task "pick up the block and place it in the bin"
```
This disconnects the arm after 60 seconds; it does not home it first.
## When to use it [#when-to-use-it]
Use this starter when you have an SO-101 arm and want [MolmoAct-2 SO-101](/docs/models/molmoact2/so101) driving it. The starter runs open-vocabulary tasks — you supply a natural-language description with `--task`. `uv run python3 run.py --reset` moves the arm to a safe rest pose without running inference.
Requires Python 3.11–3.13. Python ≥ 3.14 is not supported; it hits a torch compatibility wall.
## Setup [#setup]
For the wire contract — state shape, action shape, cameras, and caveats — see the [SO-101 model page](/docs/models/molmoact2/so101). The repo's own README is the other reference, covering the full install sequence, calibration, camera index enumeration, and troubleshooting.
---
# MolmoAct-2
URL: /docs/models/molmoact2
MolmoAct-2 lineage hub — bimanual manipulation base family from Ai2. Not directly deployable; pick a fine-tune below.
MolmoAct-2 is Ai2's vision-language-action model trained on diverse bimanual manipulation data. The MolmoAct-2 base is **not directly deployable** — Ai2 designed it as a lineage anchor from which per-embodiment fine-tunes are derived. Each fine-tune carries its own contract (state shape, camera layout, action horizon) suited to its hardware. \[source: [https://huggingface.co/allenai/MolmoAct2](https://huggingface.co/allenai/MolmoAct2)]
Pass the fine-tune's tag to `Robot(model=...)`. Passing `"molmoact2"` (the base tag) raises `BaseNotDeployableError` with the list of available fine-tunes.
```python
import newt
# This raises BaseNotDeployableError — use a fine-tune tag instead.
# robot = newt.Robot(model="molmoact2")
# Use the SO-101 fine-tune:
robot = newt.Robot(model="so101", ...)
```
## Fine-tunes [#fine-tunes]
| UID | Tag | Embodiment | Status |
| -------------------------------------------------- | ------- | ------------------------------ | ------ |
| [`ft_6341c5_d13da9`](/docs/models/molmoact2/so101) | `so101` | SO-101 single-arm (6D, 2 cams) | active |
---
# SO-101
URL: /docs/models/molmoact2/so101
Fine-tune of MolmoAct-2 for the SO-101 single-arm embodiment. 6D joint state, two cameras, 30-step action chunks. UID ft_6341c5_d13da9.
fine-tune
so101
`ft_6341c5_d13da9` — MolmoAct-2 fine-tuned on the SO-101 single-arm embodiment. 6D joint state, two RGB cameras in any order, 30-step action chunks. Norm tag `so100_so101_molmoact2`. \[source: [https://huggingface.co/allenai/MolmoAct2-SO100\_101](https://huggingface.co/allenai/MolmoAct2-SO100_101)]
## Quickstart [#quickstart]
```python
import os
import newt
import numpy as np
def read_state():
return {
"state": np.zeros(6, dtype=np.float32),
"images": {
"top": np.zeros((3, 224, 224), dtype=np.uint8),
"side": np.zeros((3, 224, 224), dtype=np.uint8),
},
}
def execute(chunk):
for joints in chunk:
pass # send 6D joint targets to your controller
robot = newt.Robot(
api_key=os.environ["NT_API_KEY"],
model="so101",
read_state=read_state,
execute=execute,
)
result = robot.run("place the block in the bin", max_duration=30.0)
```
Replace `np.zeros(...)` with your actual sensor reads. `state` is 6 floats (single-arm joint positions); each image is channels-first uint8 at 224×224. Neither camera key is individually required — a missing key zero-fills with a `DegradationWarning`, and camera order is not fixed. The [SO-101 starter](https://github.com/new-theory-research/newt-starter-so101) ships both callbacks in `embodiment.py` already wired to real lerobot `SO101Follower` hardware — what you fill in is `nt.toml` (serial port, camera indices), not the callback bodies.
## Inputs [#inputs]
`read_state` must return a dict with these keys:
| Key | Shape | Dtype | Notes |
| ------------- | --------------- | --------- | --------------------------------------------------------------------------- |
| `state` | `(6,)` | `float32` | 6D single-arm joint state. |
| `images.top` | `(3, 224, 224)` | `uint8` | Top-view camera, CHW, RGB. Missing key zero-fills with DegradationWarning. |
| `images.side` | `(3, 224, 224)` | `uint8` | Side-view camera, CHW, RGB. Missing key zero-fills with DegradationWarning. |
The server validates the `state` shape and image dimensions on the first observation frame. A `state` shape mismatch closes with code `4422`. Camera mismatches are permissive — neither key is hard-required; see [Camera roles](#camera-roles) below. See the [API reference](/docs/api) for error envelope details.
## Outputs [#outputs]
`execute` receives a chunk of shape `(30, 6)` — a 30-step action horizon, 6 action dims per step. Each row is a target joint configuration in the same 6D format as `state`.
The 6 action dims correspond to `["shoulder_pan", "shoulder_lift", "elbow_flex", "wrist_flex", "wrist_roll", "gripper"]` in that order (see [State layout](#state-layout)).
Clients typically execute a prefix of the chunk, then read fresh state and send the next observation. See the [API reference](/docs/api#what-you-get-back) for the receding-horizon pattern.
## Provenance [#provenance]
| Field | Value |
| ------------------- | -------------------------------- |
| Checkpoint UID | `ft_6341c5_d13da9` |
| Base | MolmoAct-2 (`ft_base_molmoact2`) |
| Upstream checkpoint | `allenai/MolmoAct2-SO100_101` |
| Norm tag | `so100_so101_molmoact2` |
| Precision | bf16 |
## State layout [#state-layout]
The 6D `state` vector maps to the SO-101's joint axes, gripper-last:
| Index | Name | Units |
| ----- | --------------- | ---------- |
| 0 | `shoulder_pan` | Radians |
| 1 | `shoulder_lift` | Radians |
| 2 | `elbow_flex` | Radians |
| 3 | `wrist_flex` | Radians |
| 4 | `wrist_roll` | Radians |
| 5 | `gripper` | Normalized |
Send these in the same layout to `read_state`. The SO-101 is not factory-calibrated — run lerobot's calibration flow before the first run.
## Camera roles [#camera-roles]
The two expected cameras serve fixed viewing angles in the training distribution:
| Camera key | Mount | What it sees |
| ---------- | ---------------- | ----------------------- |
| `top` | Overhead | Workspace from above |
| `side` | Fixed side mount | Workspace from the side |
Camera order is not fixed; the server does not consume them positionally. A missing key zero-fills with a `DegradationWarning` rather than closing with `4422`.
## Notes [#notes]
* **`max_action_horizon` (30) vs `flow_matching_num_steps` (10)** — `max_action_horizon` is the action chunk length you receive (30 steps); `flow_matching_num_steps` is the internal diffusion solver step count (10). HuggingFace inference examples pass `num_steps=10` referring to the latter; that value does not describe the action chunk shape.
* **Gripper axis (axis 6):** negative-signed on synthetic observations. Physical open/close semantics are unverified without hardware — the sign convention may need to be flipped on a real SO-101. Verify against your arm before deploying.
* **Eval-parity:** task-success rate against the Allen AI baseline has not been benchmarked with physical hardware. A success-rate gap should be treated as a potential ingestion issue rather than a model quality ceiling until verified on a real arm.
* To add a customer fine-tune on top of this checkpoint, declare a registry entry with `base: ft_6341c5_d13da9` — the contract is inherited automatically via recursive resolution.
## References [#references]
* Model card: [allenai/MolmoAct2-SO100\_101](https://huggingface.co/allenai/MolmoAct2-SO100_101)
* Paper: [arXiv:2605.02881](https://arxiv.org/abs/2605.02881)
* Upstream code: [github.com/allenai/molmoact2](https://github.com/allenai/molmoact2)
---
# Models
URL: /docs/reference/models
Per-model wire contracts — the shapes the NT inference API enforces for each base model.
The NT inference API supports multiple base models. Each base has a different input shape — different state dimensions, different camera names, different image resolution. These per-model shapes are the contract.
When you call `Robot(model="so101").run(...)`, the API resolves the model, reads its contract, and validates your first observation frame against it. If the shapes don't match, the connection closes with code `4422` before any inference runs.
## What a contract is [#what-a-contract-is]
A contract declares five things about a model's wire interface:
| Field | Type | What it means |
| -------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `state_shape` | list of int | Expected dimensions of the joint-state vector your `read_state()` callback returns |
| `state_dtype` | str | Expected dtype (`"float32"`) |
| `cameras` | object | `{required: [...], expected: [...]}` — `required` camera keys must be present in your `images` dict (missing one closes `4422`); `expected` camera keys zero-fill if missing |
| `image_shape` | list of int | Expected shape of each camera array, CHW (`[C, H, W]`) |
| `action_shape` | list of int | Shape of each action chunk the server returns (`[horizon, action_dim]`) |
The first four fields are checked on your first observation frame. `action_shape` is informational — the server outputs it, not you.
## How to read the contract for your model [#how-to-read-the-contract-for-your-model]
```bash
curl -H "Authorization: Bearer $NT_API_KEY" \
https://nt-registry-production.up.railway.app/v1/models/ft_6341c5_d13da9
```
The response includes the full contract block. Fine-tune entries return the resolved contract inherited from their base. The registry is an always-on service — the curl returns in about a second without waking the GPU container.
## Coercion within a contract [#coercion-within-a-contract]
The API coerces missing data within a contract, but not for every camera. If your `images` dict is missing a camera listed in `cameras.expected` (and not in `cameras.required`), the server fills that slot with zeros and the connection stays alive. A missing camera listed in `cameras.required` closes `4422` — `contract_mismatch.camera_missing` — instead.
Coercion does not apply across contracts. An `[8]`-dimensional state vector sent to a model expecting `[14]` closes `4422`, not coerced.
## Choosing a model [#choosing-a-model]
Pick the model whose embodiment matches your hardware.
* [MA2-SO-101](/docs/models/molmoact2/so101) — SO-101 single-arm, 6D state, 2 cameras