structured-psi0-latent-diffusion-dynamics

Public research repo (github.com/JacobFV/structured-psi0-latent-diffusion-dynamics), shared openly. Formerly relational-robot-policy (renamed 2026-09-25, D-030). The Python package and CLI are still rrp; the working checkout on the GB10 hosts is still ~/work/relational-robot-policy.

Research system for morphology-general robot control with a structured, semantic latent packet: a ψ₀-inspired stack in which a planner ("system i") emits a continuous latent packet z that carries the task's meaning (which object, which manipulator, what relation should change, relative geometry, contact, uncertainty), and a fast embodiment-specific controller ("system 0") realizes that same z online from the robot's own proprioception and touch. Simulation is native MuJoCo on two NVIDIA GB10 machines.

Status (2026-09-30): research code, not a product. The code for the pre-training campaign is merged (readiness round 2, 21 units; ledger in STATUS.md, plan in research/readiness.md); the campaign-opening runs T0 to T9 are planned and none has started. Results so far (through D-145): on deployable routes (no teacher, oracle or BC at run time) the latent route is competent on legged go2, hexapod6 and the t1 humanoid, and partly competent on arms (below plain BC); it does not transfer to a new arm (D-135). Legged results use contact model v1 unless a v2 tracker is named (D-093). The dual arm is parked (collect, pack and evals on an external checkpoint stay; no training, D-146). Progress states use one vocabulary everywhere: planned, implementing, test_failed, verified, running, completed, failed_hypothesis, blocked_external, budget_exhausted (rrp.core.runs.RunState). What is and is not shown: research/reports/evidence_matrix.md. Plan and workstreams: docs/strategy.md. Problem checklist for physically credible training: docs/robot_training_considerations.md. Running work: STATUS.md.

---

architecture

system ii (optional VLM) + supplied task graph (events, roles, dependencies)
        │  public observations, task/event structure, morphology
        ▼
system i   typed context (morphology · scene · task/events · interaction) → rectified-flow sampling of z
        │
        ▼
latent packet z[knots, assemblies, 64]   ← the SAME tensor is (a) supervised for semantics (probes),
        │                                   (b) transmitted, (c) consumed by system 0
        ▼
system 0   z + morphology + current joint state + touch/IMU + elapsed phase → native joint targets
        ▼
joint-target tracking in MuJoCo

• Semantics live on the packet. A probe reads z with opaque handles only (never scene features or labels). Evidence of meaning comes from probes and causal edits of the received packet.

• System 0 never sees the task. Task meaning arrives only through z; sensor feedback arrives every tick.

• Clocks differ by body family:

  family	system-i replan	system 0	physics
  arm (single, dual)	0.4 s, 4 knots × 0.2 s	20 Hz (50 ms)	500 Hz
  legged / humanoid	0.4 s (every 20 ticks), 0.8 s packet horizon	50 Hz (20 ms)	body-specific substeps

• Semantic vs no-semantic ("nosem") variants are capacity-matched: nosem sets the semantic loss weights to 0. The bounded-NLL fix (latent.probe_lv_min: -4, D-085) is part of the current sem recipe ("semfix"/"fixsem").

• Design contract: research/corrections/controller-facing-semantic-latent.md. Original assignment (historical): .old/docs/handoff/.

one repo, three interfaces (D-140)

Every approach is a policy and every simulator an environment behind one interface each; a task says which environments it exists in and how an episode is judged; one harness (rrp.harness.rollout) runs any policy × env × task that negotiate() accepts and records declined pairs with their reasons. Design, interfaces and the moved-path table: docs/architecture.md.

Lineage codes in config and run names (sfjf, nsjf2, fixsem, gendag3_noqd, …) are decoded in .old/research/naming.md. The training lineages (arm ladder, legged, dual arm) run as pipeline stages (rrp.harness.pipelines, families arm / dual / legged) from recipe files in recipes/ (templates + per-track instances); evaluation routes on matched seeds are R0 teacher, R1 oracle packet E(expert chunk) (DIAGNOSTIC), R2 generated packet (deployable), plus a BC positive control and task-context edit suites.

entry points

rrp = PYTHONPATH=src .venv/bin/python -m rrp.cli; wrap anything heavy in rrp ops run (see resources).

quickstart

Requirements: Linux aarch64 or x86_64, Python 3.12, uv, a systemd user session (the broker uses user cgroups), Node 20+ only for the room.

git clone https://github.com/JacobFV/structured-psi0-latent-diffusion-dynamics.git && cd structured-psi0-latent-diffusion-dynamics
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -e '.[sim,ml,dev]'   # CPU torch is fine for tests
PYTHONPATH=src:. .venv/bin/python -m pytest tests/unit -q -m 'not slow'   # merge gate; tests needing weights / Menagerie / warp skip naming what is missing
ops/bin/fetch_menagerie.sh                                             # pinned third-party robot assets (~1.7 GB)

# one-time: measure free capacity and create the enforced project slice + watchdog
PYTHONPATH=src python3 -m rrp.cli ops init --role host
PYTHONPATH=src python3 -m rrp.cli ops start-watchdog
PYTHONPATH=src python3 -m rrp.cli ops run --cpu 2 --mem 4G --label my_job -- <command>
PYTHONPATH=src python3 -m rrp.cli ops stop --owned-only --lease <lease_id>   # never stop other people's jobs

Datasets, packed data and checkpoints are not in git; they live on the peer store and a host mirror (~/work/rrp-data).

repository map

src/rrp/          layers import only downward (tests/unit/test_layering.py)
  core/           contracts: observation vs privileged truth, NativeCommand / ActionChunk, latent packet + system-0
                  protocol, RobotSpec (morphology graph), TaskDefinition, provenance + source labels, RunConfig, paths
  ops/            resource broker, leases, cgroup enforcement, watchdog, telemetry, peer discovery
  bodies/         procedural + Menagerie arms, grippers, aloha, legged, humanoids, G1 (g1_simple), IK, physics versions
  tasks/          task-graph compiler/runtime/receipts, TaskSpec registry (spec.py)
  envs/           base.py (Env, EnvSpec, capabilities, make_env); mujoco/ (sessions, scenes, sensors, trackers);
                  warp/ (batched GPU legged envs); simple/ (Ψ₀ SIMPLE)
  policies/       base.py (Policy, negotiate, make_policy); features/ (featurizers: the only definition of what a
                  policy may see); nets/; bc, latent, system0, legged, oracle, packets; teachers/ (scripted_teacher);
                  psi0/
  harness/        rollout.py (the one episode loop), hooks.py (rollout hooks: HOOKS, TASK_HOOKS), eval/ (evaluate.py: evaluate / matrix; suites), train/,
                  data/, pipelines/, dag.py (run-dag)
  viz/            room exporter, recorder/replay, replay-spec generator
  cli/            the `rrp` command (tools.py: `rrp <group> <tool>` for data / train / suite / stage / viz tools)
viz/room/         visualization room (exporter output in viz/data)
recipes/          recipe templates + presets + per-track instances (the run DAGs; `rrp run-dag`)
ops/bin/          peer transport (peer_run/sync/bootstrap), asset fetch, external-env setup (shell)
research/         decisions.md (append-only; appendix P = former psi1z), registry.jsonl, tracks/<open track>.md, methods/,
                  corrections/, splits/, reports/evidence_matrix.md
artifacts/        small raw results (JSON/JSONL), receipts, labelled videos
docs/             architecture.md (interfaces; section 13 = the repository schema), strategy.md, relations.md,
                  experiments_roadmap.md, robot_training_considerations.md
.old/             retired configs, DAGs, scripts, notes, handoff package, demo page (index: .old/README.md; never read by code)
schema.toml       the repository schema (every tracked path must match; tests/unit/test_layout.py)
tests/            unit/ (incl. test_golden.py: byte-identity of featurizers, teachers, physics, policies), integration/, gpu/

rules that shape the results

• Public vs privileged. Policies see only PolicyObservation. Simulator truth is used only as training labels, rewards and evaluation.
• Labels. scripted_teacher (privileged) / oracle (teacher- or expert-encoded packet: diagnostic, not deployable) / bc / learned:. Every number comes from a saved raw output; failures are kept.
• Splits are sealed before results (research/splits/, recipes/presets/eval-latent_slice1.json).
• Resources. Host: ≤80% of currently free CPU and memory, host GPU allowed, ≥100 GB disk kept free; peer: all of it (D-026, D-033, D-086). Details and the peer workflow: AGENTS.md. No cloud spend, no physical robots.

development conventions

• Branch: main is the integration branch; parallel agents work in track/<track> worktrees and rebase onto main (research/tracks/BRIEF.md). Commit small, push often.
• Testing: light at this research stage (policy in AGENTS.md): test loss/likelihood math, leakage, resource safety and split leakage; otherwise prefer a short real run. tests/unit is green in a fresh checkout; tests that need untracked weights, assets, warp or the peer skip naming what is missing, and tests over 20 s carry @pytest.mark.slow (the merge gate is -m "not slow").
• Decisions: anything that changes a plan, gate or interpretation goes into research/decisions.md with evidence.

where to look next

• current state and workstreams → STATUS.md, docs/strategy.md
• what is shown → research/reports/evidence_matrix.md
• why → research/decisions.md; names → .old/research/naming.md
• agent rules → AGENTS.md

the Ψ₀ line (W10)

Ψ₀ direct and Ψ₀ + structure on Ψ₀'s SIMPLE benchmark are policies (rrp.policies.psi0) on the simple env (rrp.envs.simple, Isaac Sim 5.1 in its own venv: ops/bin/psi0_ext.sh, extra rrp[psi0]). The separate psi1z repo is retired (D-140); its decisions are appendix P of research/decisions.md, its notes and results research/tracks/psi0.md. Open questions and planned experiments (what is still uncertain but in scope): docs/experiments_roadmap.md.

checkpoints and rollouts

Public (unlicensed; all rights reserved) on Hugging Face: jacob-valdez/rrp-checkpoints — arm v1 routes, v2/v6 BC experts, legged8 routes, gait trackers, and 160 recorded evaluation rollouts (rollouts/viz_replays). MANIFEST.tsv there lists sha256 per file. Weights are never committed to this repo.