smolvla_flow_rl

Two LIBERO Long episodes from the author's own runs, side by side. It is footage, not a measurement: two episodes, chosen, with no repetition and no statistics behind either panel.

Full video: https://www.linkedin.com/posts/mehmetturanyardimci_robotics-reinforcementlearning-vla-ugcPost-7488179589240799232-7Gv6/

Online reinforcement learning fine tuning for a flow matching vision language action policy, on a single consumer GPU.

Project page: https://mturan33.github.io/smolvla_flow_rl/

A flow matching policy produces an action chunk by integrating a learned velocity field from noise. Integrating it deterministically gives no action likelihood, so there is nothing for a policy gradient to act on. This repository takes the standard route around that, following the approach of pi_RL (see CITATION.cff): one step of the integration is treated as stochastic, so the sampled chunk has a tractable log probability. Around that it builds the machinery such a run needs in order to be trustworthy rather than merely runnable.

No results are reported here. Every number in this repository describes the machinery, what it trains, what it weighs and how long it takes, and none describes how well it works. The clip above is two episodes of footage and is not evidence of anything, whatever either panel appears to show. Measurements made with this substrate will be released with the accompanying paper.

What is in it

• A stochastic sampler for flow matching policies that leaves the pretrained action head bit identical, driving it through a callback rather than modifying it.
• A policy gradient update that grades the action the environment actually executed, over the executed part of the chunk only, with the likelihood ratio and its spread recorded once per update cycle, from the last update of that cycle.
• A critic head with adaptive target rescaling, kept outside the action head.
• A parameter manifest and a learning rate proof printed before the first update, so what is training and at what rate is a logged fact rather than an assumption.
• A process isolated simulator harness with a sparse terminal reward and no shaping.
• A deterministic evaluation entry point that reads several horizons out of one run, with a completeness gate that refuses to produce a number from an unfinished evaluation.
• A verification protocol whose rules are written down, with the subset that can be checked mechanically wired into a single gate entry point.

Scope

This is the engineering substrate. It stands on its own as a starting point for anyone doing reinforcement learning fine tuning of a flow matching policy on limited hardware.

Design: which part of the policy should train

A vision language action policy has two things that a fine tuning run can change: the representation that reads the instruction and the scene, and the decoder that turns that representation into an action. This repository offers four trainable surfaces, and the choice among them is the single argument most likely to decide whether a run does anything at all.

The recommended default is expert_full_vlm_lora: the action expert trains at full rank, low rank adapters on the language backbone train alongside it, and the flow action head trains at full rank. The vision tower and the connector stay frozen.

The reasoning is that the other three each leave out one of the two things that have to move.

• frozen adapts the language side and leaves the action decoder exactly as it was. Whatever the representation learns has to be expressed through a decoder that cannot change how it expresses anything.
• lora opens the decoder through adapters and leaves the backbone bare. The decoder can shift, but only by as much as a low rank correction allows, and the representation feeding it never adapts to the new objective.
• full opens the decoder at full rank and still leaves the backbone bare, so a changed instruction has no path to a changed representation.

Adapting the language side with low rank adapters while training the action decoder at full rank is the combination that the recent fine tuning literature reports as the one that works, and the same references also report that leaving the action head untouched is a common way to spend a training run without moving the policy:

• Kim et al., Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (OpenVLA OFT), 2025.
• RoboMME, a multimodal robot manipulation evaluation suite, 2026.
• arXiv:2607.10172.
• VLAFlow, arXiv:2607.01586.

Two things this section is not. It is not a report of a measurement made here: this repository publishes no results, and the recommendation above is a statement about published practice and about which surfaces are structurally capable of changing what. And it is not a claim that the other three regimes are useless. They are kept precisely because a comparison needs controls: full is the adapter free control, lora is the adapter only decoder, and frozen is the decoder untouched control. A run in any of them has to say so, with --control_arm and a reason, and the reason is recorded.

What the gate enforces, and what it does not

Before the first update, build_model prints the manifest and then refuses to continue unless the action expert's own parameters are at least 99 percent trainable, or the run declared itself a control. The fraction is computed rather than compared against a fixed parameter count, so it survives a change of model size, of layer count or of naming.

The check deliberately asks whether the expert's own body trains, not whether the expert appears anywhere in the trainable set. An adapter attached to the expert satisfies the second and not the first, and the difference between those two is invisible in a casually read parameter total.

The action head, and the narrowed parity invariant

In frozen, lora and full the pretrained action head does not train, and the protocol's behavioural parity check rests on that: before any parameter has moved, the wrapper must reproduce the pretrained action. The manifest's action head group reads zero in those three, and a nonzero reading is the fault.

In expert_full_vlm_lora the head trains at full rank, so the same reading inverts: nonzero is correct there and a zero is the fault. Parity with the pretrained policy is a statement about a surface that leaves the action path alone, and this regime does not. Asking it of a regime that deliberately changes the head would produce a red line that is waived on every run, which teaches a reader to ignore red lines. The scope of the invariant is therefore written into protocol/VERIFICATION_PROTOCOL.md, into src/model_setup.py, and into an executable test that fails if the scope is widened again.

The reference anchor is refused in this regime, on purpose

The KL anchor recovers the pretrained policy by switching the adapters off in place, which avoids holding a second copy of the weights. That trick is only valid when every trainable parameter lives in an adapter. In expert_full_vlm_lora the expert and the head are trained in place, so switching the adapters off returns the trained policy with its adapters disabled, not the pretrained one; the anchor would pull toward the current policy and log a penalty that means nothing.

The entry point therefore refuses --kl_coef other than zero in this regime and says why. The alternative, a full second copy of the weights held for the reference, would roughly double the memory that this regime was chosen to fit inside, so it is not offered. This is a documented refusal rather than a silent fallback, which is the whole distinction: the check that used to guard this counted adapter layers and would have passed here, since adapters are present, while the requirement it was meant to enforce was not met.

Why some of this looks paranoid

Several parts of this repository exist because the corresponding mistake was made and cost real time. They are kept because the failures share a shape: the run keeps going, the loss goes down, and nothing reports an error.

• The likelihood is summed over the executed surface only. Summing over the padded chunk inflates the variance of the log ratio, pushes most samples outside the clip range, and throttles the gradient into a slow random walk.
• The update grades stored trajectories. Drawing a fresh chunk at update time and pairing it with a stored advantage gives a zero mean gradient, whose signature is a likelihood ratio pinned at one.
• The parameter manifest is printed before training. A run that opened a surface other than the one intended still reports a loss, because whatever is trainable moves, and the manifest is the only place the difference shows.
• The evaluation completeness gate counts episodes. Checking that every task appeared at least once is satisfied almost immediately and will happily produce a number from a fraction of the intended episodes.
• The checkpoint stores the random state. Without it a resumed run draws a different sequence than an uninterrupted one and is no longer reproducible from its seed.

Hardware

Developed and tested on a single RTX 5070 Ti laptop GPU (12GB VRAM), under WSL2. Anything with 12GB or more should behave the same; less is untested.

Getting started

Linux, with a CUDA card and an EGL capable driver. EGL is the simulator's headless path, and the two run wrappers select it. The evaluation entry point is invoked directly rather than through a wrapper, so it inherits whatever the shell already has; export MUJOCO_GL=egl and PYOPENGL_PLATFORM=egl when scoring a checkpoint outside a training run.

Install the dependencies, fetch the pretrained policy, then run the single task contact test before anything else. The contact test answers whether the setup can learn at all, which is worth establishing before reading anything into what a run produces.

conda env create -f environment.yml
conda activate smolvla-flow-rl

# or, into an environment you already have:
# pip install -r requirements.txt
python -c "import libero.libero"          # answer the prompt once, see below
python scripts/download_base_policy.py --out /path/to/base
bash examples/single_task_contact.sh /path/to/base

The middle line is there because the simulator package asks an interactive question the first time it is imported, before this repository gets a say, and writes the answer to ~/.libero/config.yaml. Under a shell with no terminal attached, which is what nohup, cron and most continuous integration give it, the question cannot be answered and the import dies with an EOFError naming neither this repository nor the cause. Run it once by hand and the rest is unattended. Keep a network available for the first run beyond fetching the policy. A source checkout of the simulator carries its assets in the package; a wheel may not, in which case the first environment built fetches them. Which of those applies to the distribution declared here was not established on the machine this was developed on, where the source checkout is what is installed.

The policy must be one conditioned on this task suite. The harness supplies two camera images under particular keys, an eight dimensional state, and expects seven dimensional actions; a checkpoint declaring anything else cannot be driven by it. The generic SmolVLA base checkpoint is such a case, so the helper does not default to it. Passing a checkpoint that does not fit stops the run at construction with a message naming the mismatch, rather than failing later inside a simulator worker.

Fourteen arguments are required. Nine of them describe the run: the task list, the two paths, the number of update cycles, the rollout length, the episode length, the number of integration steps, the seed, and which part of the policy trains. The other five are tuning knobs: the exploration noise scale, the discount, the trace decay and the two learning rates.

None of the fourteen has a default, and the reason is the same in both groups. A default silently becomes a project wide constant that nobody chose and everybody inherits. The trainable surface is in that list for the same reason and not as an oversight: opening a surface other than the one intended produces a run whose arguments look right and whose manifest is the only place the difference shows.

python src/train_flow_rl.py --help

To score a checkpoint on its own, outside a training run, use the evaluation entry point. It refuses to print a rate until the completeness gate passes, then reads the same evaluation at every budget asked for.

python src/eval_cli.py --help

Verification

The mechanical checks run from one entry point.

bash scripts/run_gates.sh

That entry point runs three: a syntax and artefact check over the tree, an import closure walk that fails if any entry point reaches a module this repository does not contain, and the invariant tests. The protocol contains further rules that are stated but not automated here; each is marked in the document with how far it is taken. See protocol/VERIFICATION_PROTOCOL.md for what each rule is and why it exists.

What the update of 30 August 2026 added

These are the parts that decide whether a number produced by this substrate can be trusted. None of them is an experiment and none reports a result.

• Reference operators in one file (src/reference_ops.py): the policy loss with a dual clip at 3.0, the value loss with a trust region of 0.2 and a Huber delta of 10.0, and truncation compensation on the reward side. Each cites the reference file and configuration key it follows. They live in one module because two PPO paths that each write their own clip drift apart, and a drift between two runs you intend to compare is not a bug in one of them, it is a confound in the comparison.
• Gradient accumulation (src/ppo_step.py): the optimiser steps once per global batch rather than once per micro batch. The failure this prevents is silent. A loop that calls zero_grad and step on every micro batch trains at the micro batch size while every banner and log line reports the pooled figure; nothing errors and the curves look ordinary. accum_total=1 reproduces the old behaviour exactly.
• Seeded resets (src/seeding.py): a per-outer, per-environment stream from SeedSequence. Without it a second unseeded reset at the top of the outer loop makes two same-seed runs diverge at the first rollout step, on the observation side, before the sampler is called once.
• The single horizon gate (src/horizon_gate.py): training, in-training evaluation and the final canonical evaluation must read one horizon field, or the run does not start. A series measured at two horizons is not a series.
• The run configuration gate (src/run_config.py, scripts/diff_run_config.py): every run writes what it was, with claimed arithmetic and measured arithmetic in separate blocks, and the diff tool names which fields make two runs incomparable. Module constants and environment flags are recorded too, because the field that turns out to matter is usually the one that is not an argparse argument.
• Shape-based completeness for evaluation scans (scripts/canon_scan.sh): an interrupted scan relaunched into an appending file holds two scans, and any completeness check that counts lines will call that file complete. Completeness is the shape of the scan, not the number of lines in it.

Limitations

Stated because a substrate that only advertises its gates is advertising.

• No results. Nothing here reports how well any of this works. The gates say a number would be trustworthy; they say nothing about what the number is. A paper is in preparation.
• One benchmark, one policy family. Everything is exercised on LIBERO with a SmolVLA-family policy. None of it has been run against another simulator or another flow matching policy, so portability is untested rather than claimed.
• The likelihood is from one stochastic step. That is the simple route to a tractable log probability and it is not the only one; a learned noise network is the alternative in the literature. No comparison between those choices is made here.
• The gates are shape checks, not correctness proofs. A run that passes every gate in this repository can still be wrong. What the gates remove is a specific family of silent failures in which the run is not the run its own configuration describes.
• Single consumer GPU scale. Everything is sized for one card. Nothing here has been exercised at multi-node scale and the accumulation path in particular has only been run single process.
• bf16 is required by the training path. Ampere or newer; verify with python -c "import torch; print(torch.cuda.is_bf16_supported())" rather than inferring from the model name.

Licence and attribution

Apache License 2.0, see LICENSE. Four files are derivative works of RLinf: the sampler, the critic head, the update, and the action model wrapper. Two of those, the sampler and the wrapper, additionally follow LeRobot's own policy implementation, so they have two upstreams each. All four carry notices of modification in their headers. See NOTICE and CHANGES_FROM_UPSTREAM.md. LeRobot is therefore both a dependency and an upstream. No LIBERO code is adapted here, though its scenes and objects are what the clip at the top renders; NOTICE covers that. No source file from any of them is redistributed here.

No model weights are distributed. The download helper fetches two repositories from the Hugging Face Hub: the policy, HuggingFaceVLA/smolvla_libero, and the backbone that policy asks for, which it reads out of the policy's own configuration rather than assuming. Both state the Apache License 2.0 on their model cards at the revisions recorded in NOTICE.

Those terms are theirs and not this repository's. If you point --repo somewhere else, the helper fetches whatever backbone that policy asks for and says nothing about the terms of either, because it cannot know them.

If you use this, please cite it and the upstream work, see CITATION.cff.