RandOpt

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Yulu Gan, Phillip Isola

Paper | Project Page | Openreview | Starting with a 1D Experiment:

News

• [2026-07] Iterative RandOpt is now on the iterative-randopt branch! Also support the pip install-able implementation that drops into existing verl or huggingface/trl setups.

Requirements

Option1: Python / Conda

(optional) conda activate your_env
pip install -r requirements.txt

Option2: Docker

From the directory containing RandOpt/:

Run RandOpt

Post-train on your own dataset

Please follow the instructions in CUSTOM_DATASET_GUIDE.md

Post-train on a standard dataset

First download the data here: data/README.md

Then, from the RandOpt directory:

Distill top-k models into a single model

Please follow the instructions in distillation/README.md.

Run Baselines

Please follow the instructions in baselines/README.md

Having questions?

Open an issue @ github.com/sunrainyg/RandOpt/issues.

Citation

@misc{gan2026neuralthickets,
      title={Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights}, 
      author={Yulu Gan and Phillip Isola},
      year={2026},
      eprint={2603.12228},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2603.12228}, 
}