1. Environment

• CUDA 12.6, 64-bit Python 3.10 and PyTorch 2.5.1, other environments may also work
• Users can use the following commands to install the packages

conda create -n newtongen python=3.10
conda activate newtongen
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1



git clone https://github.com/pandayuanyu/NewtonGen.git
cd NewtonGen
pip install -r requirements.txt

One more step: you need to move the folder git under where you install package rp, we can check the location of rp by

pip show rp

Once finished, you can see the path like virtual_envs/newtongen/lib/python3.10/site-packages/rp/git/CommonSource/

2. Prepare Models and Weights

There are 3 types of weights you need to download/prepare.

1. Download CogVideoX-5B and pre-trained weights from Hugging Face
2. The lora weight from Go-with-the-Flow: T2V5B_blendnorm_i18000_DATASET_lora_weights.safetensors
3. The learned ODE weights for different dynamics: already in our folder learned_dynamics

1. Choose the Motion You Need

• Linear motions (uniform motion, acceleration, deceleration, 3D motion, slope sliding, parabolic motion)
  Use inference_Linear_withprompts.py since the object can be treated as a whole when rendering optical flow.

• Other motions (rotation, size changing, parabolic motion with rotation, damped oscillation)
  These require separate handling because different parts of the object move in different directions. Each has its own inference script.

2. Modify the Models and Weights Root

Set all the MODEL_PATH, sample_path=sample_path, output_mp4_path and other roots to your own ones in the inference_xx.py.

We have provided the well-trained NND weights at folder learned_dynamics.

3. Modify the Configuration and Prompts

Change the Z0 dict in config_list according to your need. Tips: set the size-related s l larger will have a better visual effect.

config_list = [
    dict(
        z0=[6.9901, 9.3459, 5.558, -4.8493, 0.0, 0.0, 1.0, 2.0, 2.0], # x y vx vy theta omega s l a 
        DT=0.02,                                                      # delta time between each 2 frames
        METER_PER_PX=0.05,                                            # world size: 360*0.05 = 18m
        chosen_shape="circle",                                        # the nearest shape of the object
        output_name="set_a"
    ),
    ]

4. Run

python inference_xx.py

1. Prepare Physics-Clean Videos

Run the scripts in folder data_simulation. We also provide some simulated samples in the folder.

You can also write other motions' simulators following our pipeline or collect high-fps real-world videos by yourself.

2. Encode the Physics-Clean Videos into Physical States Labels

Run the script in folder physical_encoder. For Circular Motion and Damped Oscillation, please use phys_encoder_circular_oscillation.py, all other motions, please use phys_encoder.py.

3. Training Example

Before you start NND training, set the dynamics_batch, other root paths, and hyper-parameters in the train.py,

python train.py

You can prepare validation Physical States Labels from upper step 1, or you can ignore the validation part in the train.py.