We challenge RT-DETRv4 with the high-speed scene from Furious 7. Even under extreme motion blur, rapid camera pans, and chaotic lighting, the model maintains consistent tracking and high recall, proving its reliability in dynamic environments.

1. Download COCO2017 from OpenDataLab or COCO.
2. Modify paths in coco_detection.yml

train_dataloader:
    img_folder: /data/COCO2017/train2017/
    ann_file: /data/COCO2017/annotations/instances_train2017.json
val_dataloader:
    img_folder: /data/COCO2017/val2017/
    ann_file: /data/COCO2017/annotations/instances_val2017.json

To train on your custom dataset, you need to organize it in the COCO format. Follow the steps below to prepare your dataset:

1. Set remap_mscoco_category to False:

   This prevents the automatic remapping of category IDs to match the MSCOCO categories.

   remap_mscoco_category: False
   

2. Organize Images:

   Structure your dataset directories as follows:

   dataset/
   ├── images/
   │   ├── train/
   │   │   ├── image1.jpg
   │   │   ├── image2.jpg
   │   │   └── ...
   │   ├── val/
   │   │   ├── image1.jpg
   │   │   ├── image2.jpg
   │   │   └── ...
   └── annotations/
       ├── instances_train.json
       ├── instances_val.json
       └── ...
   

   • images/train/: Contains all training images.
   • images/val/: Contains all validation images.
   • annotations/: Contains COCO-formatted annotation files.

3. Convert Annotations to COCO Format:

   If your annotations are not already in COCO format, you'll need to convert them. You can use the following Python script as a reference or utilize existing tools:

   import json
   
   def convert_to_coco(input_annotations, output_annotations):
       # Implement conversion logic here
       pass
   
   if __name__ == "__main__":
       convert_to_coco('path/to/your_annotations.json', 'dataset/annotations/instances_train.json')
   

4. Update Configuration Files:

   Modify your custom_detection.yml.

   task: detection
   
   evaluator:
     type: CocoEvaluator
     iou_types: ['bbox', ]
   
   num_classes: 777 # your dataset classes
   remap_mscoco_category: False
   
   train_dataloader:
     type: DataLoader
     dataset:
       type: CocoDetection
       img_folder: /data/yourdataset/train
       ann_file: /data/yourdataset/train/train.json
       return_masks: False
       transforms:
         type: Compose
         ops: ~
     shuffle: True
     num_workers: 4
     drop_last: True
     collate_fn:
       type: BatchImageCollateFunction
   
   val_dataloader:
     type: DataLoader
     dataset:
       type: CocoDetection
       img_folder: /data/yourdataset/val
       ann_file: /data/yourdataset/val/ann.json
       return_masks: False
       transforms:
         type: Compose
         ops: ~
     shuffle: False
     num_workers: 4
     drop_last: False
     collate_fn:
       type: BatchImageCollateFunction
   

1. Training

   CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml --use-amp --seed=0
   

2. Testing

   CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml --test-only -r model.pth
   

3. Tuning

   CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --master_port=7777 --nproc_per_node=4 train.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml --use-amp --seed=0 -t model.pth
   

For example, if you want to double the total batch size when training RT-DETRv4-L on COCO2017, here are the steps you should follow:

1. Modify your dataloader.yml to increase the total_batch_size:

   train_dataloader:
       total_batch_size: 64  # Previously it was 32, now doubled
   

2. Modify your rtv4_hgnetv2_l_coco.yml. Here’s how the key parameters should be adjusted:

   optimizer:
     type: AdamW
     params:
       -
         params: '^(?=.*backbone)(?!.*norm|bn).*$'
         lr: 0.000025  # doubled, linear scaling law
       -
         params: '^(?=.*(?:encoder|decoder))(?=.*(?:norm|bn)).*$'
         weight_decay: 0.
   
   lr: 0.0005  # doubled, linear scaling law
   betas: [0.9, 0.999]
   weight_decay: 0.0001  # need a grid search
   
   ema:  # added EMA settings
       decay: 0.9998  # adjusted by 1 - (1 - decay) * 2
       warmups: 500  # halved
   
   lr_warmup_scheduler:
       warmup_duration: 250  # halved
   

If you'd like to train RT-DETRv4 on COCO2017 with an input size of 320x320, follow these steps:

1. Modify your dataloader.yml:

   train_dataloader:
     dataset:
         transforms:
             ops:
                 - {type: Resize, size: [320, 320], }
     collate_fn:
         base_size: 320
   
   val_dataloader:
     dataset:
         transforms:
             ops:
                 - {type: Resize, size: [320, 320], }
   

2. Modify your rtv4_base.yml (or the relevant base config file):

   eval_spatial_size: [320, 320]
   

1. Setup

   pip install onnx onnxsim
   

2. Export onnx

   python tools/deployment/export_onnx.py --check -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml -r model.pth
   

3. Export tensorrt

   trtexec --onnx="model.onnx" --saveEngine="model.engine" --fp16
   

1. Setup

   pip install -r tools/inference/requirements.txt
   

2. Inference (onnxruntime / tensorrt / torch)

   Inference on images and videos is now supported.

   python tools/inference/onnx_inf.py --onnx model.onnx --input image.jpg  # or video.mp4
   python tools/inference/trt_inf.py --trt model.engine --input image.jpg
   python tools/inference/torch_inf.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml -r model.pth --input image.jpg --device cuda:0
   

1. Setup

   pip install -r tools/benchmark/requirements.txt
   

2. Model FLOPs, MACs, and Params

   python tools/benchmark/get_info.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml
   

3. TensorRT Latency

   python tools/benchmark/trt_benchmark.py --COCO_dir path/to/COCO2017 --engine_dir model.engine
   

1. Setup

   pip install fiftyone
   

2. Voxel51 Fiftyone Visualization (fiftyone)

   python tools/visualization/fiftyone_vis.py -c configs/rtv4/rtv4_hgnetv2_${model}_coco.yml -r model.pth
   

1. Auto Resume Training

   bash tools/reference/safe_training.sh
   

2. Converting Model Weights

   python tools/reference/convert_weight.py model.pth