D-FINE-seg

Real-time object detection, instance segmentation, and semantic segmentation — one codebase, one config flag (task: detect | segment | sem_seg), five model sizes (N -> X), and fused end-to-end deployment graphs for ONNX, TensorRT, OpenVINO, CoreML and LiteRT.

This repo hosts the released weights; the code, full benchmark tables and docs live in the GitHub repo. 📄 Paper (arXiv:2602.23043) · 🎬 Video tutorial · 🧪 Colab · Apache 2.0

One frame, three tasks, one config flag.

Highlights

• Three tasks, one model family — DETR-style detection (queries -> boxes), plus a lightweight Mask-DINO-style mask head for instance segmentation and a query-free head for semantic segmentation. All train-time-only additions; zero inference overhead.
• Five sizes (N/S/M/L/X) — COCO-pretrained weights for detection and instance segmentation, including dfine_*_obj2coco.pt (Objects365 -> COCO) for stronger fine-tuning inits.
• Headline accuracy — on Cityscapes fine-tuning, beats Ultralytics YOLO26 and RF-DETR on detection & instance-segmentation F1 and leads mIoU on semantic segmentation, at real-time latency with 2–3× fewer parameters (TensorRT FP16, end-to-end protocol).
• Multi-backend deployment — one export pipeline produces ONNX / TensorRT / OpenVINO / CoreML / LiteRT graphs with the postprocessor fused; INT8 quantization (OpenVINO, CoreML, LiteRT); ByteTrack tracking; SAM3 auto-labeling; Gradio demo.
• Not a fork — the detection core follows the D-FINE paper; segmentation heads, training, export and inference were reimplemented from scratch, with the mask-head design following the Mask DINO paradigm.

Results (Cityscapes, fine-tuned)

500 Cityscapes val images at original resolution, TensorRT FP16, batch 1, RTX 5070 Ti, end-to-end latency (preprocessing + forward + postprocessing). Every framework runs its own shipped inference code. Full protocol: cityscapes-benchmark.

VisDrone (detection) and TACO (instance segmentation) fine-tuning results, plus COCO-style AP and cross-format (Torch/TRT/OpenVINO/CoreML) comparisons on desktop, Intel N150 and Apple Silicon: GitHub README -> Benchmarks.

Checkpoints

All files are PyTorch .pt; weights auto-download on first use (no manual download needed).

No released semantic-segmentation weights yet — fine-tune from a dfine_seg_* checkpoint (train.pretrained_model_path); the backbone, encoder and mask fuser transfer, the neck and classifier train from scratch.

Usage

pip install dfine-seg           # inference + training
pip install 'dfine-seg[all]'    # + every export backend, SAM3, Gradio demo

from dfine_seg import load_model, read_image

model = load_model("s")                        # COCO detection, weights auto-downloaded
model = load_model("s", task="segment")        # COCO instance segmentation
model = load_model("path/to/model.pt")         # your checkpoint — size/task/classes auto-detected
model = load_model("path/to/model.engine")     # any exported artifact, picked by extension

out = model(read_image("path/to/image.jpg"))[0]
print(out["boxes"], out["scores"], [model.names[int(i)] for i in out["labels"]])

Train, export and bench from a config file:

dfine init          # writes ./config.yaml (task: detect | segment | sem_seg)
dfine split && dfine train && dfine export && dfine bench

Everything else — data formats, config keys, export knobs, inference flags — is in the GitHub README.

Example outputs

Detection on VisDrone (aerial) · instance segmentation on TACO (waste).

Architecture

Notes

• Weights are an init, not a final model — the released checkpoints are COCO-pretrained (detection/instance segmentation). Fine-tune on your own data for the task and classes you need; load_model detects task, classes and input size from the weights.
• TensorRT engines are GPU-specific — re-export on the target GPU. INT8 quantization shows a noticeable F1 drop on segmentation; FP16 is the recommended trade-off.
• Mosaic augmentation (mosaic_augs.mosaic_prob) is not recommended for the segmentation task — lower it toward 0 if masks look wrong.

Citation

@article{saakyan2026dfineseg,
  title   = {D-FINE-seg: Object Detection and Instance Segmentation Framework with Multi-Backend Deployment},
  author  = {Saakyan, Argo and Solntsev, Dmitry},
  journal = {arXiv preprint arXiv:2602.23043},
  year    = {2026},
  eprint  = {2602.23043}
}

@misc{peng2024dfine,
  title         = {D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinement},
  author        = {Yansong Peng and Hebei Li and Peixi Wu and Yueyi Zhang and Xiaoyan Sun and Feng Wu},
  year          = {2024},
  eprint        = {2410.13842},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}