SpecDETR

This is the official repository for "SpecDETR: A transformer-based hyperspectral point object detection network", and it is also part of the open-source hyperspectral object detection toolbox HODToolbox.

Paper link: ISPRS P&RS or arXiv.

Contributions

1. We verified that existing visual objec detection networks possess the capability for spatial-spectral integrated semantic representation of tiny objects. These networks can effectively detect subpixel-level objects in hyperspectral images, outperforming pixel-wise hyperspectral target detection methods.

2. We developed the first multi-class hyperspectral tiny object detectio benchmark dataset, SPOD. Based on this dataset, a comprehensive evaluation was conducted on the performance of mainstream visual object detection networks and hyperspectral target detection methods on the hyperspectral tiny object detection task.

3. We proposed SpecDETR, an innovate tiny object detection network that introduces a self-excited mechanism to enhance feature extraction and designs a streamlined yet efficient novel DETR decoder architecture tailored for tiny object characteristics. Experiments demonstrate that SpecDETR significantly outperforms existing approaches in the hyperspectral tiny object detection task.

4. We developed an open-source hyperspectral object detection toolbox, HODToolbox, facilitating the paradigm shift from traditional pixel-level hyperspectral target detection to hyperspectral object detection. The toolbox integrates the following core functionalities:

   • Convert traditional hyperspectral target detection datasets into object detection datasets, and use single-target prior spectra to generate large-scale training image sets for object detection networks training.
   • Train and test mainstream visual object detection networks on hyperspectral object detection datasets.
   • Quantitative evaluation and visual analysis of detection results.

News & Updates

• June 29, 2025: We make the following updates:

  1. Open-source the simulated training sets for three public HTD datasets: Avon, SanDiego, and MUUFLGulfport.
  2. Add support for the infrared video satellite flying airplane detection dataset IRAir.
  3. Provide pre-trained SpecDETR models for hyperspectral tiny object detection on three public datasets (Avon, SanDiego, MUUFLGulfport) and single-frame infrared tiny object detection on IRAir dataset.
  4. Released the companion toolbox HODToolbox

• May 08, 2025: We are pleased to announce that our work SpecDETR has been accepted by ISPRS Journal of Photogrammetry and Remote Sensing!

Abstract

Hyperspectral target detection (HTD) aims to identify specific materials based on spectral information in hyperspectral imagery and can detect extremely small-sized objects, some of which occupy a smaller than one-pixel area. However, existing HTD methods are developed based on per-pixel binary classification, neglecting the three-dimensional cube structure of hyperspectral images (HSIs) that integrates both spatial and spectral dimensions. The synergistic existence of spatial and spectral features in HSIs enable objects to simultaneously exhibit both, yet the per-pixel HTD framework limits the joint expression of these features. In this paper, we rethink HTD from the perspective of spatial–spectral synergistic representation and propose hyperspectral point object detection as an innovative task framework. We introduce SpecDETR, the first specialized network for hyperspectral multi-class point object detection, which eliminates dependence on pre-trained backbone networks commonly required by vision-based object detectors. SpecDETR uses a multi-layer Transformer encoder with self-excited subpixel-scale attention modules to directly extract deep spatial–spectral joint features from hyperspectral cubes. During feature extraction, we introduce a self-excited mechanism to enhance object features through self-excited amplification, thereby accelerating network convergence. Additionally, SpecDETR regards point object detection as a one-to-many set prediction problem, thereby achieving a concise and efficient DETR decoder that surpasses the state-of-the-art (SOTA) DETR decoder. We develop a simulated hyperSpectral Point Object Detection benchmark termed SPOD, and for the first time, evaluate and compare the performance of visual object detection networks and HTD methods on hyperspectral point object detection. Extensive experiments demonstrate that our proposed SpecDETR outperforms SOTA visual object detection networks and HTD methods.

Installation

The project is built upon mmdetection 3.0.0 and runs on the Ubuntu system.

1. Create a new conda environment and activate the environment. Requires Python>=3.7.

2. Install Pytorch. Requires torch>=1.8.

3. Install mmengine and mmcv.

   pip install mmengine==0.7.3
   pip install mmcv==2.0.0
   

4. Clone this repository:

   SpecDETR_ROOT=/path/to/clone/SpecDETR
   git clone https://github.com/ZhaoxuLi123/SpecDETR $SpecDETR_ROOT
   

5. Compile and install mmdet. If mmdet>=3.0.0 is already installed in the environment, skip this step.

   cd $SpecDETR_ROOT
   pip install -v -e .
   

Dataset Preparation

SPOD Dataset

1. Download SPOD_30b_8c.zip from Baidu Drive (key: 2789) or OneDrive.

2. Unzip SPOD_30b_8c.zip locally.

3. Update the dataset configuration files configs/_base_/datasetshsi_detection.py and configs/VisualObjectDetectionNetwork/_base_/datasets/hsi_detection4x.py:

     data_root = 'Dataset_Root/SPOD_30b_8c/'  # ← Update to the local path
   

Avon Dataset, SanDiego Dataset, and MUUFLGulfport Dataset

1. Download dataset zip files:

   • Avon Dataset: Baidu Drive (key: 2789) or OneDrive

   • SanDiego Dataset: Baidu Drive (key: 2789) or OneDrive

   • MUUFLGulfport Dataset: Baidu Drive (key: 2789) or OneDrive

2. Unzip dataset zip files locally.

3. Update the dataset configuration files configs/_base_/datasetshsi_detection.py and configs/VisualObjectDetectionNetwork/_base_/datasets/hsi_detection4x.py:

     data_root = 'Dataset_path/'  # ← Update to the local path of the dataset.
   

Other HTD Dataset

We provide conversion scripts from HTD Dataset to HOD Dataset in HODToolbox.

For details on adding new HOD datasets, please refer to the Training and Inference of the HOD Task section in HODToolbox.

IRAir Dataset

We will release the IRAir dataset soon. Please stay tuned on the project website: IRAir

The IRAir dataset configuration file is located at:
configs/_base_/datasets/irair_real_label.py

Model Zoo

We provide pre-trained SpecDETR models for SPOD Dataset, Avon Dataset, SanDiego Dataset, MUUFLGulfport Dataset, and IRAir Dataset.

Model Training

SpecDETR

To train SpecDETR on the SPOD dataset, execute either of the following commands:

python train.py --dataset SPOD

or

python train.py --config ./configs/specdetr/SpecDETR_SPOD_100e.py --work-dir ./work_dirs/SpecDETR/SPOD/

Note: Although we have fixed all random seeds, there may still be slight differences in AP performance each time you run. This difference originates from the underlying mechanism of CUDA.

Other Model

We provide configuration files for existing visual object detection networks on the SPOD dataset in the configs/VisualObjectDetectionNetwork directory. Execute the following command to train these models:

python train.py --config ./configs/VisualObjectDetectionNetwork/dino-5scale_swin-l-100e_hsi4x.py --work-dir ./work_dirs/dino-5scale_swin-l/SPOD/

Model Evaluation

Use SPOD dataset as example.

1. Model Inference
   Run the following command to obtain SpecDETR's inference results on the SPOD test set:

   python test.py --dataset SPOD  
   

   or

   python test.py --config ./configs/specdetr/SpecDETR_SPOD_100e.py \  
                 --work-dir ./work_dirs/SpecDETR/SPOD/ \  
                 --checkpoint ./work_dirs/SpecDETR/SpecDETR_SPOD_100e.pth \  
                 --out ./work_dirs/SpecDETR/SPOD/SpecDETR.pkl  
   

2. Detection Accuracy Evaluation
   Please refer to the Quantitative Evaluation of Results section in HODToolbox.

3. Inference Speed Evaluation
   Execute the following command to evaluate SpecDETR's inference speed on the SPOD test set:

   python benchmark.py --config ./configs/specdetr/SpecDETR_SPOD_100e.py \  
                      --checkpoint ./work_dirs/SpecDETR/SpecDETR_SPOD_100e.pth  
   

4. FLOPs Calculation
   Run the following command to compute FLOPs:

   python get_flops.py --config ./configs/specdetr/SpecDETR_SPOD_100e.py  
   

Benchmark

Partial Quantitative Results of Visual Object Detection Networks on SPOD Dataset

Partial Quantitative Results of HTD Methods on SPOD Dataset

Citation

If the work or the code is helpful, please cite the paper:

@article{li2025specdetr,
  title={SpecDETR: A transformer-based hyperspectral point object detection network},
  author={Li, Zhaoxu and An, Wei and Guo, Gaowei and Wang, Longguang and Wang, Yingqian and Lin, Zaiping},
  journal={ISPRS Journal of Photogrammetry and Remote Sensing},
  volume={226},
  pages={221--246},
  year={2025},
  publisher={Elsevier}
}

Contact

For further questions or details, please directly reach out to lizhaoxu@nudt.edu.cn.