RexSeek — Reproduction Guide

Tested environment: Ubuntu 24, Miniforge3, CUDA 12.1, GPU with compute capability 8.9 (e.g. RTX 4090). This guide documents every fix discovered during a multi-day debugging session. Follow it top to bottom without skipping steps.

---

Table of Contents

1. Prerequisites
2. Clone the Repositories
3. Create the Conda Environment
4. Install Python Packages
5. Build GroundingDINO CUDA Ops
6. Register GroundingDINO & Install RexSeek
7. Download Model Weights
8. Apply Required Patches
9. Sanity Check
10. Running Inference
11. Known Issues & Fixes

---

Prerequisites

Make sure the following are already installed on your system:

• Miniforge3 (or Anaconda/Miniconda)
• CUDA 12.1 toolkit (nvcc available at /usr/local/cuda)
• git, wget
• A GPU with CUDA compute capability 8.9 (Ada Lovelace, e.g. RTX 4090/4080). If your GPU is different, adjust TORCH_CUDA_ARCH_LIST in the build step accordingly (e.g. "8.0" for A100, "7.5" for RTX 20xx, "8.6" for RTX 30xx).

---

Clone the Repositories

mkdir -p ~/Documents/Projects
cd ~/Documents/Projects

# Main RexSeek repo
git clone https://github.com/IDEA-Research/RexSeek.git

# GroundingDINO goes INSIDE the demos folder
cd RexSeek/demos
git clone https://github.com/IDEA-Research/GroundingDINO.git

Create the output directory for results:

mkdir -p ~/Documents/Projects/RexSeek/outputs

---

Create the Conda Environment

⚠️ Critical: Use Python 3.10.11 specifically. The conda-forge build of Python 3.10.19 (h3c07f61_3_cpython) has a broken zipfile module that corrupts large CUDA wheel installs and causes random stdlib corruption at runtime (manifesting as random sre_parse, configparser, tokenize crashes). Python 3.10.11 is stable.

conda create -n rexseek python=3.10.11 -c conda-forge -y
conda activate rexseek

# Verify
python --version
# Must show: Python 3.10.11

---

Install Python Packages

⚠️ Critical: pip 26.x has a regression in its dependency resolver (UnboundLocalError: local variable 'specifier_set') that breaks PyTorch CUDA wheel installs. Downgrade pip first, then use uv for all actual package installs — it bypasses Python's zipfile module entirely and is dramatically faster.

Step 1 — Bootstrap pip and uv

python -m pip install "pip<26"
pip install uv

Step 2 — Core build tools and numpy

uv pip install "setuptools<70"
uv pip install "numpy==1.26.4"

Step 3 — PyTorch with CUDA 12.1

uv pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu121

Step 4 — Core ML packages

uv pip install transformers==4.44.2 accelerate==1.5.2 timm==1.0.15 "safetensors==0.5.3" "Pillow==11.1.0"

Step 5 — supervision (with numpy/opencv pin fix)

⚠️ supervision will pull in incompatible versions of numpy and opencv. Force-reinstall the pinned versions immediately after.

uv pip install "supervision==0.25.1"
uv pip install "numpy==1.26.4" "opencv-python==4.8.1.78" \
    --no-deps \
    --reinstall-package numpy \
    --reinstall-package opencv-python

Verify:

python -c "import numpy; import cv2; print('numpy:', numpy.__version__); print('cv2:', cv2.__version__)"
# Must show: numpy: 1.26.4   cv2: 4.8.1

Step 6 — spaCy

uv pip install "spacy==3.8.3"
python -m spacy download en_core_web_sm

Step 7 — Remaining packages

uv pip install open_clip_torch==2.31.0 pycocotools==2.0.8 deepspeed==0.16.4
uv pip install tabulate==0.9.0 termcolor==2.5.0 typer==0.15.2 shortuuid==1.0.13 \
    huggingface_hub addict yapf

---

Build GroundingDINO CUDA Ops

Step 1 — Set environment variables

export CUDA_HOME=/usr/local/cuda
export BUILD_WITH_CUDA=True
export TORCH_CUDA_ARCH_LIST="8.9"   # Change to match your GPU compute capability
export PIP_USER=false
export LD_LIBRARY_PATH=$HOME/miniforge3/envs/rexseek/lib/python3.10/site-packages/torch/lib:${LD_LIBRARY_PATH:-}

Step 2 — Patch GroundingDINO's setup.py

The original setup.py calls install_torch() which tries to reinstall PyTorch and will break your environment. Patch it out:

cd ~/Documents/Projects/RexSeek/demos/GroundingDINO

python -c "
import re
content = open('setup.py').read()
patched = re.sub(r'^(\s*)install_torch\(\)\s*$', r'\1# install_torch() # patched', content, flags=re.MULTILINE)
open('setup.py', 'w').write(patched)
print('Patched OK')
"

Step 3 — Clean and build

rm -rf build/ dist/ groundingdino.egg-info/ 2>/dev/null || true
find groundingdino/ -name "_C*.so" -delete 2>/dev/null || true

python setup.py build_ext --inplace 2>&1 | tail -5

The last output line should look like:

copying build/.../groundingdino/_C.cpython-310-x86_64-linux-gnu.so -> groundingdino

Step 4 — Verify the build

python -c "import groundingdino._C; print('CUDA ops OK')"

---

Register GroundingDINO & Install RexSeek

GroundingDINO needs to be importable from inside the RexSeek demos directory. Register it via a .pth file:

SITE_PACKAGES=$HOME/miniforge3/envs/rexseek/lib/python3.10/site-packages
echo "$HOME/Documents/Projects/RexSeek/demos/GroundingDINO" > $SITE_PACKAGES/groundingdino.pth

# Verify it resolves to the right location
python -c "import groundingdino; print(groundingdino.__file__)"
# Must show: .../RexSeek/demos/GroundingDINO/groundingdino/__init__.py

Install RexSeek in editable mode without overriding pinned deps:

cd ~/Documents/Projects/RexSeek
uv pip install -e . --no-deps

---

Download Model Weights

GroundingDINO weights (~170 MB)

mkdir -p ~/Documents/Projects/RexSeek/demos/GroundingDINO/weights

wget -q --show-progress \
    https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth \
    -O ~/Documents/Projects/RexSeek/demos/GroundingDINO/weights/groundingdino_swint_ogc.pth

RexSeek-3B weights (~7.4 GB)

cd ~/Documents/Projects/RexSeek

python -c "
from huggingface_hub import snapshot_download
snapshot_download(repo_id='IDEA-Research/RexSeek-3B', local_dir='./weights/RexSeek-3B')
print('Done')
"

---

Apply Required Patches

Two patches must be applied to rexseek_grounding_dino_spacy.py before inference will work correctly.

Patch 1 — Add re/sre_compile prewarm header

This ensures the stdlib re module is fully initialized before any downstream package touches it:

python - << 'EOF'
import os, re
path = os.path.expanduser("~/Documents/Projects/RexSeek/demos/rexseek_grounding_dino_spacy.py")
content = open(path).read()
lines_content = content.split('\n')
start = 0
for i, line in enumerate(lines_content):
    stripped = line.strip()
    if stripped and not any(stripped.startswith(p) for p in (
        'import sys', 'import types', 'import re', 'import sre',
        'import collections', 'import configparser', 'import textwrap',
        'import enum', 'import importlib', '_ds', 'sys.modules', '#'
    )):
        start = i
        break
clean = '\n'.join(lines_content[start:])
header = 'import re\nimport sre_compile\n\n'
open(path, 'w').write(header + clean)
print('Script reset OK, starts with:', lines_content[start][:60])
EOF

Patch 2 — Person-anchor noun extraction fix

The default spacy_noun_phrases function returns complex noun chunks that GroundingDINO cannot match when the prompt contains person-related words. This patch forces it to return "person" as the grounding anchor for person-related queries:

python - << 'PYEOF'
import os
path = os.path.expanduser("~/Documents/Projects/RexSeek/demos/rexseek_grounding_dino_spacy.py")
content = open(path).read()

old_func = (
    'def spacy_noun_phrases(caption):\n'
    '    """Extract noun phrases from a given text caption using spaCy.\n'
    '\n'
    '    Args:\n'
    '        caption (str): The input text caption to process.\n'
    '\n'
    '    Returns:\n'
    '        list[str]: A list of lowercase noun phrases extracted from the caption.\n'
    '    """\n'
    '    doc = nlp(caption)\n'
    '    nouns = []\n'
    '    for nounc in doc.noun_chunks:\n'
    '        nouns.append(str(nounc).lower())\n'
    '    return nouns'
)

new_func = (
    'def spacy_noun_phrases(caption):\n'
    '    doc = nlp(caption)\n'
    '    tokens = [t.text.lower() for t in doc]\n'
    '    PERSON_WORDS = {"person", "man", "woman", "people", "human", "boy", "girl",\n'
    '                    "child", "guy", "lady", "gentleman", "student", "worker"}\n'
    '    for word in PERSON_WORDS:\n'
    '        if word in tokens:\n'
    '            return ["person"]\n'
    '    nouns = [str(nounc).lower() for nounc in doc.noun_chunks]\n'
    '    return nouns if nouns else ["person"]'
)

if old_func in content:
    open(path, 'w').write(content.replace(old_func, new_func))
    print('Person-anchor patch: OK')
else:
    print('Pattern not found - may already be patched')
PYEOF

---

Sanity Check

Run this before any inference to confirm everything is wired up correctly:

cd ~/Documents/Projects/RexSeek

python -c "
import torch, numpy as np, cv2, spacy, groundingdino
import groundingdino._C
print('torch  :', torch.__version__)
print('numpy  :', np.__version__)
print('opencv :', cv2.__version__)
print('CUDA   :', torch.cuda.is_available())
assert int(np.__version__.split('.')[0]) < 2, 'numpy must be < 2!'
nlp = spacy.load('en_core_web_sm')
print('spacy  : OK')
print('gdino  :', groundingdino.__file__)
print('ALL CHECKS PASSED')
"

Expected output:

torch  : 2.5.1+cu121
numpy  : 1.26.4
opencv : 4.8.1
CUDA   : True
spacy  : OK
gdino  : .../RexSeek/demos/GroundingDINO/groundingdino/__init__.py
ALL CHECKS PASSED

---

Running Inference

cd ~/Documents/Projects/RexSeek

CUDA_VISIBLE_DEVICES=0 python demos/rexseek_grounding_dino_spacy.py \
    --image demos/demo_images/demo1.jpg \
    --output outputs/test_out.jpg \
    --referring "person that is giving a proposal" \
    --box-threshold 0.25 \
    --text-threshold 0.25

Arguments

---

Known Issues & Fixes