Qinglong Captions

AI multimodal media processing and document translation, built around Lance datasets.

Video · Image · Audio · OCR · Tagging · Translation · Source Separation · Image2PSD

中文说明 · Changelog

At a Glance

Qinglong Captions combines media understanding, batch inference, and versioned Lance data management in one GUI and scriptable toolchain.

4.7.0 Highlights

• OvisOCR2 now handles images and PDFs through either a direct Transformers runtime or an external OpenAI-compatible vLLM server, with shared bounding-box post-processing and safer generation behavior.
• MuSViT is now a complete ONNX sheet-music OMR workflow: it streams images and PDF pages and exports page diagnostics plus validated MusicXML and MIDI, including whole-PDF aggregation.
• Kimi K3 model choices and low / high / max reasoning controls are unified across Kimi VL and Kimi Code; the Gemini and MiniMax catalogs now follow their latest generation APIs.
• Image prompt templates now preserve their input and output contracts across providers, while Image2PSD / see-through gains upstream-aligned defaults, embedding reuse, and deterministic seeded SDE sampling.

Start Here

See the GUI manual, configuration guide, and troubleshooting guide for details.

Requirements

• Windows or Linux (the bundled Linux PowerShell installer currently downloads x86_64 only; install pwsh 7+ manually on ARM64 or other architectures)
• Python >=3.10,<3.13; the installer creates Python 3.11 by default
• uv; the installer attempts to install it when missing
• PowerShell 5.1+ on Windows; pwsh on Linux
• A GPU is optional for the base workflow, but local VLM/OCR/translation and Image2PSD commonly require GPU memory and substantial disk space
• Models are downloaded on first use and stored under huggingface/ by default

Installation

Windows

Run from the repository root:

.\1.install-uv-qinglong.ps1

The base install uses the default dependencies in pyproject.toml. Selecting a local Provider in the GUI installs the required optional profile incrementally.

Linux

The PowerShell installer filename contains a space, so keep the quotes:

chmod +x "./0.install pwsh.sh"
sudo bash "./0.install pwsh.sh"
pwsh ./1.install-uv-qinglong.ps1

The installer creates or reuses .venv or venv and resolves dependencies directly from pyproject.toml; this repository does not require a checked-in uv.lock for daily runs. The bundled helper downloads Linux x86_64 PowerShell; on ARM64 or another architecture, install pwsh 7+ through your distribution or the upstream documentation instead. The commands below use .venv; replace it with venv if that is the environment you kept.

Verify the installation

uv --version
uv run gui/launch.py --help

If a newly installed uv is not found, restart the shell and try again.

Complete a Job in Five Steps

1. Start the GUI

.\start_gui.ps1

The default browser URL is http://127.0.0.1:7899. If the port is busy, the GUI tries subsequent ports and prints the actual URL.

start_gui.ps1 runs uv run gui/launch.py. The Python entrypoint declares a PEP 723 script environment, so this wrapper uses a GUI-scoped runtime rather than promising to reuse the project .venv.

Direct alternatives:

uv run gui/launch.py --port 7899 --no-browser
uv run gui/launch.py --native --port 7899

• --no-browser disables automatic browser opening.
• --native uses a native window and requires pywebview.
• --cloud binds to 0.0.0.0 and should only be used on a trusted network or behind an authenticated gateway.

The GUI has no built-in login. Do not expose --cloud directly to the public internet.

2. Import media

Open Import, select a video, image, or existing data directory, and choose an output Lance path. Keep the default data version and storage options for your first run. Lance is the shared dataset used by splitting, tagging, captioning, and export.

3. Split videos

Open Split and select the imported dataset. Start with the default detector. Raise the threshold or minimum scene length when cuts are too frequent; lower the threshold when cuts are missed. Skip this step for image-only jobs.

4. Generate tags and captions

Use Tagger with WDTagger or CL Tagger, then open Caption:

1. Choose a cloud API, OpenAI-compatible server, or local model.
2. Enter the model, API key, and Base URL; follow the GUI prompt for local dependencies.
3. Select the prompt and output field, then test a small sample.
4. Run the full job after checking the sample. Jobs continue when you switch pages.

Skip Tagger when you only need captions, or skip Caption when you only need tags.

5. Export results

Open Export, select the dataset version, caption suffix, and output format, then export media and sidecar captions. Test a small sample before overwriting existing files. See the Import / Export guide.

Core Workflow

The main workflow is Import -> Split -> Tagger -> Caption -> Export. Jobs continue in GUI task tabs when you change pages.

Video splitting

Split detects scene boundaries in long videos and optionally extracts representative frames. It supports Content, Adaptive, Hash, Histogram, and Threshold detectors. See the video splitting guide for inputs and tuning.

Image tagging

Tagger runs WDTagger or CL Tagger against image directories and Lance datasets. General, character, and concept thresholds are configurable; gated models require accepted Hugging Face terms and HF_TOKEN. See the image tagging guide.

Captioning and multimodal descriptions

Caption connects cloud APIs, OpenAI-compatible servers, local OCR, local VLMs, and local ALMs to image, video, audio, and document inputs. Provider routing, segmentation, retries, prompts, and outputs are covered in the captioning guide.

Lance versions and media/caption export are documented in the Import / Export guide.

Other Tools

MuScriptor installs through the muscriptor-local profile and supports the official small, medium, and large models. Runs that request only MIDI, JSON, or JSONL do not need an audio synthesizer. If preview is enabled, its runtime preflight runs before model inference and stops the batch when FluidSynth or the official SoundFont is unavailable; disable preview to export symbolic outputs without it. The profile includes SOCKS proxy support for first-use SoundFont downloads.

Optional MIDI-only or left-original/right-synthesized preview audio, in either MP3 or WAV format, requires the native FluidSynth executable on PATH. Switching to WAV does not remove this requirement; MP3 additionally requires working soundfile/libsndfile MP3 encoding. On Windows, use an x64 build, add its extracted bin directory to PATH, restart the shell and GUI, and verify it with fluidsynth --version. The official MuseScore_General.sf2 SoundFont is resolved automatically; no system or custom SoundFont is required.

Run 2.7.1.muscriptor_webui.ps1 to try the official MuScriptor WebUI with the project's shared .venv and optimized SDPA runtime. The launcher opens the browser after the server is ready; pass -NoBrowser to disable this behavior. It accepts -Model small|medium|large (default: large), -Device auto|cpu|cuda|cuda:N, and -BatchSize N. The default 0 reads the recorded model-memory profile and selects an even batch from total VRAM without rerunning BS1/BS2 calibration. CUDA OOM retries with a smaller batch; on Windows, a detected per-process shared-GPU-memory spill reduces later batches by two. After each request, transient tensors and idle CUDA cache are released while model weights remain loaded. CPU uses BS1. Do not use uvx muscriptor serve, which creates a separate tool environment.

All project MuScriptor entrypoints use the model-memory profiles in config/muscriptor_batch_profiles.toml for minimum-VRAM preflight and the same adaptive CUDA runtime. The runtime subtracts the configured reserve and GPU memory already used outside PyTorch from its allocator budget. The GUI additionally substitutes the selected user's GPU total VRAM into those curves when the page opens to choose its batch; the reference GPU is measurement provenance, not a lookup condition. If total VRAM cannot fit the model after the size-dependent reserve, auto falls back to CPU before loading weights and explicit CUDA is rejected. Changing the GUI model or device refreshes the recommendation until the batch control is edited manually.

See the tool documentation index for every entrypoint.

Choose a Caption Model

• Cloud Providers are configured on the Caption page.
• OpenAI-compatible services use openai_base_url and openai_model_name; a local service may use a placeholder API key.
• Selecting a local OCR / VLM / ALM route lets the GUI install the matching uv extra and show memory guidance.
• Gated or private Hugging Face models require HF_TOKEN; do not commit tokens in PowerShell files or Markdown.
• See docs/openai_compatible.md for server examples.

The GUI stores environment settings in config/env_vars.json. It is plaintext local state and may contain tokens or proxy information. Do not commit, upload, or share it.

Batch Scripts

PowerShell wrappers keep editable settings in a Configuration block near the top. Edit that block, then run from the repository root:

Example:

.\lanceImport.ps1
.\4.captioner.ps1
.\lanceExport.ps1
.\5.translate.ps1

Wrappers install the selected profile incrementally. Avoid manually assembling a long list of uv sync --extra commands; see docs/configuration.en.md when you need precise profile control.

Data and security notes

• Lance is the main intermediate format for imports, captions, tags, and translations; select the intended version/tag before export.
• Translation writes language-suffixed Markdown and does not overwrite the source file by default.
• Logs may contain input paths, model names, and error details. Redact logs before sharing them.
• Some caption paths still pass API keys as command-line arguments. Do not publish full commands, process-list screenshots, or raw task logs containing credentials.
• --cloud has no built-in authentication and binds the GUI to all interfaces.

Troubleshooting

1. uv is not found: restart the shell, confirm uv --version, and rerun the installer.
2. GUI startup fails: use start_gui.ps1 and test uv run gui/launch.py --help from the repository root.
3. Port conflict: pass --port or use the URL printed after automatic port selection.
4. Missing local dependencies: reselect the route in Caption or Tools so the GUI can install the profile.
5. Hugging Face 403: accept the model terms and inject HF_TOKEN into the current runtime.
6. Out of memory: lower batch size, resolution, or concurrency; enable CPU/offload options where available.
7. Translation or Lance errors: validate the input path and run --normalize_only before full translation.

See docs/troubleshooting.en.md for the expanded checklist.

Upstream and citations

• Image2PSD / see-through integrates shitagaki-lab/see-through. The local module/see_through/ package is a workflow adaptation, not a file-for-file mirror.
• The vocal-midi path references openvpi/GAME. GAME does not currently publish an official BibTeX entry in its README, so the legacy repository-level citation is retained below; prefer a future upstream citation.
• Audio-to-MIDI transcription integrates muscriptor/muscriptor. Cite the official MuScriptor paper when this integration contributes to research results.

@article{lin2026seethrough,
  title={See-through: Single-image Layer Decomposition for Anime Characters},
  author={Lin, Jian and Li, Chengze and Qin, Haoyun and Chan, Kwun Wang and Jin, Yanghua and Liu, Hanyuan and Choy, Stephen Chun Wang and Liu, Xueting},
  journal={arXiv preprint arXiv:2602.03749},
  year={2026}
}

@InProceedings{ke2023repurposing,
  title={Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation},
  author={Bingxin Ke and Anton Obukhov and Shengyu Huang and Nando Metzger and Rodrigo Caye Daudt and Konrad Schindler},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2024}
}

@software{openvpi_game,
  title={GAME: Generative Adaptive MIDI Extractor},
  author={{OpenVPI}},
  url={https://github.com/openvpi/GAME}
}

@misc{rouard2026muscriptoropenmodelmultiinstrument,
  title={MuScriptor: An Open Model for Multi-Instrument Music Transcription},
  author={Simon Rouard and Michael Krause and Axel Roebel and Carl-Johann Simon-Gabriel and Alexandre Défossez},
  year={2026},
  eprint={2607.08168},
  archivePrefix={arXiv},
  primaryClass={cs.SD},
  url={https://arxiv.org/abs/2607.08168}
}

Follow each upstream repository's license and model terms. See CHANGELOG.md for release history.

License

The project uses the root LICENSE. Third-party directories and downloaded models remain subject to their own licenses and access terms.