GenAI-Caption-Pipeline

Reproductions of the image captioning / annotation pipelines described in industrial text-to-image technical reports.

Frontier text-to-image systems are increasingly bottlenecked not by the diffusion backbone but by what their training captions actually say. Every lab has converged on the same insight — a single flat sentence per image is not enough — yet each one solves it differently. The tech reports describe these pipelines in prose, and the code is almost never released.

This repository re-implements them, one folder per report, so the designs can be read, run and compared side by side.

Reproduced tech reports

Each folder has its own README with the pipeline description, what is and is not reproduced, and how to run it.

Why two very different designs

Both reports start from the same complaint — verbose VLM captions waste tokens and miss what matters — but they diverge on how to fix it, and on where in the training stack the captions are spent:

• ByteDance Seed goes vertical, and aims at post-training. One deeply structured record per image, with explicit slots for position, depth, occlusion and per-element photography. Information is added by filling more fields, and the field ladder makes "how much schema" a controlled variable. The annotations exist to fine-tune a diffuser on SP and to post-train a prompter (SFT → cold-start distillation → verifier-gated RFT), so the pipeline is allowed to be slow and expensive per image — it runs over a curated set, not the whole corpus.
• Meituan LongCat goes horizontal, and aims at pre-training. Four independent captions per image at different granularities, no shared schema. Robustness comes from format diversity — the model learns to accept both a keyword list and a dense photographic paragraph. These captions are consumed by pre-training across a ~1.2 B-image corpus, which forces the opposite engineering constraint: the captioner must be cheap enough to run on every image.

That difference in target stage explains almost everything else about the two designs, including the two orders of magnitude between their per-image costs below.

ByteDance Seed, Figure 8 — one image becomes one L10 record, then L9…L5 by masking field groups.

LongCat-Image, Figure 10 — one image becomes four independent captions at different granularities.

They are complementary, and neither is obviously right. Having both in one place is the point.

Status

The 100× gap is the design difference, not an implementation detail. ByteDance's SP needs two serial heavy VLM calls plus three specialist forward passes per image; LongCat's MGC is one batched VLM call that emits all four levels at once, which is what makes it affordable at pre-training scale.

Disclaimer

These are independent re-implementations written from the published reports, not official code, and not affiliated with or endorsed by ByteDance or Meituan. Prompts and hyperparameters were reconstructed from the papers and, where the papers were ambiguous or where a described choice did not survive empirical testing, deliberately changed — the folder READMEs record every such deviation. Numbers reported here are from our own runs and are not expected to reproduce the papers' benchmark results.

Figures are reproduced from the two arXiv papers, both of which are published under CC BY 4.0, with attribution in the captions.

Model weights pulled by these pipelines carry their own licenses. In particular Sapiens is CC-BY-NC-4.0 (non-commercial), while DepthAnything V2 and SAM 2.1 are Apache-2.0; see ByteDance-ImageGen/README.md.