Vision as Unified Multimodal Generation

English | 简体中文

🌟 Overview

SenseNova-Vision is a unified multimodal model for computer vision. It reformulates heterogeneous visual perception tasks as text generation, image generation, or mixed text-image generation, instead of relying on task-specific heads, decoders, or loss functions for each individual task. The model supports structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry within a shared instruction-following interface.

🚀 Model Description

SenseNova-Vision rethinks computer vision as unified multimodal generation. Traditional computer-vision systems usually attach task-specific prediction heads for detection, segmentation, depth, surface normal, or 3D geometry. SenseNova-Vision instead expresses these heterogeneous tasks through the native input-output spaces of a unified multimodal model.

Natural-language instructions and optional visual prompts specify the target task, regions, views, output schema, and decoding convention. The model then generates different target formats depending on the task:

This formulation allows a single model to cover structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry while keeping outputs decodable for standard benchmarks.

🌐 Key Features

• Unified vision-task formulation: Heterogeneous computer-vision tasks are cast into the native text, image, and mixed generation spaces of a unified multimodal model.
• No task-specific heads: The model does not rely on separate detection, segmentation, depth, normal, or geometry heads.
• Decodable outputs: Generated text and images can be converted back into benchmark-compatible boxes, points, OCR strings, masks, depth maps, normal maps, point maps, and camera records.
• Broad task coverage: The same model handles structured visual understanding, segmentation, dense geometry, and multi-view visual geometry.
• Instruction-defined task variants: Natural-language instructions enable flexible task definitions beyond fixed benchmark schemas.

🛠️ How to Use

Please use the official inference code from the SenseNova-Vision GitHub repository:

git clone https://github.com/OpenSenseNova/SenseNova-Vision.git
cd SenseNova-Vision

Environment Setup

Create the environment from the repository root:

bash setup.sh sensenova-vision
conda activate sensenova-vision

Download the Model

You can download the model weights from Hugging Face with huggingface_hub:

from huggingface_hub import snapshot_download

model_path = snapshot_download("sensenova/SenseNova-Vision-7B-MoT")
print(model_path)

The printed model_path points to the local checkpoint directory and can be used as the model path for inference.

Run the Curated Example

We provide a curated example to quickly verify the environment and model setup:

bash scripts/run_sensenova_vision.sh example

Run One Inference Request

You can also run a single inference request with the official wrapper.
For example, the following command performs binary segmentation for the target category "person":

bash scripts/run_sensenova_vision.sh inference \
  binary_seg \
  "person" \
  examples/images/2.jpg

Launch the Web Demo

You can launch the Gradio web demo using the official wrapper provided in the repository.
The wrapper will print the local URL before starting Gradio. Open the printed URL in your browser to interact with the model.

For more details, supported tasks, and additional examples, please refer to the official GitHub repository:

https://github.com/OpenSenseNova/SenseNova-Vision

🏗️ Key Contributions

• 🔗 We introduce a unified multimodal generation formulation that casts heterogeneous computer vision tasks into the native input-output spaces of UMMs.
• 🧩 We construct the SenseNova-Vision Corpus, a large-scale computer-vision instruction-response corpus with decodable text, image, and mixed text-image targets.
• ✨ We train SenseNova-Vision and show strong results across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry, while supporting language-defined task variants beyond fixed benchmark schemas.

🏆 Benchmark Results

SenseNova-Vision is evaluated across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. All tasks are formulated with natural-language instructions: textual outputs are parsed into benchmark-specific structures such as boxes, points, recognized text, keypoints, and camera parameters, while image outputs are decoded into masks, depth maps, normal maps, or 3D point maps.

Structured Visual Understanding

Structured visual understanding evaluates tasks whose outputs can be represented as structured textual predictions, including box- and point-based localization, referring detection, OCR localization, GUI grounding, and keypoint localization.

Dense Geometric Prediction

Dense geometric prediction evaluates pixel-aligned geometric outputs, including monocular depth estimation and surface normal estimation.

Segmentation

Segmentation evaluates mask prediction under semantic, referring, reasoning, grounded, and interactive guidance.

Multi-View Visual Geometry

Multi-view visual geometry evaluates geometric prediction from multiple input images, including multi-view point map reconstruction and camera pose estimation.

Comparison with Generalist Vision Models

We further compare SenseNova-Vision with recent generalist visual models that span multiple visual capabilities.

General Multimodal Capability

SenseNova-Vision largely maintains general multimodal capability while being adapted to visual perception tasks.

🖼️ Qualitative Examples

📚 Training Data

SenseNova-Vision is trained on the SenseNova-Vision-Corpus-50M, a large-scale computer-vision instruction-response corpus. The corpus converts heterogeneous annotations into a shared schema with visual inputs, natural-language instructions, and decodable targets represented as text, image, or mixed text-image responses.

The corpus covers four task families:

⚠️ Limitations

• Not a specialist model for every task: Although SenseNova-Vision covers many tasks, task-specific models may still outperform it on certain specialized benchmarks.
• Output parsing is task-dependent: Textual outputs require task-specific parsers, and image outputs require decoding rules consistent with the training protocol.
• Metric accuracy is not guaranteed: Dense depth, normal, point-map, and camera-pose predictions should be validated carefully before downstream use.
• Prompt sensitivity: As an instruction-following model, performance can vary with prompt wording, output schema, and visual prompt style.
• Dataset and benchmark bias: Model behavior reflects the distribution and annotation conventions of the training corpus.

🛡️ Ethical Considerations

SenseNova-Vision may generate incorrect localization, segmentation, depth, normal, or camera predictions. Users should avoid deploying the model in safety-critical settings without independent verification. When used for datasets involving people, faces, documents, medical scenes, surveillance imagery, or private environments, users are responsible for complying with applicable privacy, consent, and data-governance requirements.

✒️ Citation

If you find SenseNova-Vision useful, please cite the technical report:

@misc{han2026visionunifiedmultimodalgeneration,
      title={Vision as Unified Multimodal Generation}, 
      author={Xiaoyang Han and Jianhua Li and Kewang Deng and Zukai Chen and Xuanke Shi and Sihan Wang and Boxuan Li and Linyan Wang and Siyi Xie and Xin You and Jinsheng Quan and Zhongang Cai and Haiwen Diao and Ziwei Liu and Lei Yang and Dahua Lin and Quan Wang},
      year={2026},
      eprint={2607.06560},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.06560}, 
}

📜 License

The model weights are released under the CC BY-NC 4.0 license and are intended for non-commercial use only.

The source code in the official GitHub repository may follow a different license. Please refer to the repository license for code usage. Third-party datasets, tools, and assets are subject to their original licenses.

📮 Contact

For questions, issues, or collaboration requests, please use the official project repository or contact the authors through the release page once available.