World Modelings on MindSpore

Let's explore the world modeling potentials on MindSpore ;)

[!NOTE] The interactive 3D rendering UI below is better rendered in vscode.

News

Janus-Pro is supported!

MindSpore implementation for Janus-Pro training/inference is now released! Supporting both multimodal understanding and visual generation on Ascend NPU. Decoupling visual encoding generation/understanding-specific tasks surely bring Omni-capability. Details can be found here.

MVDream is supported!

MVDream is a diffusion model that is able to generate consistent multiview images from a given text prompt. It shows that learning from both 2D and 3D data, a multiview diffusion model can achieve the generalizability of 2D diffusion models and the consistency of 3D renderings. Details can be found here

These videos are rendered from the trained 3D implicit field in our MVDream model. Color meshes are extracted with the script MVDream-threestudio/extract_color_mesh.py.

InstantMesh is supported!

We support instantmesh for the 3D mesh generation using the multiview images extracted from the sv3d pipeline.

Using the multiview images input from 3D mesh extracted from the sv3d pipeline, we extracted 3D meshes as below. Please kindly find the input illustrated by following the link to the sv3d pipeline below.

The illustrations here are better viewed in viewers than with HTML support (e.g., the vscode built-in viewer).

Stable Video 3D is supported!

Output Multiview Images (21x576x576)

A camera-guided diffusion model that can generate the multiview snippet of a given image! Details can be found here.

More Inference Demos

Input	Output
aaa	aaa multiview
akun	akun multiview
anya	anya multiview
bag	bag multiview
groot	groot multiview
princess-large	princess-large multiview

Quick tour

To install MindONE v0.3.0, please install MindSpore 2.5.0 and run pip install mindone

Alternatively, to install the latest version from the master branch, please run.

git clone https://github.com/mindspore-lab/mindone.git
cd mindone
pip install -e .

We support state-of-the-art diffusion models for generating images, audio, and video. Let's get started using Stable Diffusion 3 as an example.

Hello MindSpore from Stable Diffusion 3!

import mindspore
from mindone.diffusers import StableDiffusion3Pipeline

pipe = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium-diffusers",
    mindspore_dtype=mindspore.float16,
)
prompt = "A cat holding a sign that says 'Hello MindSpore'"
image = pipe(prompt)[0][0]
image.save("sd3.png")

run hf diffusers on mindspore

• mindone diffusers is under active development, most tasks were tested with mindspore 2.5.0 on Ascend Atlas 800T A2 machines.
• compatibale with hf diffusers 0.32.2

supported models under mindone/examples

supported captioner