Shadowgraph
About
Sign in

geronimi73/Hana

A toy text-to-image model trained from scratch.

20

stars

306

commits

Jupyter Notebook

primary language

Jun 9, 2025

updated

README

Hana 🌸

My homemade text-to-image model - a frankensteined Sana (NVIDIA).

Latest specs:

This is a public log of my work in progress.

Samples

Alpha-44 samples, more

💪 Training runs - WIP, unfiltered list, mostly fails

Model      Text EncoderAETransformerDataset          ComputeModelCodeLoss    Samples                                
Beta-8SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (116M) num_layers=12, hidden_dim=768IN1k_256px recaptioned with md2+qwen2-vl+smolvlm2, AR 1:1 3:4 4:3
100 epochs, ??? steps,
BS 1024, single GPU
LR 5e-4 constant
10% label dropout		1xRTX 6000 pro ??? hrsModelCode??? loss
Beta-7SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (116M) num_layers=12, hidden_dim=768CC12M+IN21K 256px 17M subset
~4 epochs, 300k steps,
BS 256, single GPU
LR 5e-4 constant
10% label dropout		1xRTX 6000 Ada, 64 hrsModelCode1.01 loss clipscoregallery
Beta-6SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (116M) num_layers=12, hidden_dim=768IN1k_256px recaptioned with md2+qwen2-vl+smolvlm2, AR 1:1 3:4 4:3
70 epochs, 280k steps,
BS 320, single GPU
LR 5e-4 constant
20% label dropout		1xRTX 5090, 28 hrsModelCode1.03 loss clipscoregallery
Beta-5SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (116M) num_layers=12, hidden_dim=768IN1k_256px recaptioned with md2+qwen2-vl+smolvlm2, AR 1:1 3:4 4:3
40 epochs, 168k steps,
BS 320, single GPU
LR 5e-4 constant
10% label dropout		1xRTX 5090, 16 hrsModelCode1.03 loss clipscoregallery
Beta-4SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (116M) num_layers=12, hidden_dim=768IN1k_256px recaptioned with md2+qwen2-vl+smolvlm2, AR 1:1 3:4 4:3
65 epochs, 250k steps,
BS 320, single GPU
LR 5e-4 constant
no label dropout		1xRTX 5090, 25 hrsModelCode1.02 loss clipscoregallery
Alpha-45SmolLM2-360Mdc-ae-f32c32-sana-1.0Alpha-44PD12M 256px
200k steps,
BS 256 x4, grad_checkpointing
LR 4e-4 with linear decay to 1e-4 over 50k steps
10% dropout
CFG		4x3090 250hrsModelCode0.89 loss clipscoregallery output
Alpha-44SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (590M) num_layers=28, hidden_dim=1152IN1k_256px, captions by md2+qwen2-vl+smolvlm2, all aspects
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 4e-4 with linear decay to 1e-4 over 50k steps
10% dropout,
CFG		4x3090 150hrsModelCode0.992 loss_eval clipscoregallery output
Alpha-42SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (590M) num_layers=28, hidden_dim=1152IN1k_256px, captions by md2+qwen2-vl+smolvlm2, bfl16, AR 1:1 3:4 4:3 2 runs, model broke after 45 epochs, restarted at epoch 25 with lower LR
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 5e-4 + 1e-4 after restart,
10% dropout is back		4x3090 50hrs + 85 hrsModelCoderun1 run2 loss_eval clipscoregallery_small output
Alpha-39SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_256px, captions by md2+qwen2-vl+smolvlm2, bfl16, AR 1:1 3:4 4:3
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 32hrsModelCode0.99 loss_eval clipscoregallery
Alpha-38SigLIP2dc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_256px, captions by md2+qwen2-vl+smolvlm2, bfl16, AR 1:1 3:4 4:3
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 24hrsModelCode0.99 loss_eval clipscoregallery output_half
Alpha-37ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_256px, captions by md2+qwen2-vl+smolvlm2, bfl16, AR 1:1 3:4 4:3
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 24hrsModelCode0.98 loss_eval clipscoregallery_eval output_half
Alpha-35ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_256px recaptioned with md2, bfl16, AR 1:1 3:4 4:3
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 34hrsModelCode0.97 loss_evalgallery_eval output_half
Alpha-34ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel, wide variant (205M) num_layers=12, hidden_dim=1024IN1k_256px, no aug., +aspect ratios 1:1 3:4 4:3
88 epochs,
140k steps,
BS 192 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 31hrstrashedCode1.00 lossgallery
Alpha-33ModernBERT-largeKBlueLeaf/EQ-SDXL-VAESanaTransformer2DModel (116.0M) num_layers=12, hidden_dim=768imagenet1k_eqsdxlvae_latents_withShape
4.4 epochs,
16k steps,
BS 80 x4,
1000 timesteps,
LR 3e-4,
10% dropout,
CFG		4x3090 3.4hrstrashedCode0.72 loss clipscoregallery
Alpha-32ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_256px, no aug., +aspect ratios 1:1 3:4 4:3
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 24hrsModelCode0.98 W B Chart 3_2_2025, 6_29_56 PM still undertrained I guess loss_evalgallery output_half
Alpha-31ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_128px, no aug., +aspect ratios 1:1 3:4 4:3
100 epochs,
38.6k steps,
BS 832 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 12hrsModelCode1.14 loss not 100% sure if a31>30gallery gallery-2
Alpha-30ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_128px +4 augmentations
102 epochs,
32k steps,
BS 1024 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 16hrsModelCode1.14 lossgallery
Alpha-29ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_96px +4 augmentations
103 epochs,
32k steps,
BS 1024 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 12.5hrsModelCode1.21 loss less overfitting with augmentationsgallery
Alpha-27ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (116.08M) num_layers=12, hidden_dim=768IN1k_96px
143 epochs,
BS 1024 x4,
1000 timesteps,
LR 5e-4		4x3090 17hrsnone, run killed, overfittingCode1.14 lossgallery
Alpha-15ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7CIFAR10 128px
800 epochs,
BS 512,
1000 timesteps,
LR 5e-4		3x3090 8.5hrsforgot to saveCode0.78 losseval_images
Alpha-14ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7CIFAR10 128px
1000 epochs,
BS 384,
1000 timesteps,
LR 5e-4, scalingf fix		1x3090, 23hrsModelCode0.78 W B Chart 2_6_2025, 7_15_45 AMgallery
Alpha-13ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7CIFAR10 128px
1000 epochs,
BS 384,
1000 timesteps,
LR 5e-4		1x3090, 23hrstrashedCode1.9 W B Chart 2_4_2025, 9_11_30 PMmedia_images_images_eval_127_4ac141f8512cc0ee4592
Alpha-22ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (116.08M) num_layers=12, hidden_dim=768CIFAR10-augmented 64px
500 epochs,
BS 1024 * 4,
1000 timesteps,
LR 5e-4		4x3090, 2hrsModelCode1.07 W B Chart 2_17_2025, 9_57_35 PMgallery_ov copy 2 media_images_images_eval_650_90546f10f12bce35da68
Alpha-12ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7CIFAR10 64px
500 epochs,
BS 896,
1000 timesteps,
LR 5e-4		1x3090, 8hrsModelCode1.27 W B Chart 2_3_2025, 6_53_37 AMgallery ["airplane", "automobile", "bird", "cat", "deer", "dog", "frog", "horse", "ship", "truck"]
Alpha-11ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7CIFAR10 64px
500 epochs,
BS 896,
1000 timesteps,
LR 5e-4		1x3090, 12hrsModelCode1.25 W B Chart 2_2_2025, 9_08_09 AMmedia_images_images_eval_2919_7c7a3df35d986984f8b1 (1) ["airplane", "automobile", "bird", "cat", "deer", "dog", "frog", "horse", "ship", "truck"]
Alpha-20ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (17.8M) num_layers=7, hidden_dim=384Fashion MNIST
20 epochs,
1170 steps,
BS 1024,
1000 timesteps (logit normal),
LR 5e-4		1x3090, 13minsgoneCode0.61 lnormal 0.71 uniform 0.61 beta high 0.82 beta low lossnormal notes
Alpha-17ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (17.8M) num_layers=7, hidden_dim=384Fashion MNIST
20 epochs,
600 steps,
BS 2048,
1000 timesteps (uniform),
LR 5e-4		1x3090, 20minsModelCode0.85 loss clipscoregallery notes
Alpha-8dModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7Fashion MNIST
400 epochs,
BS 896,
1000 timesteps		1x4090, 6.2hrsModelCode1.14 wandb loss_comparisonmedia_images_images_eval_1463_2941cfe11fbe584e6130 ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
Alpha-8cModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7Fashion MNIST
236 epochs,
BS 896,
40 timesteps		1x4090, 2.6hrstrashedCode1.33 wandbmedia_images_images_eval_858_39143ae078839e67f230 (1)
Alpha-8bModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7Fashion MNIST
250 epochs,
BS 896,
20 timesteps		1x4090, 2.6hrstrashedCode1.33 wandbmedia_images_images_eval_891_65d6f480d656633d9ff8
Alpha-8aModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7Fashion MNIST
250 epochs,
BS 896,
10 timesteps		1x4090, 2.6hrstrashedCode1.34 wandbmedia_images_images_eval_914_984ebb3d40cd269bf7f2
Alpha-10ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7, cross_attention_dim=1152MNIST
3 epochs,
LR 5e-4
BS 256, 10 TS (lognormal)		1x4090, 10'ModelCode1.069 loss_TS-lin-vs-lognmedia_images_images_eval_41_3599422a42f52634bd6b
Alpha-7ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7, cross_attention_dim=1152MNIST
3 epochs,
LR 5e-4,
BS 256, 10 TS		1x4090, 10'ModelCode0.99media_images_images_eval_41_f15cda102f6f1ed8ee3b
Alpha-9ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7, cross_attention_dim=1152Imagenet-1k
60 epochs,
BS 320,
LR 5e-4		1x4090, 83hrsFAILCode2.32 W B Chart 1_31_2025, 3_38_00 PMmedia_images_images_eval_13125_115627eb4d3d574732aa (1)
Alpha-5ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7, cross_attention_dim=1152Imagenet-1k
20 epochs,
BS 128		1x4090, 22hrsFAILCodediverging after e2 2.57media_images_images_eval_11065_a98b1c9fd31b90e95541
Alpha-4ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7, cross_attention_dim=1152MNIST
3 epochs,
BS 128		1x4090, 9'ModelCode1.050media_images_images_eval_76_61a68bef7c70a802ced5
Alpha-3ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7, cross_attention_dim=1152MNIST
85660 steps,=150 epochs,
BS 128		1x4090, 8 hrsModelCode0.833media_images_images_eval_670_f1a015427c67c3e5933e
Alpha-2Gemma2 2bdc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) 7 layers instead of 28MNIST
7940 steps,
BS 128		1x4090, 40'ModelCode0.933images_eval_406_016a491efda29b0ea833
Alpha-1Gemma2 2bdc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) 7 layers instead of 28MNIST
5 epochs,
LR 1e-4,
300k steps,
BS 1		1x4090, 4 hoursModelCode0.958images_eval_15565_b120cf6385fa11612684

🏅🏅Thank you - people, components, videos, articles

Contributors

geronimi73

306 commits

Similar repos

KBlueLeaf/HDM-xut-340M-anime

Model

139

KR-HappyFace/KoDALLE

🇰🇷 Text to Image in Korean

86

tahahah/Sana-fork

1

PixArt-alpha/PixArt-alpha

PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis

3,304

Efficient-Large-Model/Sana_1600M_512px_MultiLing

Model

16

brycegoh/pixeart

0

gschoeni/PixArt-alpha

0

Efficient-Large-Model/SANA1.5_1.6B_1024px

Model

6

Languages

Jupyter Notebook

99.5%

geronimi73/Hana

A toy text-to-image model trained from scratch.

20

stars

306

commits

Jupyter Notebook

primary language

Jun 9, 2025

updated

README

Hana 🌸

My homemade text-to-image model - a frankensteined Sana (NVIDIA).

Latest specs:

This is a public log of my work in progress.

Samples

Alpha-44 samples, more

💪 Training runs - WIP, unfiltered list, mostly fails

Model      Text EncoderAETransformerDataset          ComputeModelCodeLoss    Samples                                
Beta-8SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (116M) num_layers=12, hidden_dim=768IN1k_256px recaptioned with md2+qwen2-vl+smolvlm2, AR 1:1 3:4 4:3
100 epochs, ??? steps,
BS 1024, single GPU
LR 5e-4 constant
10% label dropout		1xRTX 6000 pro ??? hrsModelCode??? loss
Beta-7SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (116M) num_layers=12, hidden_dim=768CC12M+IN21K 256px 17M subset
~4 epochs, 300k steps,
BS 256, single GPU
LR 5e-4 constant
10% label dropout		1xRTX 6000 Ada, 64 hrsModelCode1.01 loss clipscoregallery
Beta-6SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (116M) num_layers=12, hidden_dim=768IN1k_256px recaptioned with md2+qwen2-vl+smolvlm2, AR 1:1 3:4 4:3
70 epochs, 280k steps,
BS 320, single GPU
LR 5e-4 constant
20% label dropout		1xRTX 5090, 28 hrsModelCode1.03 loss clipscoregallery
Beta-5SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (116M) num_layers=12, hidden_dim=768IN1k_256px recaptioned with md2+qwen2-vl+smolvlm2, AR 1:1 3:4 4:3
40 epochs, 168k steps,
BS 320, single GPU
LR 5e-4 constant
10% label dropout		1xRTX 5090, 16 hrsModelCode1.03 loss clipscoregallery
Beta-4SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (116M) num_layers=12, hidden_dim=768IN1k_256px recaptioned with md2+qwen2-vl+smolvlm2, AR 1:1 3:4 4:3
65 epochs, 250k steps,
BS 320, single GPU
LR 5e-4 constant
no label dropout		1xRTX 5090, 25 hrsModelCode1.02 loss clipscoregallery
Alpha-45SmolLM2-360Mdc-ae-f32c32-sana-1.0Alpha-44PD12M 256px
200k steps,
BS 256 x4, grad_checkpointing
LR 4e-4 with linear decay to 1e-4 over 50k steps
10% dropout
CFG		4x3090 250hrsModelCode0.89 loss clipscoregallery output
Alpha-44SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (590M) num_layers=28, hidden_dim=1152IN1k_256px, captions by md2+qwen2-vl+smolvlm2, all aspects
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 4e-4 with linear decay to 1e-4 over 50k steps
10% dropout,
CFG		4x3090 150hrsModelCode0.992 loss_eval clipscoregallery output
Alpha-42SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (590M) num_layers=28, hidden_dim=1152IN1k_256px, captions by md2+qwen2-vl+smolvlm2, bfl16, AR 1:1 3:4 4:3 2 runs, model broke after 45 epochs, restarted at epoch 25 with lower LR
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 5e-4 + 1e-4 after restart,
10% dropout is back		4x3090 50hrs + 85 hrsModelCoderun1 run2 loss_eval clipscoregallery_small output
Alpha-39SmolLM2-360Mdc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_256px, captions by md2+qwen2-vl+smolvlm2, bfl16, AR 1:1 3:4 4:3
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 32hrsModelCode0.99 loss_eval clipscoregallery
Alpha-38SigLIP2dc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_256px, captions by md2+qwen2-vl+smolvlm2, bfl16, AR 1:1 3:4 4:3
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 24hrsModelCode0.99 loss_eval clipscoregallery output_half
Alpha-37ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_256px, captions by md2+qwen2-vl+smolvlm2, bfl16, AR 1:1 3:4 4:3
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 24hrsModelCode0.98 loss_eval clipscoregallery_eval output_half
Alpha-35ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_256px recaptioned with md2, bfl16, AR 1:1 3:4 4:3
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 34hrsModelCode0.97 loss_evalgallery_eval output_half
Alpha-34ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel, wide variant (205M) num_layers=12, hidden_dim=1024IN1k_256px, no aug., +aspect ratios 1:1 3:4 4:3
88 epochs,
140k steps,
BS 192 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 31hrstrashedCode1.00 lossgallery
Alpha-33ModernBERT-largeKBlueLeaf/EQ-SDXL-VAESanaTransformer2DModel (116.0M) num_layers=12, hidden_dim=768imagenet1k_eqsdxlvae_latents_withShape
4.4 epochs,
16k steps,
BS 80 x4,
1000 timesteps,
LR 3e-4,
10% dropout,
CFG		4x3090 3.4hrstrashedCode0.72 loss clipscoregallery
Alpha-32ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_256px, no aug., +aspect ratios 1:1 3:4 4:3
100 epochs,
125k steps,
BS 256 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 24hrsModelCode0.98 W B Chart 3_2_2025, 6_29_56 PM still undertrained I guess loss_evalgallery output_half
Alpha-31ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_128px, no aug., +aspect ratios 1:1 3:4 4:3
100 epochs,
38.6k steps,
BS 832 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 12hrsModelCode1.14 loss not 100% sure if a31>30gallery gallery-2
Alpha-30ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_128px +4 augmentations
102 epochs,
32k steps,
BS 1024 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 16hrsModelCode1.14 lossgallery
Alpha-29ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (125.3M) num_layers=12, hidden_dim=768IN1k_96px +4 augmentations
103 epochs,
32k steps,
BS 1024 x4,
1000 timesteps,
LR 5e-4,
10% dropout,
CFG		4x3090 12.5hrsModelCode1.21 loss less overfitting with augmentationsgallery
Alpha-27ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (116.08M) num_layers=12, hidden_dim=768IN1k_96px
143 epochs,
BS 1024 x4,
1000 timesteps,
LR 5e-4		4x3090 17hrsnone, run killed, overfittingCode1.14 lossgallery
Alpha-15ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7CIFAR10 128px
800 epochs,
BS 512,
1000 timesteps,
LR 5e-4		3x3090 8.5hrsforgot to saveCode0.78 losseval_images
Alpha-14ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7CIFAR10 128px
1000 epochs,
BS 384,
1000 timesteps,
LR 5e-4, scalingf fix		1x3090, 23hrsModelCode0.78 W B Chart 2_6_2025, 7_15_45 AMgallery
Alpha-13ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7CIFAR10 128px
1000 epochs,
BS 384,
1000 timesteps,
LR 5e-4		1x3090, 23hrstrashedCode1.9 W B Chart 2_4_2025, 9_11_30 PMmedia_images_images_eval_127_4ac141f8512cc0ee4592
Alpha-22ModernBERT-largedc-ae-f32c32-sana-1.0SanaTransformer2DModel (116.08M) num_layers=12, hidden_dim=768CIFAR10-augmented 64px
500 epochs,
BS 1024 * 4,
1000 timesteps,
LR 5e-4		4x3090, 2hrsModelCode1.07 W B Chart 2_17_2025, 9_57_35 PMgallery_ov copy 2 media_images_images_eval_650_90546f10f12bce35da68
Alpha-12ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7CIFAR10 64px
500 epochs,
BS 896,
1000 timesteps,
LR 5e-4		1x3090, 8hrsModelCode1.27 W B Chart 2_3_2025, 6_53_37 AMgallery ["airplane", "automobile", "bird", "cat", "deer", "dog", "frog", "horse", "ship", "truck"]
Alpha-11ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7CIFAR10 64px
500 epochs,
BS 896,
1000 timesteps,
LR 5e-4		1x3090, 12hrsModelCode1.25 W B Chart 2_2_2025, 9_08_09 AMmedia_images_images_eval_2919_7c7a3df35d986984f8b1 (1) ["airplane", "automobile", "bird", "cat", "deer", "dog", "frog", "horse", "ship", "truck"]
Alpha-20ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (17.8M) num_layers=7, hidden_dim=384Fashion MNIST
20 epochs,
1170 steps,
BS 1024,
1000 timesteps (logit normal),
LR 5e-4		1x3090, 13minsgoneCode0.61 lnormal 0.71 uniform 0.61 beta high 0.82 beta low lossnormal notes
Alpha-17ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (17.8M) num_layers=7, hidden_dim=384Fashion MNIST
20 epochs,
600 steps,
BS 2048,
1000 timesteps (uniform),
LR 5e-4		1x3090, 20minsModelCode0.85 loss clipscoregallery notes
Alpha-8dModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7Fashion MNIST
400 epochs,
BS 896,
1000 timesteps		1x4090, 6.2hrsModelCode1.14 wandb loss_comparisonmedia_images_images_eval_1463_2941cfe11fbe584e6130 ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
Alpha-8cModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7Fashion MNIST
236 epochs,
BS 896,
40 timesteps		1x4090, 2.6hrstrashedCode1.33 wandbmedia_images_images_eval_858_39143ae078839e67f230 (1)
Alpha-8bModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7Fashion MNIST
250 epochs,
BS 896,
20 timesteps		1x4090, 2.6hrstrashedCode1.33 wandbmedia_images_images_eval_891_65d6f480d656633d9ff8
Alpha-8aModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7Fashion MNIST
250 epochs,
BS 896,
10 timesteps		1x4090, 2.6hrstrashedCode1.34 wandbmedia_images_images_eval_914_984ebb3d40cd269bf7f2
Alpha-10ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7, cross_attention_dim=1152MNIST
3 epochs,
LR 5e-4
BS 256, 10 TS (lognormal)		1x4090, 10'ModelCode1.069 loss_TS-lin-vs-lognmedia_images_images_eval_41_3599422a42f52634bd6b
Alpha-7ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7, cross_attention_dim=1152MNIST
3 epochs,
LR 5e-4,
BS 256, 10 TS		1x4090, 10'ModelCode0.99media_images_images_eval_41_f15cda102f6f1ed8ee3b
Alpha-9ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7, cross_attention_dim=1152Imagenet-1k
60 epochs,
BS 320,
LR 5e-4		1x4090, 83hrsFAILCode2.32 W B Chart 1_31_2025, 3_38_00 PMmedia_images_images_eval_13125_115627eb4d3d574732aa (1)
Alpha-5ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7, cross_attention_dim=1152Imagenet-1k
20 epochs,
BS 128		1x4090, 22hrsFAILCodediverging after e2 2.57media_images_images_eval_11065_a98b1c9fd31b90e95541
Alpha-4ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7, cross_attention_dim=1152MNIST
3 epochs,
BS 128		1x4090, 9'ModelCode1.050media_images_images_eval_76_61a68bef7c70a802ced5
Alpha-3ModernBERT-basedc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) num_layers=7, cross_attention_dim=1152MNIST
85660 steps,=150 epochs,
BS 128		1x4090, 8 hrsModelCode0.833media_images_images_eval_670_f1a015427c67c3e5933e
Alpha-2Gemma2 2bdc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) 7 layers instead of 28MNIST
7940 steps,
BS 128		1x4090, 40'ModelCode0.933images_eval_406_016a491efda29b0ea833
Alpha-1Gemma2 2bdc-ae-f32c32-sana-1.0SanaTransformer2DModel (158.18M) 7 layers instead of 28MNIST
5 epochs,
LR 1e-4,
300k steps,
BS 1		1x4090, 4 hoursModelCode0.958images_eval_15565_b120cf6385fa11612684

🏅🏅Thank you - people, components, videos, articles

Similar repos

KBlueLeaf/HDM-xut-340M-anime

Model

139

KR-HappyFace/KoDALLE

🇰🇷 Text to Image in Korean

86

tahahah/Sana-fork

1

PixArt-alpha/PixArt-alpha

PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis

3,304

Efficient-Large-Model/Sana_1600M_512px_MultiLing

Model

16

brycegoh/pixeart

0

gschoeni/PixArt-alpha

0

Efficient-Large-Model/SANA1.5_1.6B_1024px

Model

6

Contributors

geronimi73

306 commits

Languages

Jupyter Notebook

99.5%