Update Log

• 2024.01.08: Initial Test version Release of Solar-Ko

Open-Solar-Ko ⭐🇰🇷

Solar-Ko represents an advanced iteration of the upstage/SOLAR-10.7B-v1.0 model, featuring an expanded vocabulary and the inclusion of a Korean corpus for enhanced pretraining.

Open-Solar-Ko exclusively utilizes publicly accessible Korean corpora, including sources such as AI Hub, Modu Corpus, 모두의 말뭉치, and Korean Wikipedia.

As training was conducted solely with publicly available corpora, this model is open for unrestricted use by everyone, adhering to the Apache2.0 open source License.

Model Details

Model Developers: Junbum Lee (Beomi)

Variations: Solar-Ko is available with one parameter sizes — 10B with Continual Pretrained version.

Input: The model accepts only text input.

Output: The model produces text output exclusively.

Model Architecture:

SOLAR-KO-10.7B is an auto-regressive language model that leverages an optimized transformer architecture derived from Llama-2.

Training Corpus

The model was trained using selected datasets from AIHub and Modu Corpus. Detailed information about the training datasets is available below:

• AI Hub: corpus/AI_HUB
  • Only the Training segment of the data was used.
  • The Validation and Test segments were deliberately excluded.
• Modu Corpus: corpus/MODU_CORPUS

The final JSONL dataset used to train this model is approximately 61GB in size.

Total token count: Approximately 15 billion tokens (*using the expanded tokenizer. With the original SOLAR tokenizer, >60 billion tokens.)

Vocab Expansion

Tokenizing "안녕하세요, 오늘은 날씨가 좋네요."

• SOLAR-10.7B: 26 tokens
• SOLAR-KO-10.7b: 8 tokens

Tokenizing "Meet 10.7B Solar: Elevating Performance with Upstage Depth UP Scaling!"

• SOLAR-10.7B: 22 tokens
• SOLAR-KO-10.7b: 22 tokens

LICENSE

Apache 2.0

Model Benchmark

LM Eval Harness - Korean (polyglot branch)

• Used EleutherAI's lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot

Citation

@misc {solar_ko_junbum_2023,
    author       = { {L. Junbum} },
    title        = { Solar-Ko-10.7b },
    year         = 2024,
    url          = { https://huggingface.co/beomi/SOLAR-KO-10.7B },
    publisher    = { Hugging Face }
}

Acknowledgements

• Training support was provided by the TPU Research Cloud program.
• The training corpus includes data from AI Hub, Modu Corpus, and Korean Wikipedia.