🤗 [LongAlign Dataset] • 💻 [Github Repo] • 📃 [LongAlign Paper]
LongAlign is the first full recipe for LLM alignment on long context. We propose the LongAlign-10k dataset, containing 10,000 long instruction data of 8k-64k in length. We investigate on trianing strategies, namely packing (with loss weighting) and sorted batching, which are all implemented in our code. For real-world long context evaluation, we introduce LongBench-Chat that evaluate the instruction-following capability on queries of 10k-100k length.
We open-sourced the following list of models:
| Model | Huggingface Repo | Description |
|---|---|---|
| LongAlign-6B-64k-base | 🤗 Huggingface Repo | ChatGLM3-6B with an extended 64k context window |
| LongAlign-6B-64k | 🤗 Huggingface Repo | Chat model by LongAlign training on LongAlign-6B-64k-base |
| LongAlign-7B-64k-base | 🤗 Huggingface Repo | Llama-2-7B with an extended 64k context window |
| LongAlign-7B-64k | 🤗 Huggingface Repo | Chat model by LongAlign training on LongAlign-7B-64k-base |
| LongAlign-13B-64k-base | 🤗 Huggingface Repo | Llama-2-13B with an extended 64k context window |
| LongAlign-13B-64k | 🤗 Huggingface Repo | Chat model by LongAlign training on LongAlign-13B-64k-base |
| ChatGLM3-6B-128k | 🤗 Huggingface Repo | ChatGLM3-6B with a 128k context window |
🤗 [LongAlign Dataset] • 💻 [Github Repo] • 📃 [LongAlign Paper]
LongAlign is the first full recipe for LLM alignment on long context. We propose the LongAlign-10k dataset, containing 10,000 long instruction data of 8k-64k in length. We investigate on trianing strategies, namely packing (with loss weighting) and sorted batching, which are all implemented in our code. For real-world long context evaluation, we introduce LongBench-Chat that evaluate the instruction-following capability on queries of 10k-100k length.
We open-sourced the following list of models:
| Model | Huggingface Repo | Description |
|---|---|---|
| LongAlign-6B-64k-base | 🤗 Huggingface Repo | ChatGLM3-6B with an extended 64k context window |
| LongAlign-6B-64k | 🤗 Huggingface Repo | Chat model by LongAlign training on LongAlign-6B-64k-base |
| LongAlign-7B-64k-base | 🤗 Huggingface Repo | Llama-2-7B with an extended 64k context window |
| LongAlign-7B-64k | 🤗 Huggingface Repo | Chat model by LongAlign training on LongAlign-7B-64k-base |
| LongAlign-13B-64k-base | 🤗 Huggingface Repo | Llama-2-13B with an extended 64k context window |
| LongAlign-13B-64k | 🤗 Huggingface Repo | Chat model by LongAlign training on LongAlign-13B-64k-base |
| ChatGLM3-6B-128k | 🤗 Huggingface Repo | ChatGLM3-6B with a 128k context window |
