This handles both UI and REST Calls

Text generation web UI

A Gradio web UI for Large Language Models.

Its goal is to become the AUTOMATIC1111/stable-diffusion-webui of text generation.

Features

• 3 interface modes: default (two columns), notebook, and chat
• Multiple model backends: transformers, llama.cpp, ExLlama, AutoGPTQ, GPTQ-for-LLaMa, ctransformers
• Dropdown menu for quickly switching between different models
• LoRA: load and unload LoRAs on the fly, train a new LoRA using QLoRA
• Precise instruction templates for chat mode, including Llama-2-chat, Alpaca, Vicuna, WizardLM, StableLM, and many others
• 4-bit, 8-bit, and CPU inference through the transformers library
• Use llama.cpp models with transformers samplers (llamacpp_HF loader)
• Multimodal pipelines, including LLaVA and MiniGPT-4
• Extensions framework
• Custom chat characters
• Very efficient text streaming
• Markdown output with LaTeX rendering, to use for instance with GALACTICA
• API, including endpoints for websocket streaming (see the examples)

To learn how to use the various features, check out the Documentation: https://github.com/oobabooga/text-generation-webui/tree/main/docs

Installation

One-click installers

Just download the zip above, extract it, and double-click on "start". The web UI and all its dependencies will be installed in the same folder.

• The source codes and more information can be found here: https://github.com/oobabooga/one-click-installers
• There is no need to run the installers as admin.
• Huge thanks to @jllllll, @ClayShoaf, and @xNul for their contributions to these installers.

Manual installation using Conda

Recommended if you have some experience with the command-line.

0. Install Conda

https://docs.conda.io/en/latest/miniconda.html

On Linux or WSL, it can be automatically installed with these two commands (source):

curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh"
bash Miniconda3.sh

1. Create a new conda environment

conda create -n textgen python=3.10.9
conda activate textgen

2. Install Pytorch

The up-to-date commands can be found here: https://pytorch.org/get-started/locally/.

2.1 Special instructions

• MacOS users: https://github.com/oobabooga/text-generation-webui/pull/393
• AMD users: https://rentry.org/eq3hg

3. Install the web UI

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
pip install -r requirements.txt

Note about older NVIDIA GPUs

bitsandbytes >= 0.39 may not work. In that case, to use --load-in-8bit, you may have to downgrade like this:

• Linux: pip install bitsandbytes==0.38.1
• Windows: pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl

Alternative: Docker

ln -s docker/{Dockerfile,docker-compose.yml,.dockerignore} .
cp docker/.env.example .env
# Edit .env and set TORCH_CUDA_ARCH_LIST based on your GPU model
docker compose up --build

• You need to have docker compose v2.17 or higher installed. See this guide for instructions.
• For additional docker files, check out this repository.

Updating the requirements

From time to time, the requirements.txt changes. To update, use these commands:

conda activate textgen
cd text-generation-webui
pip install -r requirements.txt --upgrade

Downloading models

Models should be placed in the text-generation-webui/models folder. They are usually downloaded from Hugging Face.

• Transformers or GPTQ models are made of several files and must be placed in a subfolder. Example:

text-generation-webui
├── models
│   ├── lmsys_vicuna-33b-v1.3
│   │   ├── config.json
│   │   ├── generation_config.json
│   │   ├── pytorch_model-00001-of-00007.bin
│   │   ├── pytorch_model-00002-of-00007.bin
│   │   ├── pytorch_model-00003-of-00007.bin
│   │   ├── pytorch_model-00004-of-00007.bin
│   │   ├── pytorch_model-00005-of-00007.bin
│   │   ├── pytorch_model-00006-of-00007.bin
│   │   ├── pytorch_model-00007-of-00007.bin
│   │   ├── pytorch_model.bin.index.json
│   │   ├── special_tokens_map.json
│   │   ├── tokenizer_config.json
│   │   └── tokenizer.model

In the "Model" tab of the UI, those models can be automatically downloaded from Hugging Face. You can also download them via the command-line with python download-model.py organization/model.

• GGML models are a single file and should be placed directly into models. Example:

text-generation-webui
├── models
│   ├── llama-13b.ggmlv3.q4_K_M.bin

Those models must be downloaded manually, as they are not currently supported by the automated downloader.

GPT-4chan

python download-model.py EleutherAI/gpt-j-6B --text-only

Starting the web UI

conda activate textgen
cd text-generation-webui
python server.py

Then browse to

http://localhost:7860/?__theme=dark

Optionally, you can use the following command-line flags:

Basic settings

Model loader

Accelerate/transformers

Accelerate 4-bit

⚠️ Requires minimum compute of 7.0 on Windows at the moment.

GGML (for llama.cpp and ctransformers)

llama.cpp

ctransformers

AutoGPTQ

ExLlama

GPTQ-for-LLaMa

DeepSpeed

RWKV

RoPE (for llama.cpp, ExLlama, and transformers)

Gradio

API

Multimodal

Presets

Inference settings presets can be created under presets/ as yaml files. These files are detected automatically at startup.

The presets that are included by default are the result of a contest that received 7215 votes. More details can be found here.

Contributing

If you would like to contribute to the project, check out the Contributing guidelines.

Community

• Subreddit: https://www.reddit.com/r/oobabooga/
• Discord: https://discord.gg/jwZCF2dPQN