Documentation

Helpers

Execute in the background with 10min (600s) timeout and pipe output to output.log

nohup timeout 600 python src/leolm.py > output.log 2>&1 &

View output

tail -f output.log

llama.cpp

Manually download & convert model to GGUF format

Based on Tutorial: How to convert HuggingFace model to GGUF format

Use already converted GGUF models

Remeber to adjust the model ID to your model

huggingface-cli download TheBloke/leo-hessianai-13B-GGUF leo-hessianai-13b.Q8_0.gguf --local-dir models/ --local-dir-use-symlinks False

Run model using llama.cpp

llama.cpp/main -m models/leo-hessianai-7b.Q4_K_M.gguf -n -2 -c 0 -ngl 33 -p "Das Wetter in Potsdam soll"

llama.cpp/main -ngl 41 -m models/leo-hessianai-13b-chat.Q8_0.gguf --color -c 0 --temp 0.7 --repeat_penalty 1.1 -n -1 -f prompts/long-doc-csv-export.txt

llama.cpp/main -m models/leo-hessianai-7b-chat.Q8_0.gguf -n -1 -c 0 -ngl 33 -i -r "Benutzer:" -f prompts/rede-mit-bob.txt

Parameters

• -m path to the model
• -n number of tokens to generate, -2 means generate until context is full
• -c context size, 0 means load from model (LeoLM uses 8192)
• -ngl n GPU layers, offload tensor layers of gguf model to GPU
  • 0 runs on CPU only
  • 33 is max for 7b model
  • 41 is max for 13b-chat model
  • between 30 and 40 for 70b model
• -p prompt
• --batch-size can be used, however it does not seem to make a difference in inference time, LeoLM default is 512 (I believe)

Run model using HTTP server (powered by llama.cpp)

Full documentation

llama.cpp/server -ngl 41 -m models/discolm_german_7b_v1.Q8_0.gguf -c 0 -n -1

ollama

Running without devcontainer

scripts/run-in-docker.sh -g 0 "scripts/start-ollama.sh"

-g 0 specifies the GPU to use. Use -g 0,1,... to specify multiple GPUs. By default, the script will run mixtral:8x7b-instruct-v0.1-q3_K_M. To run a different model, modify scripts/start-ollama.sh accordingly. Port 11434 is also being passed through to the host in order to access the API.

Running inside devcontainer or bash shell

Starting ollama server

OLLAMA_MODELS=/workspaces/ml-for-smb/models/ ollama serve

Interact with model

• chat mode

ollama run mixtral:8x7b-instruct-v0.1-q3_K_M

• with promp

ollama run mixtral:8x7b-instruct-v0.1-q3_K_M "$(cat prompts/long-doc-csv-export.txt)"

• using API (default port: 11434)

curl http://localhost:11434/api/generate -d '{
  "model": "mixtral:8x7b-instruct-v0.1-q3_K_M",
  "prompt": "Nenne mir den Hauptcharakter des Films Titanic.",
  "stream": false
}'

Benchmarking

API calls return eval_count and eval_duration in nanoseconds

$$ tokens_per_sec = \frac{eval_count}{eval_duration \cdot 10^{-9}} $$ Sample benchmarking script can be found in src/ollama-benchmark.py

Comparison of german language models

In order to gauge the performancem, quality and obedience to output format requests of the models, I attempted to benchmark both models with the same custom prompts. The prompts used are

• long-doc-json-export,
• long-doc-csv-export,
• long-doc-yesno-export.

Each prompt was run five times for each model and the median tokens per second was taken. The results are shown in the table below.

Mixtral 8x7b instruct v0.1 Q3_K_M

LeoLM 13b-chat Q8_0

DiscoLM 7b Q8_0

MiQu 70b Q4_K_M

Discussion

Leo tends to put other things before the result such as "assistant" or "JSON Output:". This makes automated parsing of the output difficult. Mixtral performs better at this task, especially for complex formats and will only output the requested information. However, the quality of the answers still varies greatly. Mixtral also sometimes struggles with the yes/no format, as it will sometimes provide additional context to the answer such as "YES. The main ...". While both models are able to run relatively fast on the GPU with 24GB of VRAM, the quality of the answers is often suboptimal.

While Disco is able to churn out about 20 tokens per second more than the other models, the quality and obediance are lacking. The model will often produce python code that attempts to generate the requested output format, instead of just outputting it itself. Other times it simply comes up with data that is entirely unrelated to the context. Each attempt at a more complex format was not usable. Simple yes/no prompts worked relatively well.

MiQu is by far biggest and therefor the slowest model. Using the 24GB of VRAM I was only able to offload 20 layers to the GPU, resulting in an inference time of less than 1 token per second. The quality of the answers is also very low, as the model struggles with the output format.

LLM on mobile workstation

Machine specs

• CPU: Intel Xeon W-11955M ProzOcta-Coreessor 8x2,60 GHz
• GPU: NVIDIA RTX A5000 with 16GB VRAM
• RAM: 32GB DDR4 SDRAM

Performance of Mixtral 8x7b instruct v0.1 for different quantizations

Pitfalls

llama.cpp

• enabling GPU support
  • llama.cpp has to be compiled with CUDA support
  • might have to update the Makefile to correct GPU architecture, as arch=native is unsupported in older Ubuntu versions
    • run nvcc --list-gpu-arch inside container to see supported architectures, compute_75 worked for me
    • in Makefile under ifdef LLAMA_CUBLAS change from

      NVCCFLAGS = --forward-unknown-to-host-compiler -use_fast_math
      

      to

      NVCCFLAGS = --forward-unknown-to-host-compiler -use_fast_math -arch=compute_75
      

    • also under ifdef CUDA_DOCKER_ARCH inside the else block change from

      NVCCFLAGS += -arch=native
      

      to

      NVCCFLAGS += -arch=compute_75
      

  • make LLAMA_CUBLAS=1
    

• -ngl|--n-gpu-layers has to be set, in order to offload to layers to the GPU (see parameters)

TODO

• run 70b model
• context size test
  • use long document as context
  • ask for details throughout the text (e.g. "what are the names of the characters?")
  • test out different output format in the prompt, such as json or csv
• configuration matrix with test results
  • comments on the results, indicating success of output format, context size, etc.
• try out DiscoLM
• run ollama without devcontainer
• promptengineering

Long document comprehension and specific output format

• all non-chat models are basically useless, don't answer questions and proceed with writing another article
• chat models are better, able to deliver consistently formatted answers
• however, quality of answers is still very low
• 13b-chat works reasonably well
• 70b-chat-Q4_K_M just produces "end of text" immediately
• 70b-chat with higher quant is TBD