Python-based solution for running a small local LLM leveraging GPU.
Python
0
1 commits
updated Nov 21, 2024
Version 1.0
Local LLM Runner is a Python-based tool that enables local execution of Large Language Models (LLMs) with GPU acceleration. It was developed for the Strategic Exercise Information and Research unit in Likes Institute, at JAMK University of Applied Sciences. The tool provides a simple command-line interface for interacting with various GGUF-format language models, with specific optimization for the OpenHermes 2.5 model.
git clone https://github.com/juhanimerilehto/local-llm-runner.git
cd local-llm-runner
python -m venv llm-env
source llm-env/bin/activate # For Windows: llm-env\Scripts\activate
pip install ctransformers huggingface_hub
Run the script with default settings (OpenHermes 2.5):
python llm_runner.py
Or specify a different model:
python llm_runner.py --model_url "TheBloke/Mixtral-8x7B-v0.1-GGUF" --model_file "mixtral-8x7b-v0.1.Q4_K_M.gguf"
local-llm-runner/
├── assets/
│ └── jamklikes.png
├── llm_runner.py
├── requirements.txt
└── README.md
This project is licensed for free use under the condition that proper credit is given to Juhani Merilehto (@juhanimerilehto) and JAMK Likes institute. You are free to use, modify, and distribute this project, provided that you mention the original author and institution and do not hold them liable for any consequences arising from the use of the software.
Python-based solution for running a small local LLM leveraging GPU.
Python
0
1 commits
updated Nov 21, 2024
Version 1.0
Local LLM Runner is a Python-based tool that enables local execution of Large Language Models (LLMs) with GPU acceleration. It was developed for the Strategic Exercise Information and Research unit in Likes Institute, at JAMK University of Applied Sciences. The tool provides a simple command-line interface for interacting with various GGUF-format language models, with specific optimization for the OpenHermes 2.5 model.
git clone https://github.com/juhanimerilehto/local-llm-runner.git
cd local-llm-runner
python -m venv llm-env
source llm-env/bin/activate # For Windows: llm-env\Scripts\activate
pip install ctransformers huggingface_hub
Run the script with default settings (OpenHermes 2.5):
python llm_runner.py
Or specify a different model:
python llm_runner.py --model_url "TheBloke/Mixtral-8x7B-v0.1-GGUF" --model_file "mixtral-8x7b-v0.1.Q4_K_M.gguf"
local-llm-runner/
├── assets/
│ └── jamklikes.png
├── llm_runner.py
├── requirements.txt
└── README.md
This project is licensed for free use under the condition that proper credit is given to Juhani Merilehto (@juhanimerilehto) and JAMK Likes institute. You are free to use, modify, and distribute this project, provided that you mention the original author and institution and do not hold them liable for any consequences arising from the use of the software.