Python-based solution for running a small local LLM leveraging CPU.
Python
0
5 commits
updated Nov 26, 2024
Version 1.0
CPU-optimized version of Local LLM Runner, designed for systems without GPU acceleration. This Python-based tool enables local execution of Large Language Models (LLMs) with CPU threading optimizations. It was developed for the Strategic Exercise Information and Research unit in Likes Institute, at JAMK University of Applied Sciences. The tool provides a simple command-line interface for interacting with various GGUF-format language models, with specific optimization for the OpenHermes 2.5 model.
git clone https://github.com/juhanimerilehto/local-llm-runner-cpu.git
cd local-llm-runner-cpu
python -m venv llm-env
source llm-env/bin/activate # For Windows: llm-env\Scripts\activate
pip install -r requirements.txt
Basic usage:
python llm_runner.py
With custom thread count and batch size:
python llm_runner.py --threads 4 --batch_size 16
--threads: Number of CPU threads (default: automatic based on CPU cores)--batch_size: Processing batch size (default: 8)--temperature: Response randomness (default: 0.7)--model_url: HuggingFace model URL--model_file: Model filename- Currently set default threads to 4, optimal for tested CPU
- Modified thread calculation to leave 2 cores free for system tasks
- Runned with: python llm_runner.py --batch_size 12
local-llm-runner-cpu/
├── assets/
│ └── likes_str_logo.png
├── llm_runner_cpu.py
├── requirements.txt
└── README.md
This project is licensed for free use under the condition that proper credit is given to Juhani Merilehto (@juhanimerilehto) and JAMK Likes institute. You are free to use, modify, and distribute this project, provided that you mention the original author and institution and do not hold them liable for any consequences arising from the use of the software.
Python-based solution for running a small local LLM leveraging CPU.
Python
0
5 commits
updated Nov 26, 2024
Version 1.0
CPU-optimized version of Local LLM Runner, designed for systems without GPU acceleration. This Python-based tool enables local execution of Large Language Models (LLMs) with CPU threading optimizations. It was developed for the Strategic Exercise Information and Research unit in Likes Institute, at JAMK University of Applied Sciences. The tool provides a simple command-line interface for interacting with various GGUF-format language models, with specific optimization for the OpenHermes 2.5 model.
git clone https://github.com/juhanimerilehto/local-llm-runner-cpu.git
cd local-llm-runner-cpu
python -m venv llm-env
source llm-env/bin/activate # For Windows: llm-env\Scripts\activate
pip install -r requirements.txt
Basic usage:
python llm_runner.py
With custom thread count and batch size:
python llm_runner.py --threads 4 --batch_size 16
--threads: Number of CPU threads (default: automatic based on CPU cores)--batch_size: Processing batch size (default: 8)--temperature: Response randomness (default: 0.7)--model_url: HuggingFace model URL--model_file: Model filename- Currently set default threads to 4, optimal for tested CPU
- Modified thread calculation to leave 2 cores free for system tasks
- Runned with: python llm_runner.py --batch_size 12
local-llm-runner-cpu/
├── assets/
│ └── likes_str_logo.png
├── llm_runner_cpu.py
├── requirements.txt
└── README.md
This project is licensed for free use under the condition that proper credit is given to Juhani Merilehto (@juhanimerilehto) and JAMK Likes institute. You are free to use, modify, and distribute this project, provided that you mention the original author and institution and do not hold them liable for any consequences arising from the use of the software.