frkri/ModelRunner

Run LLM models locally with an HTTP API

Rust

197 commits

updated Nov 27, 2024

README

ModelRunner

[!NOTE] Please be aware that this is a learning project, so there may be some bugs and the code may not be the cleanest nor the fastest.

For alternatives see

vllm

mistral.rs

[!NOTE]
Some models require that you agree to their terms of service on Hugging Face before you can use them. This requires you to create a file containing your Hugging Face API key and to store it in your configured hugging face home directory See the Hugging Face CLI guide for more info.

Contributors

frkri

121 commits

dependabot[bot]

76 commits

Similar repos

imanchoys/hug-llm-api

Попытки подружиться с нейросетью

huggingface/ml-intern

Archived — ML Intern is no longer maintained. Continue with HuggingChat.

10,822

Hansimov/hf-llm-api

Huggingface LLM Inference API in OpenAI message format

AssetsArt/lokal

Run LLMs on your own machine — fast and simple.

Uminosachi/open-llm-webui

This repository contains a web application designed to execute relatively compact, locally-operated…

AaltoSciComp/llm-examples

This repository contains some LLM deployment examples

liltom-eth/llama2-webui

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use…

1,936

docker/model-runner

Docker Model Runner

651

Languages

Rust

98.8%

Dockerfile

1.2%

frkri/ModelRunner

Run LLM models locally with an HTTP API

Rust

197 commits

updated Nov 27, 2024

See the code

README

ModelRunner

[!NOTE] Please be aware that this is a learning project, so there may be some bugs and the code may not be the cleanest nor the fastest.

For alternatives see

vllm

mistral.rs

[!NOTE]
Some models require that you agree to their terms of service on Hugging Face before you can use them. This requires you to create a file containing your Hugging Face API key and to store it in your configured hugging face home directory See the Hugging Face CLI guide for more info.

Similar repos

imanchoys/hug-llm-api

Попытки подружиться с нейросетью

huggingface/ml-intern

Archived — ML Intern is no longer maintained. Continue with HuggingChat.

10,822

Hansimov/hf-llm-api

Huggingface LLM Inference API in OpenAI message format

AssetsArt/lokal

Run LLMs on your own machine — fast and simple.

Uminosachi/open-llm-webui

This repository contains a web application designed to execute relatively compact, locally-operated…

AaltoSciComp/llm-examples

This repository contains some LLM deployment examples

liltom-eth/llama2-webui

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use…

1,936

docker/model-runner

Docker Model Runner

651

Contributors

frkri

121 commits

dependabot[bot]

76 commits

Languages

Rust

98.8%

Dockerfile

1.2%