anand-tripathi-git/local-llm-csharp

Local LLM in csharp

6 commits

updated Jul 4, 2026

See the code

README

Local LLM — C# Edition

Covers core local LLM concepts using native .NET libraries.

Technology Stack

Technology	Purpose
LLamaSharp 0.27	Runs GGUF models locally
Microsoft.Extensions.AI (`IChatClient`)	Standard AI abstraction layer
Microsoft.Extensions.AI	Chat history pattern / memory
Blazor Server	Chat UI

Prerequisites

.NET 9 SDK or later
NVIDIA GPU with CUDA 12 (Uses LLamaSharp.Backend.Cuda12 for CPU/GPU Hybrid mode)
The Llama 3.2 3B GGUF model (see below)

Step 1 — Download the Model

Download Llama-3.2-3B-Instruct-Q4_K_M.gguf from Hugging Face and place it in the models/ folder:

https://huggingface.co/hugging-quants/Llama-3.2-3B-Instruct-Q4_K_M-GGUF

The folder structure should look like:

local-llm-csharp/
└── models/
    └── Llama-3.2-3B-Instruct-Q4_K_M.gguf   ← place it here

You can also set a MODEL_PATH environment variable to point to any GGUF file elsewhere on disk.

Step 2 — Run the Console App (Phase 1)

dotnet run --project LocalLlm.Console

Pick a demo from the interactive menu:

#	Demo	Description
1	Simple Completion	Run one prompt, print result
2	Streaming Answer	Stream tokens to console as they arrive
3	Chat Prompt Engineering	Manually build system/user/response prompt format
4	Conversational Memory	Maintain history, inject into prompt
5	Agent Demo (MEAI)	Use IChatClient and session memory

Step 3 — Run the Blazor Chat UI (Phase 2)

dotnet run --project LocalLlm.Web

Open https://localhost:5001 in your browser for the chat interface.

CPU/GPU Hybrid Mode

This project is configured to use LLamaSharp.Backend.Cuda12 with gpuLayerCount: 35. Because the Llama-3.2-3B-Instruct-Q4_K_M model is small enough (~1.9 GB), this offloads 100% of the layers to your GTX VRAM.

If you decide to use a larger model that exceeds your 3GB of VRAM, you can lower the gpuLayerCount in Program.cs. This will act in a true hybrid mode, keeping some layers on your GPU and falling back to your CPU and system RAM for the rest!

CPU-Only Fallback

If you don't have an NVIDIA GPU, edit both .csproj files:

- <PackageReference Include="LLamaSharp.Backend.Cuda12" Version="0.27.0" />
+ <PackageReference Include="LLamaSharp.Backend.Cpu" Version="0.27.0" />

Then run dotnet restore and inference will use only the CPU.

Project Structure

local-llm-csharp/
├── LocalLlm.sln
├── README.md
├── models/                         ← Put your .gguf file here
├── LocalLlm.Console/               ← Console demos (Phase 1)
│   ├── Program.cs
│   ├── ModelLoader.cs
│   ├── LlamaSharpChatClient.cs     ← IChatClient bridge
│   └── Demos/
│       ├── SimpleCompletionDemo.cs
│       ├── StreamAnswerDemo.cs
│       ├── ChatPromptDemo.cs
│       ├── ConversationalMemoryDemo.cs
│       └── AgentDemo.cs
└── LocalLlm.Web/                   ← Blazor chat UI (Phase 2)
    ├── Program.cs
    ├── LlamaSharpChatClient.cs
    └── Components/Pages/Chat.razor

Similar repos

SciSharp/LLamaSharp

A C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.

3,810

ricardosmdemello/Mini-LLM-Local-C-Sharp

Network-Intelligence/LLamaSharp

elbruno/ElBruno.LocalLLMs

C# local LLM chat completions library using ONNX Runtime, compatible with Microsoft.Extensions.AI

muneeb-devp/Hearth.AI

Run any GGUF language model locally in a .NET 8/9 app with a **single line of registration**:

dranger003/llama.cpp-dotnet

Minimal C# bindings for llama.cpp + .NET core library with API host/client.

kkokosa/dotLLM

LLM inference engine written in .NET

524

Integral2u/SharpMind

SharpMind. A pure C# / .NET LLM engine — inference, training, and agent tooling in one solution.