Deployment & Inference ★ 87.0k

llama.cpp

CPU-first inference in plain C/C++ — runs where a GPU cannot go.

Visit site →

llama.cpp is the engine behind most CPU and edge LLM deployments, with GGUF quantisation that shrinks models enough to run on modest hardware. Expect lower throughput than a GPU stack, but far fewer deployment constraints.

Best for: CPU-only, edge and embedded inference

Deploy: Self-hostable

#Inference #Quantization #On-Premise #Open Source

Related tools

Deployment & Inference ★ 118.0k

Ollama

One command to run an LLM on a laptop — the fastest way to get a model in front of a customer.

Local demos, air-gapped pilots, developer machines Read more →

Deployment & Inference

TensorRT-LLM

The last few percent of GPU throughput, paid for in build complexity and hardware lock-in.

High-volume serving of a stable model set on fixed NVIDIA hardware Read more →

Deployment & Inference ★ 152.3k

Open WebUI

The front end you hand to users when the model layer is already done.

Internal chat deployments on top of self-hosted models, with real user accounts Read more →

Deployment & Inference ★ 62.0k

vLLM

High-throughput LLM inference engine with PagedAttention — the default choice when you need to squeeze a GPU.

Self-hosted production inference on NVIDIA/AMD GPUs Read more →