llama.cpp

ggml-org/llama.cpp is an open-source project by ggml-org: LLM inference in C/C++. Indexed here so it can be found — not resold.

It is free. Get it from the upstream repository: https://github.com/ggml-org/llama.cpp

From the project's own README (excerpt, reproduced for discovery under its MIT license):

llama.cpp

LLM inference in C/C++

manifesto / ggml / ops / maintainer PRs%20sort%3Aupdated-desc) / dev branches / compile times / lib llama API / llama-server REST API

Quick start

A few options to get llama.cpp installed on your machine: • Visit https://llama.app and follow the instructions • Run with Docker - see our Docker documentation • Download pre-built binaries from the releases page • Build from source by cloning this repository - check out our build guide

Once installed:

VLM session with llama cli Built-in web UI against llama serve

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud. • Plain C/C++ implementation without any dependencies • Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks • AVX, AVX2, AVX512 and AMX support for x86 architectures • RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures • 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use • Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA) • Vulkan and SYCL backend support • CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Backend Target devices --- --- BLAS All BLIS All CANN Ascend NPU CUDA Nvidia GPU HIP AMD GPU [Hexagon [In Progress]](docs/backend/snapdragon/README.md) Snapdragon IBM zDNN IBM Z & LinuxONE MUSA Moore Threads GPU Metal Apple Silicon OpenCL Adreno GPU…