vllm

vllm-project/vllm is an open-source project by vllm-project: A high-throughput and memory-efficient inference and serving engine for LLMs. Indexed here so it can be found — not resold.

It is free. Get it from the upstream repository: https://github.com/vllm-project/vllm

From the project's own README (excerpt, reproduced for discovery under its Apache-2.0 license):

Easy, fast, and cheap LLM serving for everyone

Documentation Blog Paper Twitter/X User Forum Developer Slack

🔥 We have built a vLLM website to help you get started with vLLM. Please visit vllm.ai to learn more. For events, please visit vllm.ai/events to join us.

---

About

vLLM is a fast and easy-to-use library for LLM inference and serving.

Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has grown into one of the most active open-source AI projects built and maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors.

vLLM is fast with: • State-of-the-art serving throughput • Efficient management of attention key and value memory with PagedAttention • Continuous batching of incoming requests, chunked prefill, prefix caching • Fast and flexible model execution with piecewise and full CUDA/HIP graphs • Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and more • Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton • Optimized GEMM/MoE kernels for various precisions using CUTLASS, TRTLLM-GEN, CuTeDSL • Speculative decoding including n-gram, suffix, EAGLE, DFlash • Automatic kernel generation and graph-level transformations using torch.compile • Disaggregated prefill, decode, and encode

vLLM is flexible and easy to use with: • Seamless integration with popular Hugging Face models • High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more • Tensor, pipeline, data, expert, and context parallelism for distributed inference • Streaming outputs • Generation of structured outputs using xgrammar or guidance • Tool calling and reasoning…