LLM Compressor

LLM Compressor is an open-source library (maintained under the vllm-project org, with Red Hat / Neural Magic behind it) for shrinking large language models so they run cheaper and faster at inference time. It offers a broad set of quantization algorithms and transforms — weight, activation, KV-cache, and attention quantization across precisions like W8A8 (int8/fp8), W4A16, and microscale formats (NVFP4, MXFP4, MXFP8) — plus structural techniques such as REAP expert pruning for Mixture-of-Experts models. It integrates directly with Hugging Face models and saves in the compressed-tensors format that vLLM loads natively, and supports DDP and disk offloading to compress very large models. Vendor framing (labeled): the project cites checkpoints (e.g., large MoE models) compressed with ">70% size reduction while maintaining accuracy recovery" — these are producer-reported results on specific models/benchmarks; expected accuracy/size trade-offs must be validated per model and per task. Positioning note: this is a pre-deployment optimization pipeline that produces artifacts consumed by the vLLM engine — it is not the vLLM engine itself, nor vllm-production-stack. Best fit for teams self-hosting open-weight models who want to fit larger models on fewer/cheaper GPUs. Requires Python, a calibration dataset, and GPU capacity for the compression run.