KTransformers is an open-source framework for efficient inference and supervised fine-tuning of large language models through heterogeneous CPU-GPU computing. Its current project presents two first-class paths: kt-kernel inference and SFT through a LLaMA-Factory integration. The inference path uses CPU-optimized AMX, AVX512 or AVX2 kernels, quantization and heterogeneous expert placement so hot experts can remain on GPU while colder experts use CPU memory; it also documents a Python API and SGLang integration for serving. The SFT path targets very large Mixture-of-Experts models with CPU-GPU hybrid training, quantization and LoRA-style workflows on constrained accelerator memory. Model support, kernel availability and performance depend strongly on the processor instruction set, GPU, quantization, model architecture and release. Speedup and memory claims in the repository are maintainer benchmarks for named setups and should not be generalized without local testing. KTransformers is best for experienced teams optimizing large MoE inference or fine-tuning on mixed hardware; users seeking a turnkey managed endpoint or broad plug-and-play local runtime may prefer a higher-level platform.
KTransformers
Heterogeneous CPU-GPU inference and SFT for large MoE models
Open-source framework for running and fine-tuning large Mixture-of-Experts models with heterogeneous CPU-GPU execution, optimized kernels, limited VRAM and SGLang or LLaMA-Factory integrations.
About KTransformers
Pricing & Platform Specs
Pricing Summary
Free and open source under Apache-2.0. Users pay for their own CPU/GPU hardware infrastructure; no commercial SaaS subscription exists.
Supported Platforms
Python-based CPU-GPU heterogeneous framework with kt-kernel inference, AMX/AVX acceleration, quantized MoE execution, SGLang serving integration and LLaMA-Factory SFT workflows. Current verified release: v0.6.4.
Explore categories, tags & use cases
Categories
Tags
Alternatives
Fast serving framework for LLMs and vision models
SGLang is an open-source serving framework for large language and vision-language models, designed for low latency and high throughput. It features RadixAttention for automatic KV cache reuse, compressed finite state machines for fast structured output generation, continuous batching, and tensor parallelism. With over 25,000 GitHub stars, it supports models like LLaMA, Mistral, Qwen, and Gemma on NVIDIA and AMD GPUs.
High-throughput LLM serving engine
vLLM is an Apache-2.0 LLM inference and serving engine focused on high-throughput self-hosted model APIs. It combines PagedAttention, continuous batching, prefix caching, quantization options, OpenAI-compatible serving, structured outputs, metrics, Docker/Kubernetes deployment guidance and integrations with agent and LLM frameworks.
Open-source toolkit for quantizing, deploying, and serving LLMs and vision-language models
LMDeploy is an Apache-2.0 toolkit for self-hosting LLM and vision-language model inference with TurboMind and PyTorch engines. It combines continuous batching, blocked KV cache, tensor parallelism, AWQ and KV-cache quantization with OpenAI-compatible APIs, multi-GPU distribution, offline pipelines, and production metrics.
Run LLMs locally with one command
Tool for running large language models locally on your machine with a simple CLI interface. Download and run Llama 3, Mistral, Gemma, Phi, Code Llama, and dozens of other open-source models with a single command. Features model management, GPU acceleration (NVIDIA/AMD/Apple Silicon), OpenAI-compatible API server, Modelfile for customization, and multi-model switching. Ideal for offline AI development, privacy-sensitive use cases, and local testing. 120K+ GitHub stars.
NVIDIA's LLM inference optimization and acceleration library
TensorRT-LLM is NVIDIA's open-source library for optimizing LLM inference on NVIDIA GPUs. It provides kernel fusion, quantization (FP8, INT4, INT8), KV cache optimization, and in-flight batching to maximize throughput. Supports multi-GPU and multi-node setups with tensor and pipeline parallelism, and integrates with Triton Inference Server for production deployment of models like LLaMA, GPT, Mistral, and Qwen.
Community experience
Sources & verification
- Sources checked
- Content verified
Verification dates are editorial checks. Routine CMS saves and automatic updatedAt timestamps do not advance them.
FAQ
What is KTransformers?
Open-source framework for running and fine-tuning large Mixture-of-Experts models with heterogeneous CPU-GPU execution, optimized kernels, limited VRAM and SGLang or LLaMA-Factory integrations.
Is KTransformers free?
Yes — KTransformers is open source and free to use. Free and open source under Apache-2.0. Users pay for their own CPU/GPU hardware infrastructure; no commercial SaaS subscription exists.
Is KTransformers open source?
Yes — KTransformers is open source.
Is KTransformers still maintained?
Yes — KTransformers is active. Its listing was last verified on August 26, 2026.
What are the best KTransformers alternatives?
The first editor-selected KTransformers alternatives are SGLang, vLLM, LMDeploy, and more.
