Jan is an open-source offline-first AI assistant with 25K+ GitHub stars running LLMs locally without sending data externally. Features a ChatGPT-like interface with one-click model downloads from Hugging Face, conversation management, customizable prompts, and an OpenAI-compatible local API server. Supports GGUF models via llama.cpp with GPU acceleration on NVIDIA and Apple Silicon. Built with Electron for macOS, Windows, and Linux with full data privacy.
ChatGPT-like experience on local hardware. 25K+ stars, privacy and offline-first.
One-click model downloads, conversation management, model switching. GGUF via llama.cpp.
OpenAI-compatible local API. GPU acceleration on NVIDIA CUDA and Apple Metal.
Cross-platform Electron desktop app. All data stays local.
Free and open-source
macOS, Windows, Linux
Find which AI models actually run on your hardware in one command
llmfit is a Rust-based terminal tool that matches over 200 LLM models from 30+ providers against your exact hardware specs. The interactive TUI scores each model on fit, speed, VRAM usage, and context length, helping you avoid downloading models that won't run on your machine. It supports Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio backends.
Run open-source LLMs on your phone, fully offline and private
Google AI Edge Gallery is an open-source mobile app that lets you download and run large language models like Gemma directly on Android and iOS devices with zero cloud dependency. Built on MediaPipe and LiteRT, it features AI chat with reasoning mode, multimodal image analysis, real-time audio transcription, and autonomous agent skills—all running entirely on-device for complete privacy. A reference implementation for developers building offline-first AI experiences.
Container-native local AI model serving with Podman
RamaLama is an open-source tool that containerizes AI model inference using Podman or Docker, eliminating host system configuration complexity. It auto-detects GPUs (NVIDIA, AMD, Intel, Apple Silicon), pulls models from HuggingFace, Ollama, and OCI registries, and runs them in isolated rootless containers with read-only mounts and network isolation. Developed under the Containers project (Red Hat ecosystem), it brings familiar container workflows to local LLM serving.
Anthropic's frontier AI assistant
Anthropic's AI assistant known for strong reasoning, nuanced writing, and extended context up to 200K tokens. Available in Opus (most capable), Sonnet (balanced), and Haiku (fast) tiers. Features web search, deep research, file analysis, code execution, artifacts, and Projects for organized workflows. Claude Code provides terminal-based agentic coding. API supports tool use, batch processing, and prompt caching. Available via claude.ai, mobile apps, and developer API.
Official Python SDK for the xAI API
The xAI Python SDK is the official Python client for the xAI API, giving developers a direct way to build Grok-powered apps without relying on community proxies or unofficial wrappers. It supports synchronous and asynchronous Python clients for chat completions, streaming responses, function/tool calling, and multimodal workflows, making it a clean fit for backend services, agents, notebooks, and developer tools that need programmatic xAI access.
Wafer-scale inference at thousands of tokens per second
Cerebras Inference serves open-weight LLMs like Llama, Qwen, and GPT-OSS on wafer-scale CS-3 chips through an OpenAI-compatible API, benchmarking between 1,800 and 2,600 output tokens per second on Llama 3.1 8B and several hundred on 70B models. A free tier offers one million tokens per day with no credit card, while paid pay-per-token pricing starts at $0.04 per million tokens for the smaller Llama models.
One desktop app for every LLM — private, cross-platform, extensible
Chatbox is a cross-platform desktop AI client supporting OpenAI, Claude, Gemini, DeepSeek, and local models via Ollama. All chat data stays on-device, making it ideal for privacy-conscious developers. Features include document analysis, code assistance with syntax highlighting, image generation, web search, and a local knowledge base for private Q&A. Available on Windows, macOS, Linux, Android, iOS, and web.
ML inference platform for production AI models
Baseten is the inference platform for deploying AI models at scale with dedicated and pre-optimized model APIs and performance-optimized infrastructure. Specializes in image generation, transcription, text-to-speech, LLM serving, embeddings, and compound AI workloads. Delivers 75% latency reduction with 415ms cold starts and 3000+ concurrent scaling. Available as managed cloud or self-hosted, trusted by Cursor, Notion, Descript, and Sourcegraph for production inference.
Cross-platform on-device AI model runtime
Nexa SDK enables running frontier LLMs and multimodal models locally across PC, mobile, IoT, and wearables with automatic hardware acceleration for GPU, NPU, and CPU. It supports Qwen, Gemma, Llama, DeepSeek models with Python/C++ desktop SDKs, Android/iOS mobile SDKs, and Docker for edge deployment. Includes an OpenAI-compatible API server with chat and function calling support.
