PurpleLlama

Validate and structure LLM outputs with composable Guards

Guardrails AI is an open-source Python and JavaScript framework for validating and structuring LLM outputs using composable Guards built from a Hub of pre-built validators. It handles structured data extraction with Pydantic models, content safety checks including toxicity, PII detection, competitor mentions, and bias filtering, plus automatic re-prompting when validation fails. The Guardrails Hub offers dozens of validators from regex matching to hallucination detection via LLM judges.

Programmable safety rails for LLM applications

NeMo Guardrails is NVIDIA's open-source toolkit for adding programmable safety rails to LLM applications. It supports five guardrail types — input, dialog, retrieval, execution, and output rails — covering content safety, jailbreak detection, topic control, PII masking, hallucination detection, and fact-checking. The toolkit uses Colang, a domain-specific language for defining conversational constraints, and integrates with OpenAI, Azure, Anthropic, HuggingFace, and LangChain/LangGraph.

NVIDIA's LLM vulnerability scanner and red-teaming tool

garak is NVIDIA's open-source LLM vulnerability scanner for red-teaming AI models and applications. Probes for prompt injection, data leakage, hallucination, toxicity, encoding-based attacks, and dozens of other vulnerability categories. Runs automated attack sequences against any LLM endpoint and generates detailed vulnerability reports. Features a modular probe/detector architecture that is extensible with custom attack patterns. Named after the Star Trek character known for deception.

Run and govern MCP servers across desktop, CLI and Kubernetes

Open-source MCP runtime and governance platform that runs servers in isolated containers, curates registries, enforces access policies, and operates gateways across desktop, CLI, and Kubernetes.

Craft image-scaling prompt-injection payloads to red-team multimodal AI systems

Open-source red-team toolkit from Trail of Bits that generates image-scaling attack payloads — images that look benign at full resolution but reveal a hidden prompt injection after a multimodal system downsamples them.

AI agent framework for offensive security and penetration testing

Alias Robotics' agent framework for building AI-driven offensive-security workflows — reconnaissance, exploitation, privilege escalation, and lateral movement — with multi-agent handoffs and human-in-the-loop control. Source-available, but the core is licensed for non-commercial research use only.

AI-first security scanner for LLM, agent, MCP, and RAG codebases

MEDUSA is an AGPL-3.0 AI-first security scanner from Pantheon Security that checks AI and machine-learning applications, LLM agents, MCP workflows, RAG pipelines, repository-poisoning risks, secrets, and agent-specific compromise patterns.

Open-source diagnostic for AI operational misalignment

iFixAi is an Apache-2.0 diagnostic tool for scoring AI agents and models against operational-misalignment risks such as hallucination, manipulation, sabotage, sandbagging, and oversight evasion.

UK AI Security Institute framework for LLM safety evaluations

Inspect AI is an MIT-licensed framework from the UK AI Security Institute for running large language model evaluations, including tool use, multi-turn dialogue, model-graded scoring, and reusable evaluation tasks.