MCP-Scan

NVIDIA's LLM vulnerability scanner and red-teaming tool

garak is NVIDIA's open-source LLM vulnerability scanner for red-teaming AI models and applications. Probes for prompt injection, data leakage, hallucination, toxicity, encoding-based attacks, and dozens of other vulnerability categories. Runs automated attack sequences against any LLM endpoint and generates detailed vulnerability reports. Features a modular probe/detector architecture that is extensible with custom attack patterns. Named after the Star Trek character known for deception.

Programmable safety rails for LLM applications

NeMo Guardrails is NVIDIA's open-source toolkit for adding programmable safety rails to LLM applications. It supports five guardrail types — input, dialog, retrieval, execution, and output rails — covering content safety, jailbreak detection, topic control, PII masking, hallucination detection, and fact-checking. The toolkit uses Colang, a domain-specific language for defining conversational constraints, and integrates with OpenAI, Azure, Anthropic, HuggingFace, and LangChain/LangGraph.

Input and output security scanners for LLM applications

LLM Guard is an open-source security toolkit by Protect AI that provides 15 input scanners and 20 output scanners to protect LLM applications from prompt injection, PII leakage, toxic content, secrets exposure, and data exfiltration. Each scanner is modular and independent — pick the ones you need, configure thresholds, and chain them into a pipeline. The library works with any LLM and has been downloaded over 2.5 million times. MIT licensed, Python 3.9+.

Validate and structure LLM outputs with composable Guards

Guardrails AI is an open-source Python and JavaScript framework for validating and structuring LLM outputs using composable Guards built from a Hub of pre-built validators. It handles structured data extraction with Pydantic models, content safety checks including toxicity, PII detection, competitor mentions, and bias filtering, plus automatic re-prompting when validation fails. The Guardrails Hub offers dozens of validators from regex matching to hallucination detection via LLM judges.

Open-source LLM red-teaming framework with 40+ attack types

DeepTeam is an open-source red-teaming framework for systematically testing LLM applications against 40+ adversarial attack types. It covers OWASP Top 10 for LLMs including jailbreaks, prompt injection, PII leakage, and hallucination attacks. Built as the sister project of DeepEval for security testing alongside evaluation. Apache-2.0 licensed.

Microsoft’s public-preview runtime governance toolkit for policy, identity, sandboxing, audit, and MCP security around AI agents.

Agent Governance Toolkit is Microsoft’s MIT-licensed public-preview toolkit for governing AI agent runtimes. It adds policy enforcement, zero-trust identity, execution sandboxing, audit, reliability, and MCP security-gateway patterns around tool calls and autonomous actions, helping platform teams move beyond prompt-only guardrails while preserving architecture review requirements.