ModelScan

Validate and structure LLM outputs with composable Guards

Guardrails AI is an open-source Python and JavaScript framework for validating and structuring LLM outputs using composable Guards built from a Hub of pre-built validators. It handles structured data extraction with Pydantic models, content safety checks including toxicity, PII detection, competitor mentions, and bias filtering, plus automatic re-prompting when validation fails. The Guardrails Hub offers dozens of validators from regex matching to hallucination detection via LLM judges.

Prompt registry, observability, and evaluation workflows for LLM applications.

PromptLayer is a prompt management, observability, and evaluation platform for LLM applications. Teams use its Prompt Registry, visual editor, request logs, Tables, evaluations, Tool Registry, and Skill Collections to version prompts, replay requests, compare variants, run datasets, and ship prompt changes without redeploying code. Pricing starts with Free $0 for 5 users and 2.5K requests/month, Pro $49/month, Team $500/month, and Enterprise custom.

Programmable safety rails for LLM applications

NeMo Guardrails is NVIDIA's open-source toolkit for adding programmable safety rails to LLM applications. It supports five guardrail types — input, dialog, retrieval, execution, and output rails — covering content safety, jailbreak detection, topic control, PII masking, hallucination detection, and fact-checking. The toolkit uses Colang, a domain-specific language for defining conversational constraints, and integrates with OpenAI, Azure, Anthropic, HuggingFace, and LangChain/LangGraph.

Run and govern MCP servers across desktop, CLI and Kubernetes

Open-source MCP runtime and governance platform that runs servers in isolated containers, curates registries, enforces access policies, and operates gateways across desktop, CLI, and Kubernetes.

Craft image-scaling prompt-injection payloads to red-team multimodal AI systems

Open-source red-team toolkit from Trail of Bits that generates image-scaling attack payloads — images that look benign at full resolution but reveal a hidden prompt injection after a multimodal system downsamples them.

AI agent framework for offensive security and penetration testing

Alias Robotics' agent framework for building AI-driven offensive-security workflows — reconnaissance, exploitation, privilege escalation, and lateral movement — with multi-agent handoffs and human-in-the-loop control. Source-available, but the core is licensed for non-commercial research use only.

AI-first security scanner for LLM, agent, MCP, and RAG codebases

MEDUSA is an AGPL-3.0 AI-first security scanner from Pantheon Security that checks AI and machine-learning applications, LLM agents, MCP workflows, RAG pipelines, repository-poisoning risks, secrets, and agent-specific compromise patterns.

Open-source diagnostic for AI operational misalignment

iFixAi is an Apache-2.0 diagnostic tool for scoring AI agents and models against operational-misalignment risks such as hallucination, manipulation, sabotage, sandbagging, and oversight evasion.

UK AI Security Institute framework for LLM safety evaluations

Inspect AI is an MIT-licensed framework from the UK AI Security Institute for running large language model evaluations, including tool use, multi-turn dialogue, model-graded scoring, and reusable evaluation tasks.