# llm-security

Enterprise middleware for securing AI applications against prompt attacks

Prompt Security provides enterprise security middleware that protects AI applications from prompt injection, data leakage, jailbreaks, and toxic content generation. It sits between users and LLM APIs to inspect, filter, and sanitize inputs and outputs in real-time. Supports deployment as a proxy, SDK integration, or browser extension with customizable security policies and compliance reporting.

CyberArk's open-source LLM fuzzing framework for AI security testing

FuzzyAI is CyberArk's open-source framework for fuzzing large language models to discover vulnerabilities like jailbreaks, prompt injection, guardrail bypasses, and harmful content generation. It systematically tests LLM deployments with over 20 attack techniques and generates detailed reports. Supports testing any model accessible via API including OpenAI, Anthropic, and self-hosted models.

Open-source LLM red-teaming framework with 40+ attack types

DeepTeam is an open-source red-teaming framework for systematically testing LLM applications against 40+ adversarial attack types. It covers OWASP Top 10 for LLMs including jailbreaks, prompt injection, PII leakage, and hallucination attacks. Built as the sister project of DeepEval for security testing alongside evaluation. Apache-2.0 licensed.

Security scanner for MCP servers against tool poisoning attacks

MCP-Scan is a security tool that scans MCP servers for vulnerabilities including tool poisoning, prompt injection, cross-origin escalation, and rug pull attacks. Acquired by Snyk in 2026, it is the first dedicated security scanner for the MCP ecosystem. It analyzes tool descriptions, permissions, and behavior patterns to detect malicious or compromised MCP servers before they can exploit AI agents.

AI observability with security posture management

Coralogix uses AI to provide actionable insights across logs and traces with a dedicated AI-SPM dashboard for tracking prompt injections and data leaks in AI applications. Its pay-per-use model with no upfront fees integrates security posture management directly into the observability stack, making it uniquely positioned for teams running both traditional and AI-powered production workloads.

DRM and IP protection for AI model weights

RefortifAI is a Y Combinator P2026 batch company that provides DRM and intellectual property protection for AI models by obfuscating model weights so they only run inside a hardened runtime. It solves the critical problem of model weight protection for companies distributing custom LLMs to untrusted environments, preventing IP theft while maintaining inference performance.

Prompt fuzzing tool for LLM security testing

ps-fuzz by Prompt Security is a security testing tool with 660+ GitHub stars that fuzzes system prompts against dynamic LLM-based attack scenarios including jailbreaks, prompt injection, and data extraction attempts. It helps developers harden their GenAI applications by simulating adversarial attacks in a controlled environment, turning LLM security into a testable and reproducible quality gate.

Security scanner for AI model files

ModelScan by Protect AI is an open-source tool that scans machine learning model files for malicious or unsafe code before they are loaded into production. Supporting formats like Pickle, HDF5, SavedModel, and SafeTensors, it detects hidden code execution, deserialization attacks, and supply chain threats in the AI/ML model artifact pipeline, integrating into CI/CD as a critical security gate.

Zero Trust runtime security for Kubernetes and AI

AccuKnox provides Zero Trust runtime threat prevention for Kubernetes and cloud workloads with an AI-powered prompt firewall to prevent LLM injection attacks. Built on the open-source KubeArmor project, it manages Kubernetes identities via policy-as-code, enforces runtime security policies, and provides real-time workload protection for AI-native infrastructure environments.

Input and output security scanners for LLM applications

LLM Guard is an open-source security toolkit by Protect AI that provides 15 input scanners and 20 output scanners to protect LLM applications from prompt injection, PII leakage, toxic content, secrets exposure, and data exfiltration. Each scanner is modular and independent — pick the ones you need, configure thresholds, and chain them into a pipeline. The library works with any LLM and has been downloaded over 2.5 million times. MIT licensed, Python 3.9+.

Validate and structure LLM outputs with composable Guards

Guardrails AI is an open-source Python and JavaScript framework for validating and structuring LLM outputs using composable Guards built from a Hub of pre-built validators. It handles structured data extraction with Pydantic models, content safety checks including toxicity, PII detection, competitor mentions, and bias filtering, plus automatic re-prompting when validation fails. The Guardrails Hub offers dozens of validators from regex matching to hallucination detection via LLM judges.

Programmable safety rails for LLM applications

NeMo Guardrails is NVIDIA's open-source toolkit for adding programmable safety rails to LLM applications. It supports five guardrail types — input, dialog, retrieval, execution, and output rails — covering content safety, jailbreak detection, topic control, PII masking, hallucination detection, and fact-checking. The toolkit uses Colang, a domain-specific language for defining conversational constraints, and integrates with OpenAI, Azure, Anthropic, HuggingFace, and LangChain/LangGraph.