# LLM Security

UK AI Security Institute framework for LLM safety evaluations

Inspect AI is an MIT-licensed framework from the UK AI Security Institute for running large language model evaluations, including tool use, multi-turn dialogue, model-graded scoring, and reusable evaluation tasks.

Zero Trust runtime security for Kubernetes and AI

AccuKnox provides Zero Trust runtime threat prevention for Kubernetes and cloud workloads with an AI-powered prompt firewall to prevent LLM injection attacks. Built on the open-source KubeArmor project, it manages Kubernetes identities via policy-as-code, enforces runtime security policies, and provides real-time workload protection for AI-native infrastructure environments.

AI observability with security posture management

Coralogix uses AI to provide actionable insights across logs and traces with a dedicated AI-SPM dashboard for tracking prompt injections and data leaks in AI applications. Its pay-per-use model with no upfront fees integrates security posture management directly into the observability stack, making it uniquely positioned for teams running both traditional and AI-powered production workloads.

Security scanner for AI model files

ModelScan by Protect AI is an open-source tool that scans machine learning model files for malicious or unsafe code before they are loaded into production. Supporting formats like Pickle, HDF5, and SavedModel, it detects hidden code execution, deserialization attacks, and supply chain threats in the AI/ML model artifact pipeline, integrating into CI/CD as a critical security gate.

Prompt fuzzing tool for LLM security testing

ps-fuzz by Prompt Security is a security testing tool with 680+ GitHub stars that fuzzes system prompts against dynamic LLM-based attack scenarios including jailbreaks, prompt injection, and data extraction attempts. It helps developers harden their GenAI applications by simulating adversarial attacks in a controlled environment, turning LLM security into a testable and reproducible quality gate.

LLM vulnerability scanner and red teaming kit

Agentic Security is an open-source vulnerability scanner for LLM agent workflows that tests AI systems against jailbreaks, fuzzing, and multimodal attacks. It probes weaknesses across text, image, and audio inputs through multi-step jailbreak simulations, randomized stress testing, and reinforcement learning-powered adaptive attacks. The toolkit connects directly to LLM APIs for high-volume real-world attack scenarios, helping developers identify and patch safety gaps before deployment.

Craft image-scaling prompt-injection payloads to red-team multimodal AI systems

Open-source red-team toolkit from Trail of Bits that generates image-scaling attack payloads — images that look benign at full resolution but reveal a hidden prompt injection after a multimodal system downsamples them.

Open-source LLM red-teaming framework with 40+ attack types

DeepTeam is an open-source red-teaming framework for systematically testing LLM applications against 40+ adversarial attack types. It covers OWASP Top 10 for LLMs including jailbreaks, prompt injection, PII leakage, and hallucination attacks. Built as the sister project of DeepEval for security testing alongside evaluation. Apache-2.0 licensed.

AI security platform for model protection and threat detection

HiddenLayer is an AI security platform that protects machine learning models across their full lifecycle. It provides runtime model security to detect adversarial attacks in real-time, model scanning for supply chain threats in model files, automated red teaming for vulnerability assessment, and AI guardrails for prompt injection defense. Backed by $31.9M from M12 and IBM Ventures, named Gartner Cool Vendor 2024.

Input and output security scanners for LLM applications

LLM Guard is an open-source security toolkit by Protect AI that provides 15 input scanners and 20 output scanners to protect LLM applications from prompt injection, PII leakage, toxic content, secrets exposure, and data exfiltration. Each scanner is modular and independent — pick the ones you need, configure thresholds, and chain them into a pipeline. The library works with any LLM and has been downloaded over 2.5 million times. MIT licensed, Python 3.9+.

Security scanner for MCP servers against tool poisoning attacks

MCP-Scan is a security tool that scans MCP servers for vulnerabilities including tool poisoning, prompt injection, cross-origin escalation, and rug pull attacks. Acquired by Snyk in 2026, it is the first dedicated security scanner for the MCP ecosystem. It analyzes tool descriptions, permissions, and behavior patterns to detect malicious or compromised MCP servers before they can exploit AI agents.

AI-first security scanner for LLM, agent, MCP, and RAG codebases

MEDUSA is an AGPL-3.0 AI-first security scanner from Pantheon Security that checks AI and machine-learning applications, LLM agents, MCP workflows, RAG pipelines, repository-poisoning risks, secrets, and agent-specific compromise patterns.

Autonomous AI red teaming with black-box attack simulation

Novee is an AI security platform that performs autonomous black-box red teaming of AI systems. It uses an advanced reasoning engine to simulate determined adversaries, probing for complex chained attack scenarios like prompt injection leading to database access through AI agents. Tests for jailbreaks, data extraction, unauthorized actions, and multi-step exploit chains in LLM applications.

Cryptographic signing and verification for ML models

OpenSSF Model Signing is an open-source project for cryptographically signing and verifying machine learning model files to ensure integrity and provenance. Built on Sigstore PKI, it provides CLI tools and a Python library for signing model artifacts and verifying they haven't been tampered with. Part of the OpenSSF AI/ML Working Group, reaching v1.0 in 2025 for production supply chain security.

Open-source PII detection and anonymization for AI data flows

Presidio is an MIT-licensed privacy framework for identifying and anonymizing personally identifiable information in text, images, and structured data. It can act as a de-identification layer around LLM prompts, logs, RAG corpora, and customer-data workflows.

DRM and IP protection for AI model weights

RefortifAI is a Y Combinator P2026 batch company that provides DRM and intellectual property protection for AI models by obfuscating model weights so they only run inside a hardened runtime. It solves the critical problem of model weight protection for companies distributing custom LLMs to untrusted environments, preventing IP theft while maintaining inference performance.

Open-source diagnostic for AI operational misalignment

iFixAi is an Apache-2.0 diagnostic tool for scoring AI agents and models against operational-misalignment risks such as hallucination, manipulation, sabotage, sandbagging, and oversight evasion.