aicoolies logo
rampart
rampart

Rampart

Microsoft’s pytest-native red teaming framework for turning AI agent safety findings into CI tests.

open sourceupdated Jun 3, 2026

RAMPART is an open-source Microsoft framework for safety and security testing of agentic AI applications. It brings red-team findings into a pytest-native workflow so teams can turn prompt injection, unsafe tool use, and behavioral boundary failures into repeatable regression tests. The strongest aicoolies angle is developer workflow: RAMPART makes agent safety part of CI/CD instead of a one-off security review.

Read our Rampart review

A detailed review by the aicoolies team — click to read

RAMPART is Microsoft’s pytest-native framework for safety and security testing of agentic AI applications. Its key idea is to make agent red teaming look more like normal engineering work: define scenarios, run tests, capture failures and keep those checks in CI. That is a useful shift for teams building agents that browse, call tools, write files or interact with untrusted data sources, because safety can become part of the same loop as unit tests and pull requests.

The framework is especially relevant because agent failures are often behavioral and probabilistic rather than simple string-output bugs. RAMPART is designed to help teams turn prompt injection, unsafe tool use, data exfiltration and boundary-violation findings into repeatable regression tests. Its Microsoft/PyRIT lineage and pytest shape make it easier to connect security research with code-owned test suites, statistical evaluation and pull request gates.

RAMPART is not a substitute for thoughtful threat modeling or expert red team review. Poor scenarios will still create false confidence, and model-based testing can introduce cost and flakiness if it is not designed carefully. The value is in making agent safety continuous: developers can keep known failures from returning and add new adversarial cases as the agent gains more tools and autonomy. It belongs beside tools like garak, PyRIT and Promptfoo rather than replacing every security workflow.

Pricing

Free and open source under MIT; teams still pay for any models, infrastructure, or CI resources used during test generation and evaluation.

Platforms

Python/pytest-native framework with GitHub repository, documentation site, PyRIT lineage, and CI/CD fit for agent applications.

Categories

Tags

Use Cases

Alternatives

garak logo

garak

NVIDIA's LLM vulnerability scanner and red-teaming tool

garak is NVIDIA's open-source LLM vulnerability scanner for red-teaming AI models and applications. Probes for prompt injection, data leakage, hallucination, toxicity, encoding-based attacks, and dozens of other vulnerability categories. Runs automated attack sequences against any LLM endpoint and generates detailed vulnerability reports. Features a modular probe/detector architecture that is extensible with custom attack patterns. Named after the Star Trek character known for deception.

Open Source

PyRIT

Microsoft's automated red teaming framework for AI systems

PyRIT (Python Risk Identification Toolkit) is Microsoft's open-source framework for automated red teaming of generative AI systems. It enables security researchers to probe LLMs for jailbreaks, prompt injection, content safety bypasses, and harmful output generation using multi-turn attack strategies, scoring engines, and orchestrated adversarial workflows. Supports multiple target models and integrates with Azure AI services.

Open Source
Promptfoo logo

Promptfoo

LLM testing and evaluation toolkit

Promptfoo is an OpenAI-owned open-source toolkit for evaluating, red-teaming and securing LLM applications. It supports config-driven prompt/model tests, CI regression gates, red-team scans, guardrails, model security workflows, MCP Proxy, code scanning and evaluations across prompts, agents and RAG pipelines.

Open Source
Guardrails AI logo

Guardrails AI

Validate and structure LLM outputs with composable Guards

Guardrails AI is an open-source Python and JavaScript framework for validating and structuring LLM outputs using composable Guards built from a Hub of pre-built validators. It handles structured data extraction with Pydantic models, content safety checks including toxicity, PII detection, competitor mentions, and bias filtering, plus automatic re-prompting when validation fails. The Guardrails Hub offers dozens of validators from regex matching to hallucination detection via LLM judges.

free

Related Tools

MCP for Unity logo

MCP for Unity

Open-source MCP bridge between AI assistants and the Unity Editor

MCP for Unity is CoplayDev’s MIT-licensed bridge between MCP-compatible AI assistants and the Unity Editor. It exposes tools for assets, scenes, GameObjects, scripts, tests, profiling, and build-oriented workflows. The community project supports Unity 2021.3 LTS through 6.x and is explicitly not affiliated with Unity Technologies.

Open Source
XcodeBuildMCP logo

XcodeBuildMCP

Sentry-maintained MCP server and CLI for Xcode builds, simulators, and tests

XcodeBuildMCP is a Sentry-maintained, MIT-licensed MCP server and CLI for agent-assisted iOS and macOS development. It lets MCP-compatible coding agents run Xcode build and test workflows, manage simulators, inspect failures, and work through Homebrew, npm, or on-demand client configuration, with documented Sentry telemetry controls for teams that need an opt-out.

Open SourceTelemetry
MEDUSA logo

MEDUSA

AI-first security scanner for LLM, agent, MCP, and RAG codebases

MEDUSA is an AGPL-3.0 AI-first security scanner from Pantheon Security that checks AI and machine-learning applications, LLM agents, MCP workflows, RAG pipelines, repository-poisoning risks, secrets, and agent-specific compromise patterns.

Open Source
iFixAi logo

iFixAi

Open-source diagnostic for AI operational misalignment

iFixAi is an Apache-2.0 diagnostic tool for scoring AI agents and models against operational-misalignment risks such as hallucination, manipulation, sabotage, sandbagging, and oversight evasion.

Open Source

Inspect AI

UK AI Security Institute framework for LLM safety evaluations

Inspect AI is an MIT-licensed framework from the UK AI Security Institute for running large language model evaluations, including tool use, multi-turn dialogue, model-graded scoring, and reusable evaluation tasks.

Open Source
Presidio logo

Presidio

Open-source PII detection and anonymization for AI data flows

Presidio is an MIT-licensed privacy framework for identifying and anonymizing personally identifiable information in text, images, and structured data. It can act as a de-identification layer around LLM prompts, logs, RAG corpora, and customer-data workflows.

Open Source