What ModelScan Does
ModelScan was created by Protect AI to address one of the most overlooked security risks in machine learning: model serialization attacks. When ML models are saved to disk and shared between systems, the serialization process can embed arbitrary executable code alongside the model weights. An attacker who compromises a model file can inject code that executes automatically when the model is loaded, potentially stealing cloud credentials, poisoning outputs, or opening backdoors into production systems.
Scanning Approach and Format Support
The tool works by reading model files one byte at a time, scanning for code signatures known to be unsafe without ever loading or executing the model itself. This design choice is critical because the attack vector it protects against fires at load time. If the scanner loaded models to inspect them, it would trigger the very attacks it aims to detect. The byte-level approach makes scans extremely fast, typically completing in seconds proportional to file size.
ModelScan currently supports multiple serialization formats including Pickle and its variants like cloudpickle, dill, and joblib, as well as H5 (HDF5), TensorFlow SavedModel, and formats used by PyTorch, Keras, Sklearn, and XGBoost. The format list continues to expand with each release. It categorizes detected issues by severity level from CRITICAL for potential remote code execution risks down to HIGH for significant but less immediately dangerous vulnerabilities.
Installation and Configuration
Installation is straightforward as a Python package supporting Python 3.9 through 3.12. Teams can add it to their requirements.txt or pyproject.toml for consistent scanning across the organization. The CLI interface handles both individual model files and entire directories, with exit codes designed for CI/CD integration. A Python API is also available for embedding scanning into custom workflows and automated pipelines.
Configuration is flexible through a TOML settings file that can customize reporting format, output locations, and scanning rules. JSON reporting output makes it easy to integrate scan results into security dashboards and alerting systems. The tool can be run ad-hoc during model evaluation or integrated into automated pipelines that gate model deployment on successful security scans.
Reporting and the Commercial Layer
When ModelScan detects an issue, it reports the specific unsafe operators found and their location within the model file. For example, it might flag ReadFile and WriteFile operations in a TensorFlow model that could allow an attacker to read AWS credentials and exfiltrate them. The reports provide enough detail for security teams to assess whether the flagged operations are intentional or malicious.
Protect AI positions ModelScan as the open-source foundation with Guardian as the enterprise-grade upgrade. Guardian adds cutting-edge scanning capabilities, broader model format support with automatic format detection, proactive security policies for Hugging Face models, seamless CI/CD pipeline integration, and comprehensive audit trails. This dual offering lets teams start with ModelScan for free and upgrade when they need enterprise governance.
Context and Ecosystem
The broader context makes ModelScan increasingly important. Public model repositories like Hugging Face host thousands of community-contributed models that teams download and fine-tune with minimal security scrutiny. The Protect AI team has documented how ML models are not yet scanned with the rigor applied to something as basic as a PDF attachment, despite models increasingly powering critical business decisions.
ModelScan is part of a larger Protect AI open-source ecosystem that includes LLM Guard for runtime LLM security, NB Defense for Jupyter notebook security, and huntr, the world's first bug bounty platform for AI and ML systems. This ecosystem approach means teams can build a comprehensive AI security posture from open-source components before investing in enterprise solutions.
The Bottom Line
The main limitation of ModelScan is its focused scope. It specifically addresses serialization attacks and does not cover other ML security concerns like adversarial inputs, model poisoning during training, or prompt injection. Teams need it as one layer in a defense-in-depth strategy rather than a complete AI security solution. Additionally, the format coverage, while expanding, does not yet include every possible serialization method in the ML ecosystem.
