WhyLabs is an AI observability platform created at the Allen Institute for Artificial Intelligence (AI2) by Amazon Machine Learning alumni. The complete platform has been open-sourced to support AI observability research, making all technology freely available. It enables continuous monitoring of data and model health across the ML lifecycle.
The platform monitors ML models, LLM applications, and data pipelines, surfacing data drift, quality issues, performance degradation, and bias. Automated alerting across dozens of data vitals comes with out-of-the-box configurations. Purpose-built agents analyze raw data without moving or duplicating it, ensuring privacy and security.
For LLM applications, WhyLabs extracts telemetry from prompts and responses to detect malicious prompts, toxicity, hallucinations, and jailbreak attempts. The system processes 100% of data without sampling, supporting tabular, image, text, and embedding data types across any platform.
Two open-source projects underpin the platform: whylogs for privacy-preserving data logging, and LangKit for monitoring LLM quality and safety. Both continue to be actively maintained and widely used.
Integrates with Azure, SageMaker, MLflow, Apache Spark, Pandas, Kafka, Ray, Airflow, dbt, and Databricks. Supports SOC 2 Type 2 compliance, RBAC, SAML SSO. Works in cloud, hybrid, or on-premises environments.