Alibi Detect

Alibi Detect is an open source Python library focused on outlier, adversarial and drift detection. The package aims to cover both online and offline detectors for tabular data, text, images and time series. Both TensorFlow and PyTorch backends are supported for drift detection.

For more background on the importance of monitoring outliers and distributions in a production setting, check out this talk from the Challenges in Deploying and Monitoring Machine Learning Systems ICML 2020 workshop, based on the paper Monitoring and explainability of models in production and referencing Alibi Detect.

For a thorough introduction to drift detection, check out Protecting Your Machine Learning Against Drift: An Introduction. The talk covers what drift is and why it pays to detect it, the different types of drift, how it can be detected in a principled manner and also describes the anatomy of a drift detector.

Overview

• Getting Started
• Algorithm Overview
• Saving and Loading
• Detector Configuration Files
• Roadmap

Outlier Detection

• Methods
  • Mahalanobis Distance
  • Isolation Forest
  • Variational Auto-Encoder
  • Auto-Encoder
  • Variational Auto-Encoding Gaussian Mixture Model
  • Auto-Encoding Gaussian Mixture Model
  • Likelihood Ratios for Outlier Detection
  • Prophet Detector
  • Spectral Residual
  • Sequence-to-Sequence (Seq2Seq)
• Examples
  • AE outlier detection on CIFAR10
  • AEGMM and VAEGMM outlier detection on KDD Cup ‘99 dataset
  • Isolation Forest outlier detection on KDD Cup ‘99 dataset
  • Likelihood Ratio Outlier Detection on Genomic Sequences
  • Likelihood Ratio Outlier Detection with PixelCNN++
  • Mahalanobis outlier detection on KDD Cup ‘99 dataset
  • Time-series outlier detection using Prophet on weather data
  • Seq2Seq time series outlier detection on ECG data
  • Time series outlier detection with Seq2Seq models on synthetic data
  • Time series outlier detection with Spectral Residuals on synthetic data
  • VAE outlier detection for income prediction
  • VAE outlier detection on CIFAR10
  • VAE outlier detection on KDD Cup ‘99 dataset

Drift Detection

• Background
  • 1. What is drift?
  • 2. Detecting drift
  • 3. Learned drift detection
  • 4. Online drift detection
• Methods
  • Offline
  • Online
• Examples
  • Categorical and mixed type data drift detection on income prediction
  • Learned drift detectors on Adult Census
  • Learned drift detectors on CIFAR-10
  • Context-aware drift detection on news articles
  • Context-aware drift detection on ECGs
  • Model Distillation drift detector on CIFAR-10
  • Kolmogorov-Smirnov data drift detector on CIFAR-10
  • Maximum Mean Discrepancy drift detector on CIFAR-10
  • Scaling up drift detection with KeOps
  • Model uncertainty based drift detection on CIFAR-10 and Wine-Quality datasets
  • Drift detection on molecular graphs
  • Online drift detection for Camelyon17 medical imaging dataset
  • Online Drift Detection on the Wine Quality Dataset
  • Interpretable drift detection with the spot-the-diff detector on MNIST and Wine-Quality datasets
  • Supervised drift detection on the penguins dataset
  • Drift detection on Amazon reviews
  • Text drift detection on IMDB movie reviews

Adversarial Detection

• Methods
  • Adversarial Auto-Encoder
  • Model distillation
• Examples
  • Adversarial AE detection and correction on CIFAR-10

Deployment

• Outlier, adversarial and drift detection on CIFAR10

Datasets

• Overview

Models

• Overview

API reference

• API reference

Bibliography

• Bibliography