alibi_detect.cd.base module
- class alibi_detect.cd.base.BaseClassifierDrift(x_ref, p_val=0.05, x_ref_preprocessed=False, preprocess_at_init=True, update_x_ref=None, preprocess_fn=None, preds_type='probs', binarize_preds=False, train_size=0.75, n_folds=None, retrain_from_scratch=True, seed=0, input_shape=None, data_type=None)[source]
Bases:
BaseDetector- __init__(x_ref, p_val=0.05, x_ref_preprocessed=False, preprocess_at_init=True, update_x_ref=None, preprocess_fn=None, preds_type='probs', binarize_preds=False, train_size=0.75, n_folds=None, retrain_from_scratch=True, seed=0, input_shape=None, data_type=None)[source]
A context-aware drift detector based on a conditional analogue of the maximum mean discrepancy (MMD). Only detects differences between samples that can not be attributed to differences between associated sets of contexts. p-values are computed using a conditional permutation test.
- Parameters:
x_ref (
Union[ndarray,list]) – Data used as reference distribution.p_val (
float) – p-value used for the significance of the test.x_ref_preprocessed (
bool) – Whether the given reference data x_ref has been preprocessed yet. If x_ref_preprocessed=True, only the test data x will be preprocessed at prediction time. If x_ref_preprocessed=False, the reference data will also be preprocessed.preprocess_at_init (
bool) – Whether to preprocess the reference data when the detector is instantiated. Otherwise, the reference data will be preprocessed at prediction time. Only applies if x_ref_preprocessed=False.update_x_ref (
Optional[Dict[str,int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.preprocess_fn (
Optional[Callable]) – Function to preprocess the data before computing the data drift metrics.preds_type (
str) – Whether the model outputs probabilities or logitsbinarize_preds (
bool) – Whether to test for discrepency on soft (e.g. probs/logits) model predictions directly with a K-S test or binarise to 0-1 prediction errors and apply a binomial test.train_size (
Optional[float]) – Optional fraction (float between 0 and 1) of the dataset used to train the classifier. The drift is detected on 1 - train_size. Cannot be used in combination with n_folds.n_folds (
Optional[int]) – Optional number of stratified folds used for training. The model preds are then calculated on all the out-of-fold predictions. This allows to leverage all the reference and test data for drift detection at the expense of longer computation. If both train_size and n_folds are specified, n_folds is prioritized.retrain_from_scratch (
bool) – Whether the classifier should be retrained from scratch for each set of test data or whether it should instead continue training from where it left off on the previous set.seed (
int) – Optional random seed for fold selection.data_type (
Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.
- get_splits(x_ref, x, return_splits=True)[source]
Split reference and test data in train and test folds used by the classifier.
- model: tensorflow.keras.Model | torch.nn.Module
- predict(x, return_p_val=True, return_distance=True, return_probs=True, return_model=True)[source]
Predict whether a batch of data has drifted from the reference data.
- Parameters:
return_p_val (
bool) – Whether to return the p-value of the test.return_distance (
bool) – Whether to return a notion of strength of the drift. K-S test stat if binarize_preds=False, otherwise relative error reduction.return_probs (
bool) – Whether to return the instance level classifier probabilities for the reference and test data (0=reference data, 1=test data). The reference and test instances of the associated probabilities are also returned.return_model (
bool) – Whether to return the updated model trained to discriminate reference and test instances.
- Return type:
- Returns:
Dictionary containing
'meta'and'data'dictionaries. –'meta'has the model’s metadata.'data'contains the drift prediction and optionally the p-value, performance of the classifier relative to its expectation under the no-change null, the out-of-fold classifier model prediction probabilities on the reference and test data as well as the associated reference and test instances of the out-of-fold predictions, and the trained model.
- class alibi_detect.cd.base.BaseContextMMDDrift(x_ref, c_ref, p_val=0.05, x_ref_preprocessed=False, preprocess_at_init=True, update_ref=None, preprocess_fn=None, x_kernel=None, c_kernel=None, n_permutations=1000, prop_c_held=0.25, n_folds=5, batch_size=256, input_shape=None, data_type=None, verbose=False)[source]
Bases:
BaseDetector- __init__(x_ref, c_ref, p_val=0.05, x_ref_preprocessed=False, preprocess_at_init=True, update_ref=None, preprocess_fn=None, x_kernel=None, c_kernel=None, n_permutations=1000, prop_c_held=0.25, n_folds=5, batch_size=256, input_shape=None, data_type=None, verbose=False)[source]
Maximum Mean Discrepancy (MMD) based context aware drift detector.
- Parameters:
x_ref (
Union[ndarray,list]) – Data used as reference distribution.c_ref (
ndarray) – Context for the reference distribution.p_val (
float) – p-value used for the significance of the permutation test.x_ref_preprocessed (
bool) – Whether the given reference data x_ref has been preprocessed yet. If x_ref_preprocessed=True, only the test data x will be preprocessed at prediction time. If x_ref_preprocessed=False, the reference data will also be preprocessed.preprocess_at_init (
bool) – Whether to preprocess the reference data when the detector is instantiated. Otherwise, the reference data will be preprocessed at prediction time. Only applies if x_ref_preprocessed=False.update_ref (
Optional[Dict[str,int]]) – Reference data can optionally be updated to the last N instances seen by the detector. The parameter should be passed as a dictionary {‘last’: N}.preprocess_fn (
Optional[Callable]) – Function to preprocess the data before computing the data drift metrics.x_kernel (
Optional[Callable]) – Kernel defined on the input data, defaults to Gaussian RBF kernel.c_kernel (
Optional[Callable]) – Kernel defined on the context data, defaults to Gaussian RBF kernel.n_permutations (
int) – Number of permutations used in the permutation test.prop_c_held (
float) – Proportion of contexts held out to condition on.n_folds (
int) – Number of cross-validation folds used when tuning the regularisation parameters.batch_size (
Optional[int]) – If not None, then compute batches of MMDs at a time (rather than all at once).data_type (
Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.verbose (
bool) – Whether or not to print progress during configuration.
- predict(x, c, return_p_val=True, return_distance=True, return_coupling=False)[source]
Predict whether a batch of data has drifted from the reference data, given the provided context.
- Parameters:
c (
ndarray) – Context associated with batch of instances.return_p_val (
bool) – Whether to return the p-value of the permutation test.return_distance (
bool) – Whether to return the conditional MMD test statistic between the new batch and reference data.return_coupling (
bool) – Whether to return the coupling matrices.
- Return type:
- Returns:
Dictionary containing
'meta'and'data'dictionaries. –'meta'has the model’s metadata.'data'contains the drift prediction and optionally the p-value, threshold, conditional MMD test statistic and coupling matrices.
- class alibi_detect.cd.base.BaseLSDDDrift(x_ref, p_val=0.05, x_ref_preprocessed=False, preprocess_at_init=True, update_x_ref=None, preprocess_fn=None, sigma=None, n_permutations=100, n_kernel_centers=None, lambda_rd_max=0.2, input_shape=None, data_type=None)[source]
Bases:
BaseDetector- __init__(x_ref, p_val=0.05, x_ref_preprocessed=False, preprocess_at_init=True, update_x_ref=None, preprocess_fn=None, sigma=None, n_permutations=100, n_kernel_centers=None, lambda_rd_max=0.2, input_shape=None, data_type=None)[source]
Least-squares Density Difference (LSDD) base data drift detector using a permutation test.
- Parameters:
x_ref (
Union[ndarray,list]) – Data used as reference distribution.p_val (
float) – p-value used for the significance of the permutation test.x_ref_preprocessed (
bool) – Whether the given reference data x_ref has been preprocessed yet. If x_ref_preprocessed=True, only the test data x will be preprocessed at prediction time. If x_ref_preprocessed=False, the reference data will also be preprocessed.preprocess_at_init (
bool) – Whether to preprocess the reference data when the detector is instantiated. Otherwise, the reference data will be preprocessed at prediction time. Only applies if x_ref_preprocessed=False.update_x_ref (
Optional[Dict[str,int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.preprocess_fn (
Optional[Callable]) – Function to preprocess the data before computing the data drift metrics.sigma (
Optional[ndarray]) – Optionally set the bandwidth of the Gaussian kernel used in estimating the LSDD. Can also pass multiple bandwidth values as an array. The kernel evaluation is then averaged over those bandwidths. If sigma is not specified, the ‘median heuristic’ is adopted whereby sigma is set as the median pairwise distance between reference samples.n_permutations (
int) – Number of permutations used in the permutation test.n_kernel_centers (
Optional[int]) – The number of reference samples to use as centers in the Gaussian kernel model used to estimate LSDD. Defaults to 1/20th of the reference data.lambda_rd_max (
float) – The maximum relative difference between two estimates of LSDD that the regularization parameter lambda is allowed to cause. Defaults to 0.2 as in the paper.data_type (
Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.
- predict(x, return_p_val=True, return_distance=True)[source]
Predict whether a batch of data has drifted from the reference data.
- Parameters:
- Return type:
- Returns:
Dictionary containing
'meta'and'data'dictionaries. –'meta'has the model’s metadata.'data'contains the drift prediction and optionally the p-value, threshold and LSDD metric.
- class alibi_detect.cd.base.BaseLearnedKernelDrift(x_ref, p_val=0.05, x_ref_preprocessed=False, preprocess_at_init=True, update_x_ref=None, preprocess_fn=None, n_permutations=100, train_size=0.75, retrain_from_scratch=True, input_shape=None, data_type=None)[source]
Bases:
BaseDetector- __init__(x_ref, p_val=0.05, x_ref_preprocessed=False, preprocess_at_init=True, update_x_ref=None, preprocess_fn=None, n_permutations=100, train_size=0.75, retrain_from_scratch=True, input_shape=None, data_type=None)[source]
Base class for the learned kernel-based drift detector.
- Parameters:
x_ref (
Union[ndarray,list]) – Data used as reference distribution.p_val (
float) – p-value used for the significance of the test.x_ref_preprocessed (
bool) – Whether the given reference data x_ref has been preprocessed yet. If x_ref_preprocessed=True, only the test data x will be preprocessed at prediction time. If x_ref_preprocessed=False, the reference data will also be preprocessed.preprocess_at_init (
bool) – Whether to preprocess the reference data when the detector is instantiated. Otherwise, the reference data will be preprocessed at prediction time. Only applies if x_ref_preprocessed=False.update_x_ref (
Optional[Dict[str,int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.preprocess_fn (
Optional[Callable]) – Function to preprocess the data before computing the data drift metrics.n_permutations (
int) – The number of permutations to use in the permutation test once the MMD has been computed.train_size (
Optional[float]) – Optional fraction (float between 0 and 1) of the dataset used to train the kernel. The drift is detected on 1 - train_size. Cannot be used in combination with n_folds.retrain_from_scratch (
bool) – Whether the kernel should be retrained from scratch for each set of test data or whether it should instead continue training from where it left off on the previous set.data_type (
Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.
- get_splits(x_ref, x)[source]
Split reference and test data into two splits – one of which to learn test locations and parameters and one to use for tests.
- kernel: tensorflow.keras.Model | torch.nn.Module
- predict(x, return_p_val=True, return_distance=True, return_kernel=True)[source]
Predict whether a batch of data has drifted from the reference data.
- Parameters:
- Return type:
Dict[Dict[str,str],Dict[str,Union[int,float,Callable]]]- Returns:
Dictionary containing
'meta'and'data'dictionaries. –'meta'has the detector’s metadata.'data'contains the drift prediction and optionally the p-value, threshold, MMD metric and trained kernel.
- class alibi_detect.cd.base.BaseMMDDrift(x_ref, p_val=0.05, x_ref_preprocessed=False, preprocess_at_init=True, update_x_ref=None, preprocess_fn=None, sigma=None, configure_kernel_from_x_ref=True, n_permutations=100, input_shape=None, data_type=None)[source]
Bases:
BaseDetector- __init__(x_ref, p_val=0.05, x_ref_preprocessed=False, preprocess_at_init=True, update_x_ref=None, preprocess_fn=None, sigma=None, configure_kernel_from_x_ref=True, n_permutations=100, input_shape=None, data_type=None)[source]
Maximum Mean Discrepancy (MMD) base data drift detector using a permutation test.
- Parameters:
x_ref (
Union[ndarray,list]) – Data used as reference distribution.p_val (
float) – p-value used for the significance of the permutation test.x_ref_preprocessed (
bool) – Whether the given reference data x_ref has been preprocessed yet. If x_ref_preprocessed=True, only the test data x will be preprocessed at prediction time. If x_ref_preprocessed=False, the reference data will also be preprocessed.preprocess_at_init (
bool) – Whether to preprocess the reference data when the detector is instantiated. Otherwise, the reference data will be preprocessed at prediction time. Only applies if x_ref_preprocessed=False.update_x_ref (
Optional[Dict[str,int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.preprocess_fn (
Optional[Callable]) – Function to preprocess the data before computing the data drift metrics.sigma (
Optional[ndarray]) – Optionally set the Gaussian RBF kernel bandwidth. Can also pass multiple bandwidth values as an array. The kernel evaluation is then averaged over those bandwidths.configure_kernel_from_x_ref (
bool) – Whether to already configure the kernel bandwidth from the reference data.n_permutations (
int) – Number of permutations used in the permutation test.data_type (
Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.
- predict(x, return_p_val=True, return_distance=True)[source]
Predict whether a batch of data has drifted from the reference data.
- Parameters:
- Return type:
- Returns:
Dictionary containing
'meta'and'data'dictionaries. –'meta'has the model’s metadata.'data'contains the drift prediction and optionally the p-value, threshold and MMD metric.
- class alibi_detect.cd.base.BaseUnivariateDrift(x_ref, p_val=0.05, x_ref_preprocessed=False, preprocess_at_init=True, update_x_ref=None, preprocess_fn=None, correction='bonferroni', n_features=None, input_shape=None, data_type=None)[source]
Bases:
BaseDetector,DriftConfigMixin- __init__(x_ref, p_val=0.05, x_ref_preprocessed=False, preprocess_at_init=True, update_x_ref=None, preprocess_fn=None, correction='bonferroni', n_features=None, input_shape=None, data_type=None)[source]
Generic drift detector component which serves as a base class for methods using univariate tests. If n_features > 1, a multivariate correction is applied such that the false positive rate is upper bounded by the specified p-value, with equality in the case of independent features.
- Parameters:
x_ref (
Union[ndarray,list]) – Data used as reference distribution.p_val (
float) – p-value used for significance of the statistical test for each feature. If the FDR correction method is used, this corresponds to the acceptable q-value.x_ref_preprocessed (
bool) – Whether the given reference data x_ref has been preprocessed yet. If x_ref_preprocessed=True, only the test data x will be preprocessed at prediction time. If x_ref_preprocessed=False, the reference data will also be preprocessed.preprocess_at_init (
bool) – Whether to preprocess the reference data when the detector is instantiated. Otherwise, the reference data will be preprocessed at prediction time. Only applies if x_ref_preprocessed=False.update_x_ref (
Optional[Dict[str,int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.preprocess_fn (
Optional[Callable]) – Function to preprocess the data before computing the data drift metrics. Typically a dimensionality reduction technique.correction (
str) – Correction type for multivariate data. Either ‘bonferroni’ or ‘fdr’ (False Discovery Rate).n_features (
Optional[int]) – Number of features used in the statistical test. No need to pass it if no preprocessing takes place. In case of a preprocessing step, this can also be inferred automatically but could be more expensive to compute.input_shape (
Optional[tuple]) – Shape of input data. Needs to be provided for text data.data_type (
Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.
- predict(x, drift_type='batch', return_p_val=True, return_distance=True)[source]
Predict whether a batch of data has drifted from the reference data.
- Parameters:
drift_type (
str) – Predict drift at the ‘feature’ or ‘batch’ level. For ‘batch’, the test statistics for each feature are aggregated using the Bonferroni or False Discovery Rate correction (if n_features>1).return_p_val (
bool) – Whether to return feature level p-values.return_distance (
bool) – Whether to return the test statistic between the features of the new batch and reference data.
- Return type:
- Returns:
Dictionary containing
'meta'and'data'dictionaries. –'meta'has the model’s metadata.'data'contains the drift prediction and optionally the feature level p-values, threshold after multivariate correction if needed and test statistics.