This page was generated from cd/methods/fetdrift.ipynb.
Fisher’s Exact Test
Overview
The FET drift detector is a nonparametric drift detector. It applies Fisher’s Exact Test (FET) to each feature, and is intended for application to Bernoulli distributions, with binary univariate data consisting of either (True, False)
or (0, 1)
. This detector is ideal for use in a supervised setting, monitoring drift in a model’s instance level accuracy (i.e. correct
prediction = 0, and incorrect prediction = 1).
The detector is primarily intended for univariate data, but can also be used in a multivariate setting. For multivariate data, the obtained pvalues for each feature are aggregated either via the Bonferroni or the False Discovery Rate (FDR) correction. The Bonferroni correction is more conservative and controls for the probability of at least one false positive. The FDR correction on the other hand allows for an expected fraction of false positives to occur. As with other univariate detectors such as the KolmogorovSmirnov detector, for highdimensional data, we typically want to reduce the dimensionality before computing the featurewise univariate FET tests and aggregating those via the chosen correction method. See Dimension Reduction for more guidance on this.
For the \(j^{th}\) feature, the FET detector considers the 2x2 contingency table between the reference data \(x_j^{ref}\) and test data \(x_j\) for that feature:
True (1) 
False (0) 


\(x_j\) 
\(N_1\) 
\(N_0\) 
\(x_j^{ref}\) 
\(N^{ref}_1\) 
\(N^{ref}_0\) 
where \(N^{ref}_1\) represents the number of 1’s in the reference data (for the \(j^{th}\) feature), \(N^{ref}_0\) the number of 0’s, and so on. These values can be used to define an odds ratio:
The null hypothesis is \(H_0: \widehat{OR}=1\). In other words, the proportion of 1’s to 0’s is unchanged between the test and reference distributions, such that the odds of 1’s vs 0’s is independent of whether the data is drawn from the reference or test distribution. The offline FET detector can perform onesided or twosided tests, with the alternative hypothesis set by the alternative
keyword argument:
If
alternative='greater'
, the alternative hypothesis is \(H_a: \widehat{OR}>1\) i.e. proportion of 1’s versus 0’s has increased compared to the reference distribution.If
alternative='less'
, the alternative hypothesis is \(H_a: \widehat{OR}<1\) i.e. the proportion of 1’s versus 0’s has decreased compared to the reference distribution.If
alternative='twosided'
, the alternative hypothesis is \(H_a: \widehat{OR} \ne 1\) i.e. the proportion of 1’s versus 0’s has changed compared to the reference distribution.
The pvalue returned by the detector is then the probability of obtaining an odds ratio at least as extreme as that observed (in the direction specified by alternative
), assuming the null hypothesis is true.
Usage
Initialize
Arguments:
x_ref
: Data used as reference distribution. Note this should be the raw data, for examplenp.array([0, 0, 1, 0, 0, 0])
, not the 2x2 contingency table.
Keyword arguments:
p_val
: pvalue used for significance of the FET test. If the FDR correction method is used, this corresponds to the acceptable qvalue.preprocess_at_init
: Whether to already apply the (optional) preprocessing step to the reference data at initialization and store the preprocessed data. Dependent on the preprocessing step, this can reduce the computation time for the predict step significantly, especially when the reference dataset is large. Defaults to True. It is possible that it needs to be set to False if the preprocessing step requires statistics from both the reference and test data, such as the mean or standard deviation.x_ref_preprocessed
: Whether or not the reference datax_ref
has already been preprocessed. If True, the reference data will be skipped and preprocessing will only be applied to the test data passed topredict
.update_x_ref
: Reference data can optionally be updated to the last N instances seen by the detector or via reservoir sampling with size N. For the former, the parameter equals {‘last’: N} while for reservoir sampling {‘reservoir_sampling’: N} is passed.preprocess_fn
: Function to preprocess the data before computing the data drift metrics. Typically a dimensionality reduction technique.correction
: Correction type for multivariate data. Either ‘bonferroni’ or ‘fdr’ (False Discovery Rate).alternative
: Defines the alternative hypothesis. Options are ‘greater’ (default), ‘less’ or ‘twosided’.n_features
: Number of features used in the FET test. No need to pass it if no preprocessing takes place. In case of a preprocessing step, this can also be inferred automatically but could be more expensive to compute.input_shape
: Shape of input data.data_type
: can specify data type added to metadata. E.g. ‘tabular’ or ‘image’.
Initialized drift detector example:
from alibi_detect.cd import FETDrift
cd = FETDrift(x_ref, p_val=0.05)
Detect Drift
We detect data drift by simply calling predict
on a batch of instances x
. We can return the featurewise pvalues before the multivariate correction by setting return_p_val
to True. The drift can also be detected at the feature level by setting drift_type
to ‘feature’. No multivariate correction will take place since we return the output of n_features univariate tests. For drift detection on all the features combined with the correction, use ‘batch’. return_p_val
equal to True will also return the threshold used by the detector (either for the univariate case or after the multivariate correction).
The prediction takes the form of a dictionary with meta
and data
keys. meta
contains the detector’s metadata while data
is also a dictionary which contains the actual predictions stored in the following keys:
is_drift
: 1 if the sample tested has drifted from the reference data and 0 otherwise.p_val
: contains featurelevel pvalues ifreturn_p_val
equals True.threshold
: for featurelevel drift detection the threshold equals the pvalue used for the significance of the FET test. Otherwise the threshold after the multivariate correction (either bonferroni or fdr) is returned.distance
: Featurewise test statistics between the reference data and the new batch ifreturn_distance
equals True. In this case, the test statistics correspond to the odds ratios.
preds = cd.predict(x, drift_type='batch', return_p_val=True)