alibi.explainers.shap_wrappers module

class alibi.explainers.shap_wrappers.KernelShap(predictor, link='identity', feature_names=None, categorical_names=None, task='classification', seed=None)[source]

Bases: alibi.api.interfaces.Explainer, alibi.api.interfaces.FitMixin

__init__(predictor, link='identity', feature_names=None, categorical_names=None, task='classification', seed=None)[source]

A wrapper around the shap.KernelExplainer class. It extends the current shap library functionality by allowing the user to specify variable groups in order to treat one-hot encoded categorical as one during sampling. The user can also specify whether to aggregate the shap values estimate for the encoded levels of categorical variables as an optional argument to explain, if grouping arguments are not passed to fit.

Parameters
  • predictor (Callable) – A callable that takes as an input a samples x features array and outputs a samples x n_outputs model outputs. The n_outputs should represent model output in margin space. If the model outputs probabilities, then the link should be set to ‘logit’ to ensure correct force plots.

  • link (str) –

    Valid values are ‘identity’ or ‘logit’. A generalized linear model link to connect the feature importance values to the model output. Since the feature importance values, \(\phi\), sum up to the model output, it often makes sense to connect them to the ouput with a link function where \(link(output - expected\_value) = sum(\phi)\). Therefore, for a model which outputs probabilities, link=’logit’ makes the feature effects have log-odds (evidence) units and link=’identity’ means that the feature effects have probability units. Please see this example for an in-depth discussion about the semantics of explaining the model in the probability or margin space.

  • feature_names (Union[List[str], Tuple[str], None]) – Used to infer group names when categorical data is treated by grouping and group_names input to fit is not specified, assuming it has the same length as the groups argument of fit method. It is also used to compute the names field, which appears as a key in each of the values of explanation.data[‘raw’][‘importances’].

  • categorical_names (Optional[Dict[int, List[str]]]) – Keys are feature column indices in the background_data matrix (see fit). Each value contains strings with the names of the categories for the feature. Used to select the method for background data summarisation (if specified, subsampling is performed as opposed to k-means clustering). In the future it may be used for visualisation.

  • task (str) – Can have values ‘classification’ and ‘regression’. It is only used to set the contents of explanation.data[‘raw’][‘prediction’]

  • seed (Optional[int]) – Fixes the random number stream, which influences which subsets are sampled during shap value estimation.

build_explanation(X, shap_values, expected_value, **kwargs)[source]

Create an explanation object. If output summarisation is required and all inputs necessary for this operation are passed, the raw shap values are summed first so that a single shap value is returned for each categorical variable, as opposed to a shap value per dimension of categorical variable encoding.

Parameters
  • X (Union[ndarray, DataFrame, spmatrix]) – Instances to be explained.

  • shap_values (List[ndarray]) – Each entry is a n_instances x n_features array, and the length of the list equals the dimensionality of the predictor output. The rows of each array correspond to the shap values for the instances with the corresponding row index in X. The length of the list equals the number of model outputs.

  • expected_value (List[float]) – A list containing the expected value of the prediction for each class. Its length should be equal to that of shap_values.

Return type

Explanation

Returns

explanation – An explanation object containing the shap values and prediction in the data field, along with a meta field containing additional data. See usage examples for details.

explain(X, summarise_result=False, cat_vars_start_idx=None, cat_vars_enc_dim=None, **kwargs)[source]

Explains the instances in the array X.

Parameters
  • X (Union[ndarray, DataFrame, spmatrix]) – Instances to be explained.

  • summarise_result (bool) – Specifies whether the shap values corresponding to dimensions of encoded categorical variables should be summed so that a single shap value is returned for each categorical variable. Both the start indices of the categorical variables (cat_vars_start_idx) and the encoding dimensions (cat_vars_enc_dim) have to be specified

  • cat_vars_start_idx (Optional[Sequence[int]]) – The start indices of the categorical variables. If specified, cat_vars_enc_dim should also be specified.

  • cat_vars_enc_dim (Optional[Sequence[int]]) – The length of the encoding dimension for each categorical variable. If specified cat_vars_start_idx should also be specified.

  • kwargs

    Keyword arguments specifying explain behaviour. Valid arguments are:

    • nsamples: controls the number of predictor calls and therefore runtime.

    • l1_reg: the algorithm is exponential in the feature dimension. If set to auto the algorithm will first run a feature selection algorithm to select the top features, provided the fraction of sampled sets of missing features is less than 0.2 from the number of total subsets. The Akaike Information Criterion is used in this case. See our examples for more details about available settings for this parameter. Note that by first running a feature selection step, the shapley values of the remainder of the features will be different to those estimated from the entire set.

    For more details, please see the shap library documentation .

Return type

Explanation

Returns

explanation – An explanation object containing the algorithm results.

fit(background_data, summarise_background=False, n_background_samples=300, group_names=None, groups=None, weights=None, **kwargs)[source]

This takes a background dataset (usually a subsample of the training set) as an input along with several user specified options and initialises a KernelShap explainer. The runtime of the algorithm depends on the number of samples in this dataset and on the number of features in the dataset. To reduce the size of the dataset, the summarise_background option and n_background_samples should be used. To reduce the feature dimensionality, encoded categorical variables can be treated as one during the feature perturbation process; this decreases the effective feature dimensionality, can reduce the variance of the shap values estimation and reduces slightly the number of calls to the predictor. Further runtime savings can be achieved by changing the nsamples parameter in the call to explain. Runtime reduction comes with an accuracy trade-off, so it is better to experiment with a runtime reduction method and understand results stability before using the system.

Parameters
  • background_data (Union[ndarray, spmatrix, DataFrame, Data]) – Data used to estimate feature contributions and baseline values for force plots. The rows of the background data should represent samples and the columns features.

  • summarise_background (Union[bool, str]) – A large background dataset impacts the runtime and memory footprint of the algorithm. By setting this argument to True, only n_background_samples from the provided data are selected. If group_names or groups arguments are specified, the algorithm assumes that the data contains categorical variables so the records are selected uniformly at random. Otherwise, shap.kmeans (a wrapper around sklearn k-means implementation) is used for selection. If set to ‘auto’, a default of KERNEL_SHAP_BACKGROUND_THRESHOLD samples is selected.

  • n_background_samples (int) – The number of samples to keep in the background dataset if summarise_background=True.

  • groups (Optional[List[Union[Tuple[int], List[int]]]]) – A list containing sub-lists specifying the indices of features belonging to the same group.

  • group_names (Union[List[str], Tuple[str], None]) – If specified, this array is used to treat groups of features as one during feature perturbation. This feature can be useful, for example, to treat encoded categorical variables as one and can result in computational savings (this may require adjusting the nsamples parameter).

  • weights (Union[List[float], Tuple[float], ndarray, None]) – A sequence or array of weights. This is used only if grouping is specified and assigns a weight to each point in the dataset.

  • kwargs – Expected keyword arguments include keep_index (bool) and should be used if a data frame containing an index column is passed to the algorithm.

Return type

KernelShap

class alibi.explainers.shap_wrappers.TreeShap(predictor, model_output='raw', feature_names=None, categorical_names=None, task='classification', seed=None)[source]

Bases: alibi.api.interfaces.Explainer, alibi.api.interfaces.FitMixin

__init__(predictor, model_output='raw', feature_names=None, categorical_names=None, task='classification', seed=None)[source]

A wrapper around the shap.TreeExplainer class. It adds the following functionality:

  1. Input summarisation options to allow control over background dataset size and hence runtime

  2. Output summarisation for sklearn models with one-hot encoded categorical variables.

Users are strongly encouraged to familiarise themselves with the algorithm by reading the method overview in the documentation.

Parameters
  • predictor (Any) – A fitted model to be explained. XGBoost, LightGBM, CatBoost and most tree-based scikit-learn models are supported. In the future, Pyspark could also be supported. Please open an issue if this is a use case for you.

  • model_output (str) –

    Supported values are: ‘raw’, ‘probability’, ‘probability_doubled’, ‘log_loss’:

    • ’raw’: the raw model of the output, which varies by task, is explained. This option should always be used if the fit is called without arguments. It should also be set to compute shap interaction values. For regression models it is the standard output, for binary classification in XGBoost it is the log odds ratio.

    • ’probability’: the probability output is explained. This option should only be used if fit was was called with the background_data argument set. The effect of specifying this parameter is that the shap library will use this information to transform the shap values computed in margin space (aka using the raw output) to shap values that sum to the probability output by the model plus the model expected output probability. This requires knowledge of the type of output for predictor which is inferred by the shap library from the model type (e.g., most sklearn models with exception of sklearn.tree.DecisionTreeClassifier, sklearn.ensemble.RandomForestClassifier, sklearn.ensemble.ExtraTreesClassifier output logits) or on the basis of the mapping implemented in the shap.TreeEnsemble constructor. Only trees that output log odds and probabilities are supported currently.

    • ’probability_doubled’: used for binary classification problem in situations where the model outputs the logits/probabilities for the positive class but shap values for both outcomes are desired. This option should be used only if fit was called with the background_data argument set. In this case the expected value for the negative class is 1 - expected_value for positive class and the shap values for the negative class are the negative values of the positive class shap values. As before, the explanation happens in the margin space, and the shap values are subsequently adjusted. convert the model output to probabilities. The same considerations as for probability apply for this output type too.

    • ’log_loss’: logarithmic loss is explained. This option shoud be used only if fit was called with the background_data argument set and requires specifying labels, y, when calling explain. If the objective is squared error, then the transformation \((output - y)^2\) is applied. For binary cross-entropy objective, the transformation \(log(1 + exp(output)) - y * output\) with \(y \in \{0, 1\}\). Currently only binary cross-entropy and squared error losses can be explained.

  • feature_names (Union[List[str], Tuple[str], None]) – Used to compute the names field, which appears as a key in each of the values of the importances sub-field of the response raw field.

  • categorical_names (Optional[Dict[int, List[str]]]) – Keys are feature column indices. Each value contains strings with the names of the categories for the feature. Used to select the method for background data summarisation (if specified, subsampling is performed as opposed to kmeans clustering). In the future it may be used for visualisation.

  • task (str) – Can have values ‘classification’ and ‘regression’. It is only used to set the contents of the prediction field in the data[‘raw’] response field.

Notes

Tree SHAP is an additive attribution method so it is best suited to explaining output in margin space (the entire real line). For discussion related to explaining models in output vs probability space, please consult this resource.

build_explanation(X, shap_output, expected_value, **kwargs)[source]

Create an explanation object. If output summarisation is required and all inputs necessary for this operation are passed, the raw shap values are summed first so that a single shap value is returned for each categorical variable, as opposed to a shap value per dimension of categorical variable encoding. Similarly, the shap interaction values are summed such that they represent the interaction between categorical variables as opposed to levels of categorical variables. If the interaction option has been specified during explain, this method computes the shap values given the interactions prior to creating the response.

Parameters
  • X (Union[ndarray, DataFrame, Pool')]) – Instances to be explained.

  • shap_output (List[ndarray]) – If explain is callled with interactions=True then the list contains tensors of dimensionality n_instances x n_features x n_features of shap interaction values. Otherwise, it contains tensors of dimension n_instances x n_features representing shap values. The length of the list equals the number of model outputs.

  • expected_value (List[float]) – A list containing the expected value of the prediction for each class. Its length is equal to that of shap_output.

Return type

Explanation

Returns

explanation – An Explanation object containing the shap values and prediction in the data field, along with a meta field containing additional data. See usage examples here for details.

explain(X, y=None, interactions=False, approximate=False, check_additivity=True, tree_limit=None, summarise_result=False, cat_vars_start_idx=None, cat_vars_enc_dim=None, **kwargs)[source]

Explains the instances in X. y should be passed if the model loss function is to be explained, which can be useful in order to understand how various features affect model performance over time. This is only possible if the explainer has been fitted with a background dataset and requires setting model_output=’log_loss’.

Parameters
  • X (Union[ndarray, DataFrame, Pool')]) – Instances to be explained.

  • y (Optional[ndarray]) – Labels corresponding to rows of X. Should be passed only if a background dataset was passed to the fit method.

  • interactions (bool) – If True, the shap value for every feature of every instance in X is decomposed into X.shape[1] - 1 shap value interactions and one main effect. This is only supported if fit is called with background_dataset=None.

  • approximate (bool) –

    If True, an approximation to the shap values that does not account for feature order is computed. This was proposed by Ando Sabaas here . Check this resource for more details. This option is currently only supported for xgboost and sklearn models.

  • check_additivity (bool) – If True, output correctness is ensured if model_output=’raw’ has been passed to the constructor.

  • tree_limit (Optional[int]) – Explain the output of a subset of the first tree_limit trees in an ensemble model.

  • summarise_result (bool) – This should be set to True only when some of the columns in X represent encoded dimensions of a categorical variable and one single shap value per categorical variable is desired. Both cat_vars_start_idx and cat_vars_enc_dim should be specified as detailed below to allow this.

  • cat_vars_start_idx (Optional[Sequence[int]]) – The start indices of the categorical variables.

  • cat_vars_enc_dim (Optional[Sequence[int]]) – The length of the encoding dimension for each categorical variable.

Return type

Explanation

fit(background_data=None, summarise_background=False, n_background_samples=1000, **kwargs)[source]

This function instantiates an explainer which can then be use to explain instances using the explain method. If no background dataset is passed, the explainer uses the path-dependent feature perturbation algorithm to explain the values. As such, only the model raw output can be explained and this should be reflected by passing model_output=’raw’ when instantiating the explainer. If a background dataset is passed, the interventional feature perturbation algorithm is used. Using this algorithm, probability outputs can also be explained. Additionally, if the model_output=’log_loss’ option is passed to the explainer constructor, then the model loss function can be explained by passing the labels as the y argument to the explain method. A limited number of loss functions are supported, as detailed in the constructor documentation.

Parameters
  • background_data (Union[ndarray, DataFrame, None]) – Data used to estimate feature contributions and baseline values for force plots. The rows of the background data should represent samples and the columns features.

  • summarise_background (Union[bool, str]) – A large background dataset may impact the runtime and memory footprint of the algorithm. By setting this argument to True, only n_background_samples from the provided data are selected. If the categorical_names argument has been passed to the constructor, subsampling of the data is used. Otherwise, shap.kmeans (a wrapper around sklearn.kmeans implementation) is used for selection. If set to ‘auto’, a default of TREE_SHAP_BACKGROUND_WARNING_THRESHOLD samples is selected.

  • n_background_samples (int) – The number of samples to keep in the background dataset if summarise_background=True.

Return type

TreeShap

alibi.explainers.shap_wrappers.rank_by_importance(shap_values, feature_names=None)[source]

Given the shap values estimated for a multi-output model, this function ranks features according to their importance. The feature importance is the average absolute value for a given feature.

Parameters
  • shap_values (List[ndarray]) – Each element corresponds to a samples x features array of shap values corresponding to each model output.

  • feature_names (Union[List[str], Tuple[str], None]) – Each element is the name of the column with the corresponding index in each of the arrays in the shap_values list.

Return type

Dict

Returns

importances

A dictionary of the form:

{
    '0': {'ranked_effect': array([0.2, 0.5, ...]), 'names': ['feat_3', 'feat_5', ...]},
    '1': {'ranked_effect': array([0.3, 0.2, ...]), 'names': ['feat_6', 'feat_1', ...]},
    ...
    'aggregated': {'ranked_effect': array([0.9, 0.7, ...]), 'names': ['feat_3', 'feat_6', ...]}
}

The keys of the first level represent the index of the model output. The feature effects in ranked_effect and the corresponding feature names in names are sorted from highest (most important) to lowest (least important). The values in the aggregated field are obtained by summing the shap values for all the model outputs and then computing the effects. Given an output, the effects are defined as the average magnitude of the shap values across the instances to be explained.

alibi.explainers.shap_wrappers.sum_categories(values, start_idx, enc_feat_dim)[source]

This function is used to reduce specified slices in a two- or three- dimensional tensor.

For two-dimensional values arrays, for each entry in start_idx, the function sums the following k columns where k is the corresponding entry in the enc_feat_dim sequence. The columns whose indices are not in start_idx are left unchanged. This arises when the slices contain the shap values for each dimension of an encoded categorical variable and a single shap value for each variable is desired.

For three-dimensional values arrays, the reduction is applied for each rank 2 subtensor, first along the column dimension and then across the row dimension. This arises when summarising shap interaction values. Each rank 2 tensor is a E x E matrix of shap interaction values, where E is the dimension of the data after one-hot encoding. The result of applying the reduction yields a rank 2 tensor of dimension F x F, where F is the number of features (ie, the feature dimension of the data matrix before encoding). By applying this transformation, a single value describing the interaction of categorical features i and j and a single value describing the intearction of j and i is returned.

Parameters
  • values (ndarray) – A two or three dimensional array to be reduced, as described above.

  • start_idx (Sequence[int]) – The start indices of the columns to be summed.

  • enc_feat_dim (Sequence[int]) – The number of columns to be summed, one for each start index.

Returns

new_values – An array whose columns have been summed according to the entries in start_idx and enc_feat_dim.