# White-box and black-box models

Explainer algorithms can be categorised in many ways (see this
table), but perhaps the most
fundamental one is whether they work with **white-box** or **black-box**
models.

**White-box** is a term used for any model that the explainer method can
“look inside” and manipulate arbitrarily. In the context of `alibi`

this category of models corresponds to Python objects that represent
models, for example instances of `sklearn.base.BaseEstimator`

,
`tensorflow.keras.Model`

, `torch.nn.Module`

etc. The exact type of
the white-box model in question enables different white-box explainer
methods. For example, `tensorflow`

and `torch`

models are
*gradient-based* which enables *gradient-based* explainer methods such
as Integrated Gradients 1
whilst various types of tree-based models are supported by
TreeShap.

On the other hand, a **black-box** model describes any model that the
explainer method may not inspect and modify arbitrarily. The only
interaction with the model is via calling its `predict`

function (or
similar) on data and receiving predictions back. In the context of
`alibi`

black-box models have a concrete definition—they are functions
that take in a `numpy`

array representing data and return a `numpy`

array representing a prediction. Using type
hints we can define a
general black-box model (also referred to as a *prediction function*) to
be of type `Callable[[np.ndarray], np.ndarray]`

2. Explainers
that expect black-box models as input are very flexible as *any* type of
function that conforms to the expected type can be explained by
black-box explainers.

Note

In addition to the expected type, black-box models **must** be
compatible with batch prediction. I.e. `alibi`

explainers assume
that the first dimension of the input array is always batch.

Warning

There is currently one exception to the black-box interface: the
`AnchorText`

explainer expects the prediction function to be of
type `Callable[[List[str], np.ndarray]`

, i.e. the model is expected
to work on batches of raw text (here `List[str]`

indicates a batch
of text strings). See this
example for more
information.

## Wrapping white-box models into black-box models

Models in Python all start out as white-box models (i.e. custom Python
objects from some modelling library like `sklearn`

or `tensorflow`

).
However, to be used with explainers that expect a black-box prediction
function, the user has to define a prediction function that conforms to
the black-box definition given above. Here we give a few common examples
and some pointers about creating a black-box prediction function from a
white-box model. In what follows we distinguish between the original
white-box `model`

and the wrapped black-box `predictor`

function.

### Scikit-learn models

All `sklearn`

models expose a `predict`

method that already conforms
to the black-box function interface defined above which makes it easy to
create black-box predictors:

```
predictor = model.predict
explainer = SomeExplainer(predictor, **kwargs)
```

In some cases for classifiers it may be more appropriate to expose the
`predict_proba`

or `decision_function`

method instead of
`predict`

, see an example on ALE for
classifiers.

### Tensorflow models

Tensorflow models (specifically instances of `tensorflow.keras.Model`

)
expose a `predict`

method that takes in `numpy`

arrays and returns
predictions as `numpy`

arrays 3:

```
predictor = model.predict
explainer = SomeExplainer(predictor), **kwargs)
```

### Pytorch models

Pytorch models (specifically instances of `torch.nn.Module`

) expect
and return instances of `torch.Tensor`

from the `forward`

method,
thus we need to do a bit more work to define the `predictor`

black-box
function:

```
model.eval()
@torch.no_grad()
def predictor(X: np.ndarray) -> np.ndarray:
X = torch.as_tensor(X, dtype=dtype, device=device)
return model.forward(X).cpu().numpy()
```

Note that there are a few differences with `tensorflow`

models:

Ensure the model is in the evaluation mode (i.e.,

`model.eval()`

) and that the mode does not change to training (i.e.,`model.train()`

) between consecutive calls to the explainer. Otherwise consider including`model.eval()`

inside the`predictor`

function.Decorate the

`predictor`

with`@torch.no_grad()`

to avoid the computation and storage of the gradients which are not needed.Explicit conversion to a tensor with a specific

`dtype`

. Whilst`tensorflow`

handles this internally when`predict`

is called, for`torch`

we need to do this manually.Explicit device selection for the tensor. This is an important step as

`numpy`

arrays are limited to cpu and if your model is on a gpu it will expect its input tensors to be on a gpu.Explicit conversion of prediction tensor to

`numpy`

. We first send the output to the cpu and then transform into`numpy`

array.

If you are using Pytorch Lightning
to create `torch`

models, then the `dtype`

and `device`

can be
retrieved as attributes of your `LightningModule`

, see
here.

### General models

Given the above examples, the pattern for defining a black-box predictor
from a white-box model is clear: define a `predictor`

function that
manipulates the inputs to and outputs from the underlying model in a way
that conforms to the black-box model interface `alibi`

expects:

```
def predictor(X: np.ndarray) -> np.ndarray:
inp = transform_input(X)
output = model(inp) # or call the model-specific prediction method
output = transform_output(output)
return output
explainer = SomeExplainer(predictor, **kwargs)
```

Here `transform_input`

and `transform_output`

are general
user-defined functions that appropriately transform the input `numpy`

arrays in a format that the model expects and transform the output
predictions into a `numpy`

array so that `predictor`

is an `alibi`

compatible black-box function.

- 1
At the time of writing

`IntegratedGradients`

only supports`tensorflow`

models.- 2
Note that this definition limits black-box models to be

*single-input*and*single-output*which are what most black-box`alibi`

explainers can handle. In the general case the definition can be extended to*multi-input*and*multi-output*models, i.e. taking in and/or returning multiple arrays.- 3
This is in contrast to the

`__call__`

and`call`

methods which expect and return`tensorflow.Tensor`

objects. However, using`__call__`

may be preferable for performance in some cases (this would require transforming inputs and outputs similar to the`torch`

example).