alibi_detect.utils.prediction module

alibi_detect.utils.prediction.tokenize_transformer(x, tokenizer, max_len, backend)[source]

Batch tokenizer for transformer models.

Parameters:

  • x (Union[list, ndarray]) – Batch of instances.

  • tokenizer (Callable) – Tokenizer for model.

  • max_len (int) – Max token length.

  • backend (str) – PyTorch (‘pt’) or TensorFlow (‘tf’) backend.

Return type:

dict

Returns:

Tokenized instances.