- class alibi_detect.models.tensorflow.embedding.TransformerEmbedding(model_name_or_path, embedding_type, layers=None)[source]
- __init__(model_name_or_path, embedding_type, layers=None)[source]
Extract text embeddings from transformer models.
str) – Name of or path to the model.
Type of embedding to extract. Needs to be one of pooler_output, last_hidden_state, hidden_state or hidden_state_cls.
From the HuggingFace documentation:
Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pre-training. This output is usually not a good summary of the semantic content of the input, you’re often better with averaging or pooling the sequence of hidden-states for the whole input sequence.
Sequence of hidden-states at the output of the last layer of the model.
Hidden states of the model at the output of each layer.
See hidden_state but use the CLS token output.
int]]) – If “hidden_state” or “hidden_state_cls” is used as embedding type, layers has to be a list with int’s referring to the hidden layers used to extract the embedding.
Extract embeddings from hidden attention state layers.
- Return type:
Tensor with embeddings.