​
BaseTokenizer Objects

class BaseTokenizer()

Abstract Tokenizer class

​
vocab_size

@property
def vocab_size() -> int

Returns the vocabulary size of the tokenizer

Returns:

int: Vocabulary size of tokenizer

​
config

@property
def config() -> dict

The tokenizer config

Returns:

dict: dictionary of tokenizer config

​
save

def save(path: str | Path) -> None

Saves the configuration of the tokenizer

Saves these files:

  • tokenizer.json: saved configuration of the tokenizer

Arguments:

  • path (str, :class:pathlib.Path“): Path to save the tokenizer to

​
load

@classmethod
def load(cls, path: str | Path) -> "BaseTokenizer"

Returns a :class:bm25_engine.tokenizer.BaseTokenizer object from saved configuration

Requires these files:

  • tokenizer.json: saved configuration of the tokenizer

Arguments:

  • path (str, :class:pathlib.Path“): Path to load the tokenizer from

Returns:

BaseTokenizer: Configured BaseTokenizer

​
PretrainedTokenizer Objects

class PretrainedTokenizer(BaseTokenizer)

Wrapper for HuggingFace tokenizers, representing a pretrained tokenizer (i.e. bert-base-uncased).

Extends the :class:semantic_router.tokenizers.BaseTokenizer class.

Arguments:

  • tokenizer (class:tokenizers.Tokenizer“): Binding for HuggingFace Rust tokenizers
  • add_special_tokens (bool): Whether to accept special tokens from the tokenizer (i.e. [PAD])
  • pad (bool): Whether to pad the input to a consistent length (using [PAD] tokens)
  • tokenizer0 (tokenizer1): HuggingFace ID of the model (i.e. tokenizer2)

​
__init__

def __init__(model_ident: str,
             custom_normalizer: Any = None,
             add_special_tokens: bool = False,
             pad: bool = True) -> None

Constructor method

​
vocab_size

@property
def vocab_size()

Returns the vocabulary size of the tokenizer

Returns:

int: Vocabulary size of tokenizer

​
config

@property
def config() -> dict

The tokenizer config

Returns:

dict: dictionary of tokenizer config

​
tokenize

def tokenize(texts: str | list[str], pad: bool = True) -> np.ndarray

Tokenizes a string or list of strings into a 2D :class:numpy.ndarray of token ids

Arguments:

  • texts (str, list): Texts to be tokenized
  • pad (bool): unused here (configured in the constructor)

Returns:

class:numpy.ndarray“: 2D numpy array representing token ids

​
TokenizerFactory Objects

class TokenizerFactory()

Tokenizer factory class

​
get

@staticmethod
def get(type_: str, **tokenizer_kwargs) -> BaseTokenizer

Get a configured :class:bm25_engine.tokenizer.BaseTokenizer

Arguments:

  • type_ (str): Tokenizer type to instantiate
  • \**kwargs: kwargs to be passed to Tokenizer constructor

Returns:

bm25_engine.tokenizer.BaseTokenizer: Tokenizer