​
BaseTokenizer Objects

class BaseTokenizer()
Abstract Tokenizer class

​
vocab_size

@property
def vocab_size() -> int
Returns the vocabulary size of the tokenizer Returns: int: Vocabulary size of tokenizer

​
config

@property
def config() -> dict
The tokenizer config Returns: dict: dictionary of tokenizer config

​
save

def save(path: str | Path) -> None
Saves the configuration of the tokenizer Saves these files:
  • tokenizer.json: saved configuration of the tokenizer
Arguments:
  • path (str, :class:pathlib.Path“): Path to save the tokenizer to

​
load

@classmethod
def load(cls, path: str | Path) -> "BaseTokenizer"
Returns a :class:bm25_engine.tokenizer.BaseTokenizer object from saved configuration Requires these files:
  • tokenizer.json: saved configuration of the tokenizer
Arguments:
  • path (str, :class:pathlib.Path“): Path to load the tokenizer from
Returns: BaseTokenizer: Configured BaseTokenizer

​
PretrainedTokenizer Objects

class PretrainedTokenizer(BaseTokenizer)
Wrapper for HuggingFace tokenizers, representing a pretrained tokenizer (i.e. bert-base-uncased). Extends the :class:semantic_router.tokenizers.BaseTokenizer class. Arguments:
  • tokenizer (class:tokenizers.Tokenizer“): Binding for HuggingFace Rust tokenizers
  • add_special_tokens (bool): Whether to accept special tokens from the tokenizer (i.e. [PAD])
  • pad (bool): Whether to pad the input to a consistent length (using [PAD] tokens)
  • tokenizer0 (tokenizer1): HuggingFace ID of the model (i.e. tokenizer2)

​
__init__

def __init__(model_ident: str,
             custom_normalizer: Any = None,
             add_special_tokens: bool = False,
             pad: bool = True) -> None
Constructor method

​
vocab_size

@property
def vocab_size()
Returns the vocabulary size of the tokenizer Returns: int: Vocabulary size of tokenizer

​
config

@property
def config() -> dict
The tokenizer config Returns: dict: dictionary of tokenizer config

​
tokenize

def tokenize(texts: str | list[str], pad: bool = True) -> np.ndarray
Tokenizes a string or list of strings into a 2D :class:numpy.ndarray of token ids Arguments:
  • texts (str, list): Texts to be tokenized
  • pad (bool): unused here (configured in the constructor)
Returns: class:numpy.ndarray“: 2D numpy array representing token ids

​
TokenizerFactory Objects

class TokenizerFactory()
Tokenizer factory class

​
get

@staticmethod
def get(type_: str, **tokenizer_kwargs) -> BaseTokenizer
Get a configured :class:bm25_engine.tokenizer.BaseTokenizer Arguments:
  • type_ (str): Tokenizer type to instantiate
  • \**kwargs: kwargs to be passed to Tokenizer constructor
Returns: bm25_engine.tokenizer.BaseTokenizer: Tokenizer