> ## Documentation Index
> Fetch the complete documentation index at: https://docs.aurelio.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# semantic_router.tokenizers

## BaseTokenizer Objects

```python theme={null}
class BaseTokenizer()
```

Abstract Tokenizer class

#### vocab\_size

```python theme={null}
@property
def vocab_size() -> int
```

Returns the vocabulary size of the tokenizer

**Returns**:

`int`: Vocabulary size of tokenizer

#### config

```python theme={null}
@property
def config() -> dict
```

The tokenizer config

**Returns**:

`dict`: dictionary of tokenizer config

#### save

```python theme={null}
def save(path: str | Path) -> None
```

Saves the configuration of the tokenizer

Saves these files:

* tokenizer.json: saved configuration of the tokenizer

**Arguments**:

* `path` (`str, :class:`pathlib.Path\`\`): Path to save the tokenizer to

#### load

```python theme={null}
@classmethod
def load(cls, path: str | Path) -> "BaseTokenizer"
```

Returns a :class:`bm25_engine.tokenizer.BaseTokenizer` object from saved configuration

Requires these files:

* tokenizer.json: saved configuration of the tokenizer

**Arguments**:

* `path` (`str, :class:`pathlib.Path\`\`): Path to load the tokenizer from

**Returns**:

`BaseTokenizer`: Configured BaseTokenizer

## PretrainedTokenizer Objects

```python theme={null}
class PretrainedTokenizer(BaseTokenizer)
```

Wrapper for HuggingFace tokenizers, representing a pretrained tokenizer (i.e. bert-base-uncased).

Extends the :class:`semantic_router.tokenizers.BaseTokenizer` class.

**Arguments**:

* `tokenizer` (`class:`tokenizers.Tokenizer\`\`): Binding for HuggingFace Rust tokenizers
* `add_special_tokens` (`bool`): Whether to accept special tokens from the tokenizer (i.e. `[PAD]`)
* `pad` (`bool`): Whether to pad the input to a consistent length (using `[PAD]` tokens)
* `tokenizer`0 (`tokenizer`1): HuggingFace ID of the model (i.e. `tokenizer`2)

#### \_\_init\_\_

```python theme={null}
def __init__(model_ident: str,
             custom_normalizer: Any = None,
             add_special_tokens: bool = False,
             pad: bool = True) -> None
```

Constructor method

#### vocab\_size

```python theme={null}
@property
def vocab_size()
```

Returns the vocabulary size of the tokenizer

**Returns**:

`int`: Vocabulary size of tokenizer

#### config

```python theme={null}
@property
def config() -> dict
```

The tokenizer config

**Returns**:

`dict`: dictionary of tokenizer config

#### tokenize

```python theme={null}
def tokenize(texts: str | list[str], pad: bool = True) -> np.ndarray
```

Tokenizes a string or list of strings into a 2D :class:`numpy.ndarray` of token ids

**Arguments**:

* `texts` (`str, list`): Texts to be tokenized
* `pad` (`bool`): unused here (configured in the constructor)

**Returns**:

`class:`numpy.ndarray\`\`: 2D numpy array representing token ids

## TokenizerFactory Objects

```python theme={null}
class TokenizerFactory()
```

Tokenizer factory class

#### get

```python theme={null}
@staticmethod
def get(type_: str, **tokenizer_kwargs) -> BaseTokenizer
```

Get a configured :class:`bm25_engine.tokenizer.BaseTokenizer`

**Arguments**:

* `type_` (`str`): Tokenizer type to instantiate
* `\**kwargs`: kwargs to be passed to Tokenizer constructor

**Returns**:

`bm25_engine.tokenizer.BaseTokenizer`: Tokenizer