semantic_router.llms.openai_compatible

OpenAICompatibleLLM Objects

class OpenAICompatibleLLM(BaseLLM)

LLM for any OpenAI compatible chat completions endpoint, such as LLM routers and proxies. Configure the base URL and the name of the environment variable that holds the API key, for example OpenAICompatibleLLM(name="provider/model", base_url="https://api.example.com/v1", api_key_var_name="EXAMPLE_API_KEY").

Limits inherited from semantic-router 0.x:

• Dynamic routes extract function inputs only when called synchronously. BaseLLM has no async extraction, so an async call returns the route with function_call=None and logs an error.
• A router config file saves only the LLM's class and model name, so a router using OpenAICompatibleLLM can't be reloaded from one: reloading raises, because there is no base_url.

__init__

def __init__(name: Optional[str] = None,
             base_url: Optional[str] = None,
             api_key: Optional[str] = None,
             api_key_var_name: Optional[str] = None,
             temperature: float = 0.01,
             max_tokens: int = 200)

Initialize the OpenAICompatibleLLM.

Arguments:

• name (Optional[str]): The name of the model to use.
• base_url (Optional[str]): The base URL of the OpenAI compatible API.
• api_key (Optional[str]): The API key. Takes precedence over api_key_var_name.
• api_key_var_name (Optional[str]): The name of the environment variable holding the API key, used when api_key is not provided.
• temperature (float): The temperature of the LLM.
• Optional[str]0 (Optional[str]1): The maximum number of tokens to generate.

__call__

def __call__(messages: List[Message]) -> str

Call the OpenAICompatibleLLM.

Arguments:

• messages (List[Message]): The messages to pass to the OpenAICompatibleLLM.

Returns:

str: The response from the OpenAICompatibleLLM.