Decision models

Some routing decisions depend on instructions rather than resemblance: "refunds and charges, but not pricing questions before a purchase". Example utterances struggle to draw lines like that. A decision model can: it's a small, fast model trained to answer typed questions about a piece of text, with a probability for every answer.

DecisionRouter routes with one. The model is a component you pass in, the way you pass an encoder to an IndexedRouter. The models that ship speak the /v1/systemone API that TypeSafe's Jev introduced, which Laya and Von also implement.

Learn about decision models

New to decision models? Two walkthroughs by James Briggs explain what they are and where they fit:

• Article: Understanding Jev and Where It Fits in the AI Stack covers what a decision model is and how it differs from an LLM, what a calibrated probability gives you, where to use one and where not to, and how open models such as GLiNER2.5 compare, with benchmarks.
• Video: I Tested Jev AI Against LLMs and Open Source (34 min) covers the same ground with live tests. Jump to the three types of decision, calibrated probability, open-source alternatives or GLiNER.

Route with a decision model

from semantic_router import DecisionRouter, Route
from semantic_router.decision_models import TypeSafeDecisionModel

routes = [
    Route(name="billing", description="Money already paid or owed: refunds, charges, invoices."),
    Route(name="technical", description="Something in the product is broken or not working."),
]

router = DecisionRouter(model=TypeSafeDecisionModel(), routes=routes)  # reads JEV_API_KEY

router("why was I charged twice this month?")

[Out]: RouteChoice(name='billing', confidence=1.0, calibrated=True)

There's no encoder and no index. The router turns your routes into one multiple-choice question, sends it with the query, and gets back a probability for every route. Those are already probabilities, so results say calibrated=True without a fit().

The router takes:

The model holds the connection settings:

Use another server

TypeSafeDecisionModel is TypeSafe's Jev with its defaults filled in. For a Laya or Von deployment, or a server of your own that speaks /v1/systemone, use SystemOneDecisionModel and say where it is:

from semantic_router.decision_models import SystemOneDecisionModel

model = SystemOneDecisionModel(name="my-model", base_url="http://localhost:8080")
router = DecisionRouter(model=model, routes=routes)

It sends a key only if you pass api_key, so a local server needs none.

To plug in a different kind of model, subclass DecisionModel and implement aask(state, questions), which returns an answer for each question.

Run a decision model locally

GLiNER2DecisionModel runs GLiNER2.5-Decide, Fastino's open decision model, in your own process. Your queries never leave your infrastructure, and there's no API key.

pip install "semantic-router[local]"  # Python 3.10 to 3.12

from semantic_router import DecisionRouter
from semantic_router.decision_models import GLiNER2DecisionModel

router = DecisionRouter(model=GLiNER2DecisionModel(), routes=routes)
router.fit(X=held_out_queries, y=held_out_labels)

router("why was I charged twice this month?")

The first time it loads, it downloads the weights from Hugging Face (about 1.8 GB). It answers the same choice, noul and score questions as Jev, so aask() works the same way.

On the benchmark (CLINC150, 15 routes), it comes close to Jev:

Three things to know:

• Fit it before you set a threshold. Its raw scores rank the routes well, but they aren't calibrated: a 0.8 isn't right 80% of the time. fit() on held-out queries fixes that.
• Its speed depends on your routes. Every route's name and description go through the model in one pass with the query, so more routes and longer descriptions take longer. On an Apple M4 Max (MPS), a query takes about 50 ms with two short routes and about 210 ms with the benchmark's 15 described routes; on CPU, about 330 ms. Keep descriptions short, or put an IndexedRouter in front of it with a Chain.
• It needs Python 3.12 or earlier, like the other models in the local extra.

The GLiNER chapter of the Jev video covers how it works and how it compares.

Write literal descriptions

The model reads each route's description, and it takes you at your word. Say what belongs in the route and, where it helps, what doesn't:

Route(
    name="billing",
    description="Money already paid or owed: refunds, charges, invoices. Not pricing questions before a purchase.",
)

A route with no description falls back to its first five utterances, joined together. That works, but a written description is usually better.

Ask several questions in one call

A decision model can answer several questions about the same message in one pass, for about the cost of one. aask() sends the message (the state) once, with a dict of named questions, and returns an answer for each.

from semantic_router.decision_models import choice, noul, score

answers = await router.aask(
    state="my card was charged twice and I need it fixed today or I'm cancelling",
    questions={
        "route": router.route_question(),
        "urgent": noul(instructions="Is this time-sensitive?", yes="needs action now", no="can wait"),
        "mood": score(instructions="How upset is the user?", levels=["calm", "annoyed", "angry"]),
    },
)
answers["route"]["choice"], answers["urgent"]["noul"], answers["mood"]["score"]

There are three kinds of question:

router.route_question() is the choice question the router asks for routing, built from your routes. The answers above look like this:

{
  "route": {"type": "choice", "choice": "billing", "confidence": 1.0,
            "probabilities": {"billing": 1.0, "technical": 0.0}},
  "urgent": {"type": "noul", "noul": 0.96},
  "mood": {"type": "score", "score": 1.92, "confidence": 0.88,
           "legend": {"0": "calm", "1": "annoyed", "2": "angry"},
           "probabilities": {"0": 0.0, "1": 0.08, "2": 0.92}}
}

Noul is TypeSafe's name for a yes/no question, short for Bernoulli: one trial, two outcomes. The answer is the probability of yes, so you choose the cut-off.

So one call can route a message, check it against a guardrail and score its urgency, where 0.x needed a separate router for each.

aask() is async only. In a notebook you can await it directly; in a script, run it with asyncio.run().

Latency

A decision model is a network call. Measured against Jev (p50, after warm-up):

• Batch questions into one aask() rather than fanning out calls.
• Warm up at startup. The first call to a model can take many seconds; later ones take a few hundred milliseconds. Make one call when your app starts, never on a user's first request.
• Put a fast router in front. A Chain answers confident queries from an IndexedRouter in milliseconds and only sends the ambiguous ones to the decision model.

Close the session

A DecisionRouter's model keeps a pooled HTTP session, so repeated calls reuse the connection. Close it when you're done. After async calls (acall, aask), prefer:

await router.aclose()

From sync code, router.close() does the same. A sync close() from inside a running event loop raises an error that tells you to await router.aclose() instead.