Back to Blog

How long until model routing is just… how AI works?

TL;DR

Is model routing already happening today, or is it still coming?
It is already happening. Platforms like OpenRouter, Snowflake, and Databricks route requests to different models right now, but plenty of companies still hardcode their apps to one fixed model.

When will model routing become the default, not a choice?
Most analysts expect static, single-model API selection to be mostly replaced by dynamic routing control planes by 2027 to 2028, as the cost gap between models keeps widening.

What is the difference between model routing and provider routing?
Model routing decides which model architecture answers a prompt, such as a small open model or a frontier reasoning model. Provider routing decides which host runs that model, based on price, speed, and uptime at that moment.

Will I have to build routing myself, or will platforms just do it for me?
Increasingly, platforms will do it for you. Snowflake’s Cortex AI Gateway and Databricks’ Unity AI Gateway already route model calls automatically inside the customer’s own data environment.

How much does it cost to keep pinning an app to one model?
A frontier reasoning model can cost several hundred times more per token than a specialized open-weight model. Preference-trained routers like RouteLLM have already cut token costs by 70 to 85 percent while keeping about 95 percent of a frontier model’s answer quality.

Which companies are already baking routing into their platforms?
Snowflake, Databricks, NVIDIA, OpenRouter, and Martian are the furthest along, each building routing into a different layer of the stack, from data governance to raw hardware.

Will single-model API calls disappear completely?
Not entirely, but they are expected to become the exception rather than the default, reserved for narrow cases where a company deliberately wants one model and nothing else.

Right now, routing is still something you have to choose

Most companies built their first AI features the simple way: pick a model, wire an API key into the app, done. Nobody routes anything. Whatever model you picked on day one is the model that answers every request, forever, until an engineer manually swaps it.

That is still how most AI features work today. It is also the part of this story that is about to change.

The reason it is changing comes down to money. A frontier reasoning model can cost several hundred times more per token than a lightweight, specialized open-weight model. Send a basic entity extraction task to a top-tier model and you are paying premium prices for a job a much cheaper model handles just as well. Send a complex coding task to a small model and you get a higher failure rate instead. A single fixed model is the wrong tool for at least some share of any company’s traffic, and agentic systems make the math worse, since one agent loop can fire off dozens or hundreds of model calls to finish a single task.

That gap is exactly what is pulling routing out of “nice to have” territory and into “how did we ever ship without this.”

Two decisions that are quietly becoming automatic

Ask ten people what “AI routing” means and most will describe one decision. There are actually two, and both are on track to stop being something a developer thinks about at all.

  • Model routing answers: which model architecture should handle this request? It weighs task complexity, how much context the prompt needs, how much reasoning depth is required, and the cost ceiling for that call.
  • Provider routing answers: once we know which model to use, where should it actually run? It weighs real-time price per token, time-to-first-token, uptime, and how reliably that host executes tool calls.

OpenRouter already handles both decisions in one platform. Snowflake leans almost entirely on model routing, matched to its own data governance rules. NVIDIA pushes both decisions down into the hardware layer itself, through tools like NeMo Switchyard. None of these require a developer to manually pick a model per request. That is the part worth paying attention to. The decision still gets made, it just stops being a human’s job to make it.

How the routers already decide, before anyone asks them to

Different routing products use very different math to make the same basic call, and they are already running in production, not still in a lab.

RouteLLM, the open-source framework out of UC Berkeley’s LMSYS group and Anyscale, trains its router on human preference data, the same kind of data behind Chatbot Arena rankings. It learns to predict whether a cheap model’s answer will satisfy a human judge, or whether the request needs to escalate to a stronger model. On benchmarks like MT-Bench and GSM8K, that approach has cut token costs by 70 to 85 percent while holding onto roughly 95 percent of a frontier model’s score.

Martian takes a different route. It builds a model of each model, converting weights and transformer behavior into a shared vector space using mechanistic interpretability, then predicts how well a given prompt will do on a given model before ever running it.

At the provider layer, OpenRouter runs live telemetry through variants like :exacto and :nitro, which continuously score hosting providers on tool-calling accuracy and latency variance. Providers that fail tool calls too often or show unstable response times get deprioritized automatically. That kind of monitoring has reportedly cut tool-call error rates on open models like DeepSeek V3.2 and GLM-5 by up to 88 percent. None of that runs by hand. It runs continuously, in the background, which is the whole point.

The platforms already building this in as a default, not an add-on

This is the strongest evidence for how close “native everywhere” already is. A handful of companies are not experimenting with routing, they are shipping it as a built-in feature customers do not have to configure:

  • Snowflake Cortex AI Gateway: routes model calls inside the customer’s governed data boundary automatically, with a reported 3x token efficiency gain in agentic pipelines.
  • Databricks Unity AI Gateway: centralized model catalog, fallback policies, and Model Context Protocol integration built into the platform.
  • NVIDIA NeMo Switchyard: an open-source, Apache 2.0 proxy library that pushes routing decisions into the hardware execution layer itself.
  • OpenRouter: independent aggregator giving developers access to over 400 models across more than 80 hosting providers, with live telemetry-based routing running by default.
  • Martian: intent and capability-based routing using interpretability-driven model mapping, with no manual model picking required.
  • Portkey: open-source gateway with semantic caching, load balancing, and automated failover across 200-plus models.

Every one of these treats “pick the right model” as infrastructure, not a decision a developer makes at build time. That is a meaningful signal about where the timeline actually is.

Open source is what is setting the clock

Routing would still be a minor feature if only two or three proprietary frontier models existed. There would not be much to route between, and the timeline for “everywhere by default” would stretch out for years.

That is not the market anymore. Meta’s Llama family, DeepSeek’s V3 and R1 models, Qwen, GLM, and Mixtral have created a genuinely fragmented model landscape, with real differences in size, fine-tuning, and serving optimization. That fragmentation is what makes routing valuable in the first place, and it feeds itself. More specialized open models widen the price-performance gap, a wider gap makes routing pay off faster, and better routing economics push more traffic, and more investment, toward open-weight models. Snowflake’s decision to pair its Cortex AI Gateway launch with native access to DeepSeek-V4-Flash and GLM-5.3 is a direct example of that loop speeding up, not slowing down.

Every major player is racing to make routing invisible on their own terms

The question is not whether routing becomes native everywhere. It is whose version of “native” wins, and that race is what is actually setting the pace of adoption.

Cloud hyperscalers like AWS, Microsoft Azure, and Google Cloud face a real risk of their hosted models becoming interchangeable, commodity endpoints if neutral routers become the default developer interface first. Their defense is building their own gateways, like AWS Bedrock Routing and Azure AI Gateway, so routing becomes native to their cloud specifically, not to a neutral third party.

Snowflake and Databricks are moving fastest of all, because they already sit on top of the customer’s actual data and access policies. Routing is already a background feature of a platform companies trust with sensitive information, which shortens their path to “default” considerably.

NVIDIA benefits almost no matter which version of native wins. Cheaper, more efficient inference lowers the cost of adoption, which raises total transaction volume, and NVIDIA’s NeMo Switchyard keeps its hardware stack positioned underneath all of it regardless of which gateway a customer picks.

Frontier model vendors like OpenAI and Anthropic have the most reason to slow this down. Neutral routers exist specifically to send routine traffic away from their most expensive endpoints. Their counter-move is building their own internal routing and Mixture-of-Experts architecture behind a single branded API name, so routing becomes native to their product too, just invisible to the customer.

Independent routers like OpenRouter, Martian, and Portkey are racing against all of the above, since hyperscalers and data platforms can eventually build native routing directly into their own stacks. Their advantage right now is neutrality and the operational telemetry that comes from processing enormous volumes of cross-provider traffic, which buys them time but not forever.

So, how long until this is just how AI works?

Most forecasts land on the same window: 2027 to 2028 for static, single-model API selection to be mostly replaced by dynamic, multi-vector routing control planes. Instead of a simple cost-versus-quality tradeoff, routing decisions are expanding to weigh six variables at once: quality target, cost ceiling, latency budget, tool reliability, context volume, and governance requirements like Zero Data Retention and regional data boundaries.

Enterprise infrastructure teams are increasingly expected to support Bring Your Own Keys arrangements, keeping direct commercial agreements with model providers while an external routing layer handles failover and performance monitoring.

Agentic systems are pushing the timeline even faster with patterns like Mixture of Models and Agents. A complex task gets broken into sub-tasks first, then split across models by capability: a lightweight model handles structure parsing, a mid-tier model handles code generation, and a high-tier reasoning model only steps in for edge cases that actually need it. Once agents are the ones making the calls instead of humans, “should a person pick the model” stops being a real question at all.

Building the team for whichever timeline you are on

Whether a company is already running on a platform that routes automatically, or still hardcoding calls to one model, the gap between those two states is closing fast, and someone has to manage that transition. Enterprises adopting model and provider routing need engineers who understand inference economics, AI gateway architecture, and agentic system design, and those people are genuinely hard to find right now because the discipline is so new.

This is exactly the kind of specialized hiring challenge STEM Search Group was built to solve. A company might need an early AI infrastructure hire who can evaluate routing platforms. It might need a platform engineer who can build an internal gateway, or a technical leader who can set governance policy across a multi-model stack. STEM Search Group brings the technical depth and proactive recruiting approach needed to find candidates who are not sitting on a job board waiting to be found. The platforms will keep getting better at making routing invisible. Finding the people who can implement, govern, and improve it is still very much a human problem.

Sources

Recruiting redefined; built for high-tech,
high-growth teams

Hire Talent
Job Board