How Companies are Actually Putting AI Agents into Production with OpenRouter's COO

there as well. And without further ado, I'm going to turn it over to you, Chris. Cool. Hey, everyone. Good to meet you. I'm Chris. As David

Chris. Cool. Hey, everyone. Good to meet you. I'm Chris. As David

said, I'm the co-founder and chief operating officer here at OpenRouter. I'm going to share some data that we have and sort of the story that we've seen play out over, you know, really primarily the past two quarters in terms of some of what we've seen around like the telltale signs that you know, agents are making their way from sort of somewhat theoretical to very practical in a number of companies and just

sort of share some data in our perspective. First, to talk a little bit about OpenRouter, what we do and why we have some of this information. We are the world's largest AI gateway. So we work with about 70 different cloud providers, model labs, sort of you name them, we work with them, right? Including the

major closed source labs like Anthropic, OpenAI, Google, you know, Mistral, Meta, XAI, all of them are close partners of ours, as well as huge numbers of open source NeoClouds, some more esoteric hosting providers, and ultimately give access to hundreds of models through tons of different clouds across lots of geographies with lots of features and

normalize that all down to a single API to a single billing relationship to make it really easy to use the models that you need. We manage quota and capacity for you. We handle all sorts of availability and uptime things. We make sure

for you. We handle all sorts of availability and uptime things. We make sure that tool calling works exactly the same way everywhere. And this has become really, really popular with developers. And so many, well, a few million developers have tried the platform all over the world. And we are a very pure play, sort of like

PLG type company. We're now working our way up in sort of mid-market enterprise and enterprise. but really are very much like bottoms up developer tooling is the way that

enterprise. but really are very much like bottoms up developer tooling is the way that we get adopted. And it's been just a super fun business to get to sit at the intersection of all of these different AI trends and see how LLMs in particular are being used all around the world and how companies are making use of

them internally. So we have a real bird's eye view of a lot of

them internally. So we have a real bird's eye view of a lot of things. We process, literally like a trillion plus tokens a

things. We process, literally like a trillion plus tokens a day. And for anyone who looks at their own internal AI usage, a trillion tokens

day. And for anyone who looks at their own internal AI usage, a trillion tokens a day is like very legitimate volume. It's like a real player on sort of a global scale. We're across all geographies, all countries, and we monitor all this stuff in real time, which is really fun. So we get to see like who's up and who's down. And you may see one purported latency number that this provider

is the fastest or the slowest or the most expensive or the least expensive. And

All of this is available in real time on Open Router. We monitor hundreds or thousands of API endpoints and manage all of this stuff all over the world and work with these companies all the time to say like, hey, there's a quirk in your API or this thing's not behaving the way that we expect. And we have a ton of quantitative data across all of this, but we also have a lot

of qualitative data because we understand what the different use cases are. And so a key belief at Open Router is that you know, as we think about different models and how they're being used and you know, you have Sonnet and Flash and you have free models and open source models and closed source models, like the ultimate benchmark to understand what model should I be using for what is actually what they're actually

being used for in the real world. So if you go to OpenRouter and you look at our rankings, we sample and anonymously categorize a ton of prompts securely on our infrastructure and then aggregate statistics around like which models are being used for which prompts. whether it's marketing automation or programming or

prompts. whether it's marketing automation or programming or like scientific usage or financial advice. And we publish a lot of this data. We

have a lot more of it internally that allows us to make good recommendations and help individuals, companies, enterprises pick models and providers that are relevant for them and make sure that they can like implement the inference that they get across their entire organization. So super fun place to work. We have tons of data internally and we get to like sort of see trends play out on a global scale.

We published a state of AI report at the end of 2025, which you can find on our site, and I think got some good press that explains how some of these trends came together. And happy to share some of those with you. Like

I said, we process trillions of tokens a week. We've been growing pretty quickly and continue to grow. Have really material spend into the hundreds of millions of annual AI spend across all of these clouds. So have some real scale that's been built over the last two years as the company has really performed well. So

it's been really fun. Okay, so what do we see? We get to look at all kinds of cool stuff. So here is a look at the last, I guess this is actually, is this two years? I guess I should have looked at this chart more closely. It looks like it goes back to 2024 of closed versus open source model adoption. By the way, we do have the

chat open. I have it open on my screen. So if you have questions about

chat open. I have it open on my screen. So if you have questions about this stuff or something I said that doesn't quite click, by all means, put it in the chat and I'll do my best to respond. But one of the things that we've been looking at is, and I think a lot of people wonder about, is closed source versus open model adoption. And if you look, you can kind

of see that open source proprietary split has actually been pretty steady for really quite some time. The real story here is the rise of Chinese open-weight models, which is the lighter blue at the bottom of the screen. You can see that that's actually taken dominant share from a lot of US, primarily at least like US open-weight models. Mistral has a number of these that are quite good as well, but

open-weight models. Mistral has a number of these that are quite good as well, but the rise of the Chinese open-weight models we can see sort of coming on strong here. And by the way, these are used heavily, perhaps most heavily by the US

here. And by the way, these are used heavily, perhaps most heavily by the US US firms disproportionately use, in particular, the Quinn family of models quite heavily in agentic flows. And firms like Minimax have come out with sort of agent-focused models. And we'll

flows. And firms like Minimax have come out with sort of agent-focused models. And we'll

talk about sort of the role that those have to play in just a second.

So when we talk about agents, we are talking about tool calls. And for some people, this will be like, yeah, obviously, Chris, I know exactly how this works. But

people will be like, wait, I didn't quite realize exactly how tool calls work. But

when you think about agents, a simple way of thinking about them is it is a large language model in a loop interacting with tools, interacting with the outside world.

And an interesting property of an LLM is that inherently they can't do anything. All

they get to do is they take text in and they output text. That's it.

They don't inherently have the capability to use tools or interact with the outside world.

So what actually happens when we use tool calls with LLMs? Well, the loop looks something like this. a software harness or even a person interacting with an LLM says like, hey, tell me if this is a valuable customer. That actually is my email if you want to email me. You have some tools available like Stripe or Hacks, which is an analytics tool. Obviously, this is being done not in English, but you

can sort of conceptually actually do it in English because LMs are programmable in English.

And the large language model comes back and says, hey, great, please, I think the Stripe tool makes sense. Please call that for me because the LLM can only help with text. It can't call a Stripe tool. It can just say, here's, some here's

with text. It can't call a Stripe tool. It can just say, here's, some here's some text that's a function call please do it for me. The software harness executes it and says, okay, great. Hey, LLM here's the result of that tool call. Here's

the, here's the Stripe data. And the LLM says, okay, great. Based on that, I can see the user has been $500. Sounds valuable. And each of these times it provides what's called the finished reason. So you can see already, this is a loop, right? A software harness is looping over multiple LLM calls. And an agent is doing

right? A software harness is looping over multiple LLM calls. And an agent is doing this more and more times, it is a looping call over LLMs with tool calls involved. And that's the essence of an agent, right? There's often many other subsystems involved,

involved. And that's the essence of an agent, right? There's often many other subsystems involved, but the essence of an agent is looping over LLM calls with tools. And each

of these tool steps outputs a finish reason, which is why did the LLM stop printing text? There's a handful of canonical finish reasons available for all LLMs. They're not

printing text? There's a handful of canonical finish reasons available for all LLMs. They're not always shared, but they're sort of like conceptually similar. OpenRouter normalizes them. But one of them is tool call, where it says like, call a tool for me. It says

the LLM is requesting a tool be used. And then when the agent exits, it will finish with the fission tree and stop, which is like, I believe I'm done with my task. There is nothing else for me to do here. So we process billions and billions of LLM calls every week. And we monitor and normalize finish reasons across billions and billions of LLM calls. So when we look at the world and

look at some major trends, what do we see? Over the last year, this is specifically for Anthropic models because I think they're represented in some ways that you look at in aggregate all models, they might not be, but this is a good representation.

This is a percent of requests to Anthropic models on OpenRouter, which I would argue is representative of a lot of global usage trends. How many of those API calls to Anthropic ended with the LLM requesting to use a tool? And the growth year in one year is like unbelievable. And this is the trend. It is we see

certain models, so open-weight models like Minimax M2, which I can talk about in a second, where the number of the tool call rate might be in the 80%, right?

It might be really high. But for all anthropic models, here's the trend. It's like

sub 5% to well north of 25%, and this is trending up rapidly. And this

is like sort of the exhaust, right? This is the signature, the footprints in the snow of agents being deployed into production. and all the development community and the AI community realizing that like, oh, these can do a lot more than simply answer questions or respond to English text, or they can start to do tool calls. So we

expect to see this trend sort of continue linearly, and for some models to actually go even higher. And so there's like this quantitative information, right, where we can actually see evidence of agents being deployed into production, right? This isn't like, and we're talking about real scale here. This is like a large number of requests. This

isn't people kicking the tires on agents. The other thing that happened, which is really interesting, probably right around July, which is a bit more anecdotal, I think we have some of the BD and sales folks in this call from the open router side, it's like suddenly we started getting questions from customers in

calls and in MSAs about our SLAs and our uptime, right?

And there was actually this like pretty stark transition where, you know, in March of last year, no one was really asking us about this. We were signing commits, we were signing deals. But then all of a sudden in July, people started saying like, well, what's your uptime? Like, how should we think about your uptime relative to the LLM's uptime and like failover? And like, how should we reason about that? Which is

a extremely strong, anecdotal admittedly, but extremely strong indicator that these things have suddenly gone from like groups of companies testing them out to being very much in production. And

if they go down, it starts to matter. So you all, this group is like now asking us about SLAs when we're negotiating these contracts. So the other major trend playing out along, this is a longer time horizon, but is the number of reasoning tokens. So reasoning models, it's sort of hard to believe, but you'll notice that this

tokens. So reasoning models, it's sort of hard to believe, but you'll notice that this is a zero before sort of mid-January of last year. We only just eclipsed to the one-year mark of the release of the first mainstream reasoning model, which was 03. from OpenAI, which was not sort of practical to use in a number of dimensions. It was too expensive and it was slow. But that

was really sort of the first reasoning model. And it's actually only been 12 months.

It's like kind of wild. There were no reasoning models like 12 months ago. No

one was doing it. And now it's 50% of the output tokens that we see are internal reasoning tokens from the models versus like the actual output, sort of the end user or the agentic harness or whatever. So I think a little bit harder for me to predict like how this trends over time But the shift in one year and the adoption of the reasoning models is pretty stark and dramatic. And what

we're seeing as folks are building agents is that they're using a mix of sort of frontier state-of-the-art models, the best stuff that the major labs have put out there, which are in almost all cases reasoning models. They're using those to plan And then they're often

using smaller specialty models to do tool call requests and to execute and look at tools. And so we are seeing that agents that are being deployed today are

at tools. And so we are seeing that agents that are being deployed today are most frequently a mix of multiple models, or if they may not start off as a mix, they may start with like all sort of a frontier model and then very quickly folks start managing down cost by bringing open weight models into the mix, particularly around decision-making and tool calling versus like the overall sort of judgment where they

are less smart from a judgment perspective, but extremely accurate and extremely good with tool use. And so we're seeing these models get mixed and matched in agentic flows

use. And so we're seeing these models get mixed and matched in agentic flows in production in a number of companies. So these reasoning models that are powering the actual logic, and you have these tool calls happening, a lot of what's happening in small models to actually interact with the outside world in highly accurate, fairly reliable ways

that were much less reliable a year ago. I don't see any questions in chat yet, so I'm going to keep going. Okay. So one thing that's really interesting about tool calling and inference and models is that, and I think is generally underappreciated, is that not all inference is created equal. And

by the way, this slide is the current state of the world is different than this slide, so don't read this as like Azure is underperforming parasail, for GPT-OSS-120b. That

is no longer the case. But this is a true representation of what was going on in the world at one point in time. And so what this is showing here is a single model, which is GPT-OSS-120b, pretty popular open source model from OpenAI.

And this shows, against a pretty well-known benchmark, GPQA16, how did the model perform? Well, in the exact same model, which in theory are the exact same model ways, performed fairly differently across a number of providers all hosting the same model. And to be clear, this was done by

our friends over at Artificial Analysis. We've partnered with them on a number of things like this. This was run a few weeks after GPT-OS

like this. This was run a few weeks after GPT-OS S120 was released. And there's a burn in time for the different providers to learn how to best run these models. But it's worth noting that all inference is not created equal. And for a given model, there's a huge number of different ways that

created equal. And for a given model, there's a huge number of different ways that model can actually practically perform in production. And just because it's the same model in the same weights, even in situations, depending on how deep you want to go, like quantizations, let's say they're the same, you can still have meaningful differences in how the inference stack is being executed. and how accurate things are. This is just a raw

intelligence scoring benchmark, right? But this can also happen very much, very specifically in tool calling. So this is again, not like naming and shaming, this is not categorically true,

calling. So this is again, not like naming and shaming, this is not categorically true, this changes all the time, right? But this shows for a particular model at a particular period of time, how often that model chose to use the tools it was given, right? So this is to say like, hey, you know, is,

CC at openrouter.ai, a valuable customer, you can use Stripe if you want, right?

So the exact same model, but hosted by different clouds, decided to use that tool with differing frequency, right? So this is across like many, many millions of requests, but the model's implementations can subtly vary, something really counterintuitive, right? It's very counterintuitive, like why would the exact same model and the exact same smarts choose to use tools

differently in different situations. And there's a whole bunch of possible reasons for it that don't really come down to the model's judgment, but actually the subtleties of how this stuff is implemented. And so one of the things that we work really hard on in OpenRouter is making sure that agents are successful and the tool calls work. So

we get to monitor all this stuff in real time. Now, this is not necessarily a bug. It's not obviously the case that more tool calls are less good than

a bug. It's not obviously the case that more tool calls are less good than few tool calls. But in general, you would expect the variance in performance to be like a little bit tighter here versus so spread out. So we spent a huge amount of time monitoring this sort of thing and actually have dedicated endpoints for open weight models that only route to providers that we have benchmarked in doing a good

job with tool calling. So even amongst tool calling specialty models, we might have 15 different clouds that support a GPT-OSS 120B But some of those clouds tend to be more accurate when calling tools than others for whatever reason. And so we create routing pools of just those endpoints on just those providers to ensure accuracy specifically for agentic

tool callers, right? So there's a whole bunch of stuff that can be done to drive agent accuracy up that is sort of outside of the domain of the actual agent harness and the prompting itself. It's due

to the quality of the inference. And so we publish a bunch of data around this. We have these things called Xacto endpoints that are tool call specialists. We have

this. We have these things called Xacto endpoints that are tool call specialists. We have

things like Nitra endpoints, which are specifically designed to give high throughput, all sorts of like different ways of balancing inference across the needs and price latency, accuracy requirements of our customers, even when in theory, the underlying model is the same. So

it's a super interesting and very dynamic ecosystem. And, you know, net-net as, these agents come online, like we've had to do more and more to make sure that we can optimize inference specifically for agents, right? So we can, you can expressively tell OpenRouter, hey, here are my context lengths that I'm expecting or that I actually have. My latency requirements look like this. Do I need to be geo locked

actually have. My latency requirements look like this. Do I need to be geo locked in certain areas? What kind of data policies might I need? And you can set routing rules around this and we can basically source the inference that you need based on that set of constraints. And then of course do things like automatic failover, and bursty workloads, this is a big problem for a lot of companies. They have an

agent that runs periodically throughout the day with fairly large bursts of token needs. That

makes it almost impossible to buy a commit, right? You can't like buy a committed workload for that sort of thing, but because of our scale, because we have so many customers all doing bursty workloads, we can very often, you know, we don't have infinite capacity, but for most intents and purposes for sort of most companies, we can absorb like spiky workloads very easily. especially when agents are running on like cron jobs

and overnight, we're playing with ideas of like how we can saturate GPUs overnight. So

if you have agents that can run, not during the day, like can you get cheaper inference at night? So we're gonna have some experiments running around that. Again, I

talked about tool calling accuracy and the ability to just simply switch and try new APIs very quickly is like critical in developing agents. We often see like an agent that, a company that's developed an agent Let's say it's for doing some sort of response to inbound leads. You're like, geez, this thing is just not smart enough. It's

not accurate enough to actually put into production. It's like close, but it's only right 95% of the time. That's just not good enough. And you might be building on an opening eye model. And then Geno 9.3 comes out in December and you say, oh, geez, I wonder if this might help. Well, it's non-trivial to switch between OpenAI and Gemini 3 if your company has a relationship with OpenAI or if you don't

have a Google account spun up. So OpenRouter really helps with agent development in that way as well because you can immediately try a new frontier model to see if you can get the error rates down enough or the false positive rate down enough so that you can actually release this to production. So I'd say the two big use cases we see for switching models is one, a new frontier model comes out

and you're like, can this now solve my problem? It couldn't before. And two is I have an agent that's running and it's working, but I'm using frontier models anywhere, everywhere. Can I down clutch to an open weight model now that I understand my

everywhere. Can I down clutch to an open weight model now that I understand my use case, get the same results, but with much lower cost. So all kinds of things about budget management, guardrails, all kinds of things around logging and integration. So basically

our goal is to be the best place to source orchestrate and control in a way that you feel good about all of the needs of your inference across the company, make sure that all your agents are happy and like getting the tokens they need to run successfully. So that's it for me. Happy to open up to questions. Hey, Chris, that was really, that was super interesting.

to questions. Hey, Chris, that was really, that was super interesting.

Thank you. One question here that I got from the live stream, which I thought was interesting from your vantage point. across hundreds of models and providers, what's the biggest mistake founders make when building agents today?

The biggest mistake founders make when building agents?

That's a good question. I mean, I think, I think, you know, look, obviously I'm gonna like talk my own book a little bit here, right? I think

that like building for optionality is sort of the key. I think avoiding lock-in, because I think we would all agree that it's extremely hard to predict what we're going to need in 12 months and where that influence will come from and what kind of models we might need. And so what we often, like situating where we often show up where people have built something in a stack that is hard to move

to other models or other technologies, and they realize, geez, we really need some more flexibility here. So introduce us as a gateway layer in order to be able to

flexibility here. So introduce us as a gateway layer in order to be able to quickly try out new things, different agents or different models or different clouds without having to rewrite your code. One of the things that I think is unintuitive to a lot of people who do open router from a dev first standpoint is that we are most often graduated into, not graduated out of. Most companies start on with a

first party relationship somewhere, often like burning down cloud credits they've received as part of a startup program and then realize, oh shoot, I need failover. And then, oh, shoot, I need better monitoring. And then you're like, oh no, I need a different model.

It's not available on this cloud, right? And that's sort of when we come in.

And so I just think knowing that your agent's gonna be unstable. And so how do you plan for the future accurately and make technology choices that preserve optionality against a number of different dimensions? Okay, yeah, super interesting. Question,

how should go-to-market teams think about using OpenRouter?

Well, I think, yeah, I mean, one of the, go to marketing specifically, one of the big kind of open, I don't know if it's an open question exactly, there's sort of, there's obviously going to be a lot of vertically built agents that are specialized SaaS, right? There may be people on here, certainly there's a lot of people on here who are buying that stuff, there may be a lot of people on

here who are also building those things, right? But you're like, hey, Harvey is like the, agentic thing for lawyers, right? And you have, you know, agentic doctors and agentic, you know, customer support and there's all these other verticals. So I

think in, for go-to-market teams, there's gonna, there are, and they're going to be, and they're gonna merge some winners who are really great verticalized go-to-market agentic stuff that's like packaged up and really good wrappers. But the count, the sort of counterweight to that is that it's obviously one of the other major trends here that's like, way easier to write custom software these days, right? And it's actually nearly trivial to, in some

cases, write your own in-house agents. So the question is, how will the market sort of settle? And is it the case that, look, a four-person GTM team who has some tech jobs but not engineers can just kind of build the agent they need through cloud code I need an agent to do this thing. I'm going to use a framework like crew or master or AWS agent core,

thing. I'm going to use a framework like crew or master or AWS agent core, whatever, or just roll it from scratch and do it yourself. So I think there's a stronger argument these days, especially in go-to-market stuff to maybe try and roll it yourself and see how far you get because those tools are just becoming incredibly powerful.

And so I think just being unafraid to like just build your own agent internally using an open source framework or using a hosted framework and seeing how far you get before going to sort of like bundled SaaS stuff because go-to-market is really nuanced that like in many ways is the way you touch your customers and it is the identity of the company it is the product in its own right and so

merits a lot of sort of custom thought and ability to have full control over it. Very cool. There's a question here in the chat I don't know if you

it. Very cool. There's a question here in the chat I don't know if you see it about NVIDIA and Nemo yeah. We do host the Nemo models. If you

go to openrouter.ai slash models, you can see what we host. Something interesting about the Nemo models, I think, you're going to have to double check this because if I'm wrong, this would be a bad thing to be wrong about. The Nemo models are interesting because this is like a wonky AI thing, but they allow distillation, which is kind of interesting. Distillation is when you pull a bunch of information out of a

model through a huge series of prompts and then train a different model on those outputs. That is very much not allowed for almost all models. But interestingly, NVIDIA, Obviously

outputs. That is very much not allowed for almost all models. But interestingly, NVIDIA, Obviously they want more people to build more models. So they've built Nemo and Nemotron and allow people to distill them to build their own in-house models, which is like a kind of interesting use case around those specifically.

Okay. Very cool. Oh, what do we got here? Hold on. Oh, got a thank you from Michael. You got it. What do you think founders should optimize for first? Model quality? Inference routing or observability?

None of those. I mean, look, the key to getting an agent really works. It's like, look, go, like, use open router. Like, if you're starting from scratch,

works. It's like, look, go, like, use open router. Like, if you're starting from scratch, my advice would be use open router, pick a frontier state-of-the-art model. So you're like, if it's going to fail, it's not because you're like, the model's not smart enough, you shouldn't pick somebody smarter, right? And then, you know, the... you're going to need some good traceability understanding of like where agents are falling down. But then, you know,

largely this is like still a lot of the magic is in the context engineering and getting the right information that's thing to use in the context of the prompts.

So I would think about the routing and the model piece is something that's like, look, get the best stuff in market from a routing standpoint. And then

start development there if you're coming to this new, and basically take those factors out of play as a reason to fail. That's not a good reason to fail. And

then given the best quality inference and the best quality routing, the best quality model you can, see if your use case is possible and see how well you can do and build from there. But if you do a great job building and you use like crappy quality inference and you draw a false negative, right, this could have worked, but it didn't for sort of more mechanical reasons than is really necessary. Yeah,

that's great advice. I think we're just about at time here, Chris. Actually, there's one last question if you want to address it in the chat. I think it's a couple of things. I mean, it really varies. It depends on the way the enterprise is managed. Very often, there's a lot of just like and large enterprises, obviously,

is managed. Very often, there's a lot of just like and large enterprises, obviously, red tape and just getting to the inference quality and permissions and throughput and bandwidth and budget that you need to actually run this stuff. The other really big one that's a very legitimate concern is like what's going into this stuff and where, even if you're working with like pretty good, I'll wrap in 30 seconds, I promise, pretty

good, like reputable inference providers, like where are their data centers? Do they actually own the GPUs or do they have like GPUs that are leased to a number of data centers in different places and where's the decryption happening and like, that data policy stuff really matters, especially like enterprises handling sensitive data. So figuring out how to get a good control environment over where your prompts are going is critical. And we've made

huge investments in being able to do that. And there's like a ton more to go there. But I think in terms of broad rollout is often the case that

go there. But I think in terms of broad rollout is often the case that like, this is sometimes the most sensitive data in your company and you're sort of just like, it's going to AI, right? And it's like making companies comfortable with exactly what's happening and that that environment is controlled is like, is really, really important. Yeah.

Yeah, that's great. Well, Chris, thank you so much for a great session. And your

email for anyone who wants to reach you? CC at openrouter.ai. Okay, perfect.

CC at openrouter.ai. And thank you all for joining today's AI Day session. I think there's one more session on tap. And Chris, thanks so much for

session. I think there's one more session on tap. And Chris, thanks so much for great content. You got it. Thanks. All right. Bye, everyone.

great content. You got it. Thanks. All right. Bye, everyone.