Building Enterprise Ready AI with Koog | Vadim Briliantov

Hi everyone.

Thanks for joining my session about why most AI agents never scale and how you can build the ones that do.

My name is Vadim Briliantov. I'm a

technical lead of Kook and some of you might remember that exactly 1 year ago on the same Kotlin Conf, on the same stage, I have presented Kook, the Kotlin

AI agent framework.

And I also promised that the year 2025 will be the year of agentic AI.

Please raise your hand if you actually noticed lots of new AI agents coming up at the news.

Great.

So indeed lots of new startups, experiments, demos, and definitely hype.

But what about this year?

Well, I think that this year, 2026, will be the year when AI agents would finally enter production.

And we have to be prepared, right?

So, let's take a look at the retrospective for the Kook framework. Where we have been so far? So, as I said, May 2020

Sorry, May 2025, 22nd, exactly 1 year ago, we have released the first version of Kook.

Later on, we also added persistence so that Kook agents can also be durable and fault tolerant.

Then we also integrated with different observability tooling so that you can also monitor and debug your running agents.

Later on, at the beginning of this year, on the latest Java One, we actually also presented Kook for Java. So, please

don't forget to tell about that to your Java colleagues. They have to be also

Java colleagues. They have to be also happy about that.

Then, um 1 month ago, on the Spring I/O we also launched full Spring Boot and Spring AI integrations for Kook.

And the last thing is we just uh launched the local model on-device support for Kook so that you can run agents on Android devices even without

the internet.

All of that means that so far Kook has matured.

But today is a very special day for Kook and I want to take a moment here on the stage to say happy birthday because Kook turns 1 year.

And as you probably already uh know from the keynote, any birthday doesn't uh is not like really a birthday without presents and we also have one which is

our 1.0 release. Isn't it the best way to celebrate the birthday of the framework right?

Um and now let's get back to the actual question. Why most AI agents never

question. Why most AI agents never scale?

Well, you probably expect that I would just tell you and answer this question, but let's first ask Kook itself.

Um we've just uh built a demo uh local Android and iOS app which is called Ask Kook.

And it can tell something about the Kotlin Conf, Kotlin AI, and Kook, of course.

So, Hey Kook, um tell me why most AI agents never scale.

So, Most AI agents never scale because they are built in Python.

Ah.

That's a good one.

But tech stack and platform choices do matter once your cute demo agent meets real users.

And you know what? I would

Indeed, uh you have to build agents where your enterprise already lives, whether it's JVM, JavaScript Android iOS or

WebAssembly.

You don't have to switch the stack.

But, that's not just it.

Agents don't just scale because you are build them in different platform.

And there are other reasons.

So, if you ask me, but not the AI, why most AI agents uh never scale, I would also tell you that, well, in real scenarios, you need control.

And you probably know that there's been a lot of hype about the AGI and these magical black boxes that just work and

can solve any possible problem on Earth.

But, the hype is almost over. You

probably noticed that even OpenAI recently released uh the agent kit so that you can build specialized agent workflows.

So, naive prompting never work in real system and you can never trust them. You

have to be able to control what your agent is doing.

Um next Next thing is uh well, money.

Who loves money?

I love money. We all love money.

And I'm pretty sure that uh many companies also love to earn money and to not spend too much money, right?

And that's especially true when you are building AI applications because, well, they come with a cost.

Moreover, there are context limits for most of the models and it's pretty easy to hit them. And if you do, your agent doesn't work.

You cannot just deploy it to production and be sure that it's not going to let you down.

Um last but not least, um as I already told you in the keynote, agents are long-running processes and that they might fail and you have to be able to recover from any crashes.

Otherwise, you cannot call them scalable.

So, uh, the plan for today is we're going to build a simple agent.

And that's it.

Well, of course no. We also going to develop the tools, integrate into a backend. It's going to be a spring

backend. It's going to be a spring backend, but you can also integrate it with any other backend.

Uh, then we going to uh, let our agents remember the previous conversations.

Why? Because they are not just demos in a Python or Jupiter or Kotlin notebook.

They are real systems. They need to be able to interact with the users and remember everything about the previous conversations. And then we're going to

conversations. And then we're going to solve the first problem of the scalability, which is no control. We're

going to add control to our agents. Then

we're going to also solve the second problem and make them fault tolerant.

We're going to also integrate with uh, monitoring tools. And last but not

monitoring tools. And last but not least, we're going to solve the money problem.

So, let's go.

Uh, let's build a simple agent with Cook. Just a uh,

Cook. Just a uh, brief uh, technical introduction. So,

what is an agent? An agent is actually a three-component system.

Um, it's of course the agent itself, the environment, and models, one or many.

From the user's perspective, of course, it's very simple black box. The user

would just click a button or send a request and get back the response.

Pretty simple, right? But under the hood, the whole process is happening.

So, the agent will be calling models and the models will be coming back with model responses that are either assistant messages or tool calls.

If the model decides to call a tool, then the agent will go to your environment, actually call the tool, and come back with the tool result.

But that's not it. The process will continue again and again and again until the strategy powering the logic of your agent will decide to stop.

And that's only when your user will see the final result. Until then, the agent is still working.

Um any ideas what's the environment for you if you're building a Kotlin backend or mobile application?

It's actually your Kotlin application.

It's going to be the environment for your agent. The agent will be calling

your agent. The agent will be calling tools from your app.

And um to build a simple agent is cook. In cook

you just have to explain the picture on the left with the code on the right.

First, register the tools from your app.

We're going to go through that.

Second, optionally define a strategy, the controllable logic for your agent.

Then configure the agent to work with a specific model to execute the specific prompt and provide any other limitations and parameter like maximum iterations, for example.

And of course, we have to connect to models. For example, OpenAI, Anthropic,

models. For example, OpenAI, Anthropic, Llama, Mistral, whatever models you prefer, one or many.

And that's it.

So, um if you want to build a simple banking backend, and that's by the way what we're going to be building today, uh it's pretty easy. Just AI agent, tool

registry, register my banking tools, your banking assistant, that's going to be the system prompt. I'm going to work with GPT 5.4 Pro just because I want,

and connect to OpenAI prompt executor.

That's it. 1 2 3 4 lines of code and your banking application becomes instantly agentic.

And you don't even need to provide a strategy.

And that's amazing, great, perfect agentic demo.

Not the real app.

Um but you can run it. You can ask the agent to send $100 to Mike.

And probably the agent will call some tools like sending money in your banking app, get your account, get Mike's account, and Mike will be $100 richer.

Hopefully, if it works.

And it will during the demo.

Um and by the way, um no RPC is required. You don't need MCP to connect to some other external service.

Banking tools are the tools from your current Kotlin app that you are developing, and the agent is working just right there in your Kotlin app.

Um so, we just built a very simple agent, and now we are ready to develop tools in Kotlin. So, let's do that.

Let's ask um Kook, "How can I develop tools?" And it will tell you.

The tools are basically any Kotlin functions or method, and you just have to put a tool annotation on top.

And also optionally, you can explain what the tool is doing to the LM using another description.

So, let's do that. But first, we have to understand that well well, you define the methods, the model will construct the arguments. You cannot control that.

the arguments. You cannot control that.

You don't know what will be provided.

And you have to also write lots of checks so that the tools are not uh unreliable.

Um but you can control the outputs and the logic for the tools because you are the developer, while the model will control the order of the execution.

And that's very important because that's true only if you don't provide a strategy.

You might think that um the banking agent will first get the list of contacts, and then send it to to Mike the $100.

But maybe no, maybe it will not get the list of contacts and just send $100 to some random other person.

Um so, let's develop tools. Um

let's say that I have a send email function in Kotlin in my banking app with two parameters. How to make it a tool? Just call it a tool. Pretty easy,

tool? Just call it a tool. Pretty easy,

right? But what is the format for the email and what does it return?

Even human cannot guess just from this signature of the function. So, you have to explain to the human or to the model.

You can optionally provide descriptions.

And now it's clear that it sends email and gets the reply. And the email is, well, just the email.

Um but you can develop much more tools.

Let's connect our agent with your backend. So, the agent will be able to

backend. So, the agent will be able to uh read something from the database. And

for that, we're going to define a tool set.

Account reading tools.

Tool set is just a marker interface from Cook. You just have to inherit any class

Cook. You just have to inherit any class from it, and that's it. And inside you can define the methods.

Uh but first, remember that I told you that well, agents will construct all the arguments for you.

Do you want to you the agent to use some other account, not yours? Well,

probably it's not reliable enough. All

the secrets, user IDs, any sensitive information should never be exposed to the model. You have to inject it from

the model. You have to inject it from your application so that you control that. And the model will just see that.

that. And the model will just see that.

Start date, status, and other arguments, but not the user ID.

So, I'm going to define the tools like the agent will be able to get the latest transactions, account balance, and account information.

But that's boring. We don't want to build yet just another child support. We

want our agents to be productive and proactive so that they can help the user to fix the problem for them, not just explain the problem. So, let's connect

to the to the writing tools.

So, the agents will be able to initiate a dispute about the transaction, cancel transaction, and even transfer money.

So far, agent can do anything. It can do the conversation reads and writes.

And that's how we develop the tools.

And then we're going to explain like explore what's the problem with that.

But now let's first integrate it with your backend with Spring.

So, for that, instead of manually defining the connections to OpenAI or Anthropic, I'm going to use Spring Boot integration. You just open your Spring

integration. You just open your Spring Boot config and define AI Cook OpenAI, AI Cook Anthropic keys, and that's it.

Or uh you can also use uh Spring AI integrations just with a different keys.

And then uh prompt executor, this is the abstractions the the abstraction from Cook, it will be injected as a bean.

And then you can actually build agent.

Just create one and launch it.

And you can notice that there is a second parameter in the run function.

Apart from the question, we also provide the user session ID.

Because the agent might work in parallel with different users, and it needs to be able to keep track of all the sessions.

So, there is a parameter for that.

Now let's define the first API methods.

Uh for that, I'm going to define the API controller that has our uh Cook agent service and /support endpoint for our agent. It's going to get the principal

agent. It's going to get the principal for authorization with the user and the user's request.

Then it's going to launch the support agent and come back with a response once ready. No worries, it's a suspend

ready. No worries, it's a suspend function. It will recover sometime and

function. It will recover sometime and come back with the response.

Pretty easy, right? But, there's a problem.

How your agent can remember what was before? What if the user comes back and

before? What if the user comes back and asks to follow up? For example, like ask I ask send $100 to Mike, then send again the next day. What's again? To whom?

What to do? So, it has to remember the previous conversation.

Let's remember that.

The good news is the AI agent's instance is not single use.

It remembers the session, but what if your application restarts on another machine? For that, you need chat memory

machine? For that, you need chat memory that will dump all the agent memory into a database. Let's do that.

a database. Let's do that.

Pretty easy, just install a feature.

Please raise your hand if you know anything about Ktor.

Cool.

So, you probably know that in Ktor you can install features into HTTP server. It's kind of the same

HTTP server. It's kind of the same concept here. If you don't know, this is

concept here. If you don't know, this is just a way of doing dependency injection. It will intercept different

injection. It will intercept different places of the agent's execution and perform different actions. So, in this case, after the agent has finished, it will dump all the state into

the storage that you can provide by yourself.

And of course, you don't want to pollute your database or S3 storage with too many messages. So, for that, there are

many messages. So, for that, there are filters and size read windows. You can

also play with that.

Um well, chat history provider is a primitive interface, just store and load.

Uh but, usually you don't need to implement one uh because there is Postgres uh via exposed integration, uh MySQL, all of that is available. You just connect your

database and uh it will store it in a table. But, if

you're using Spring, it's even simpler.

You can use Spring Boot data source integration, and connect to your JDBC Postgres endpoint, and from the Spring AI, we just get chat memory repository,

and it will wrap it into Cook's chat memory history provider. That's it.

Um that's how it works.

That's the bean from Spring AI, and you just install it to your agent.

That's it.

Now, let's take a look how it works.

Um I'm asking, "Send $100 to Mike."

And it says, "Failed."

I continue the conversation the next day, come back to the same endpoint, and say, "Why?"

"Why?" And it says, "Ah, because you only have $99. You're not as rich."

$99. You're not as rich."

"Want to send less?"

Yes, I do. 20 bucks.

And it does it for me. That's chat

memory in action.

But that's not it.

Uh even if your agent is not single use, it's not yet scalable. And now we are hitting the first problem. You don't

have control over all the operations.

Let's analyze our agent.

So far we have connected our agents with many different tools. It can do the conversation, read from the database, write to the database, and so far LLM model will rule everything, all of the

operations in your banking system. Is

that really what you want?

Well, probably no.

So, there are obviously some problems. And the most essential one ones it is that, well, all these randomly working guessing machines, I love that one.

Uh if you give too much control, they will mess up your system.

The problem is actually in this default strategy that we know what never talked about before. That's how it works. You

about before. That's how it works. You

just start your agent, it asks the LLM what to do, LLM comes back with a tool call, you call it tool, and then you continue the process until a lamp decides to finish. No

control, no focus, no limits, no guardrails.

That almost would never work in real productions at scale.

Um and by the way, you can also implement the strategy that's actually cook uh code. You can do that. Uh it's

doing a while true loop almost.

Uh but well, it doesn't work.

Uh except demos.

So, the problem is well, first of all, uh all tools are available at once. And

if you give too much control and too many tools, a lamp will be just destructive distracted with the uh amount of informations and with the freedom.

Moreover, you cannot just specify any guardrails except you prompt them.

And if you prompt the guardrails, there is no guarantee that the models will follow the prompt. I mean, of course, you can hope, but that's it.

To solve this problem, uh you have to define a custom specialized strategy.

And we're going to use domain modeling approach for that, which is by the way uh a gift from the JVM and from Kotlin because you cannot not do that with

untyped languages.

So, the idea that we have to understand is the idea of subtasks. They are almost like subagents, like smaller ones focused on a subset of tools and a smaller task, not the whole one.

But the differ difference between that subtask and subagent is that subagent work independently. It doesn't know

work independently. It doesn't know anything around it. It doesn't remember what was

around it. It doesn't remember what was in the agent before it, but subtask is a part of a workflow. It is integrated and it shares the context and the conversation with everything that has

been before.

Importantly, it also has strictly typed input and strictly typed output.

And under the hood it can call many loops sorry many tools in a loop.

So essentially instead of describing the expected output as a text, you can describe what you want as the output data class.

Like that. That's what I want from this step. I need account number, username,

step. I need account number, username, related transaction ID, dispute ID if relevant and so on. Everything is

strictly typed and it's also described described for the model and of course it has to be serializable.

Importantly instead of just guessing from the prompt what you want, you are defining the form that the model has to fill.

If you think of that from the real world analogy perspective, imagine that you are coming to a bank and you're applying for a loan.

And well, you cannot impress banking manager with your oratory skills. You

cannot talk your way into getting a loan. You have to bring your passport,

loan. You have to bring your passport, your certificate, your you know, salary documents and all of that and then fill in the form. And if you don't bring your

passport, if you forget that or if you don't fill the form correctly, it will not be able to get a loan.

And even if you try to break through the system, probably the security guard in a bank will stop you.

So essentially cook serves this security guard kind of purpose. It makes sure that the agent is not allowed to proceed further to the next subtask until the

form is filled in.

So let's illustrate that. The agent

comes in at and it's required to fill in the account issue summary form.

And it starts calling the tools.

It gets the account info and fills in well, account number and the username.

Not enough. It has to proceed further.

Gets account balance and fills in the balance.

Uh Oh.

Well, I'm so rich. Cool.

Then uh not enough. It needs the information

not enough. It needs the information about the transactions. And it gets the related transaction ID and dispute ID is null. No dispute has been initiated so

null. No dispute has been initiated so far. And that's when the form is filled

far. And that's when the form is filled in.

And only now the agent is allowed to proceed further to the next step of the workflow.

So, even for the agent sometimes sometimes uh a little bit of bureaucracy is a good thing, especially when in Germany.

To sum it up uh well, naive prompting gives you only hope, but the main modeling approach gives you a contract that you can trust.

And now with all that knowledge in mind, let's finally develop a custom strategy for our banking assistant assistant. For

that, let's start with the first node.

It's called start, and it's going to get us the user's question, what the agent needs to do. It's going to be string.

Then the first subtask will be identify the problem with only access to conversation tools and account reading tools. No modifications is allowed.

tools. No modifications is allowed.

And I'm requesting my agent to fill in account issue summary form with all that details about the problem. And only after the

form is filled in, then it can proceed further to fixing the problem step.

That's going to be focused on another part of the subtask, and it's going to get only the access to database reads and database writes. No user

conversations are required here.

And I'm asking to provide me with the account issue solution with some other fields.

So, are we done yet? We just identified the problem and fixed the problem.

Well, probably as developers you never just commit untested code to master, right? Do you? Raise your hand if you do

right? Do you? Raise your hand if you do that.

Okay, I also do that.

But, usually we have to verify.

We have to check that the account issue solution is actually correct. And for

that, I'm going to add verify solution step that will take the account issue solution.

And by the way, please notice that I can only connect the steps that are type compatible. The previous one gets me

compatible. The previous one gets me account issue solution, and the next step takes the account issue solution.

Otherwise, the code will not compile and I will I will I will show it to you.

And this step will uh produce a critic result. That's a standard class from

result. That's a standard class from Cook. It has three fields.

Cook. It has three fields.

Whether verification was successful, optional feedback, what has to be improved, and the input, which in our case is account issue solution so that

we can pass through it further.

And this step will be focused just on user conversations and database exploration. No modifications are

exploration. No modifications are allowed here.

Then, I can actually write code based on that. If successful is true, I'm passing

that. If successful is true, I'm passing the original solution to the finish step. That's when I finish and stop my

step. That's when I finish and stop my agent. Otherwise, I'm passing the

agent. Otherwise, I'm passing the feedback to the next step to adjust.

And then coming back with the new updated account issue solution.

Please note that it's not possible to exit this loop only Oh, sorry. Before

successful is true.

This workflow is much more reliable than anything we have we have built so far before.

So, now let's learn how to develop that.

Well, let's code the strategy. For that,

let's understand how to build this basic component using the domain modeling approach, the subtask.

For that, I'm going to use the uh sub graph with task out of the box component from Cook.

That requires the input class, the output class. I'm just declaring what I want, just the class.

List of tools.

And optionally, I can even switch the model on a fly during the subtask. So,

each subtask might have separate model suitable for this specific step.

And then I'm explaining how to uh create the input, like what's sort of what the prompts for the model, what's the task.

So, in this case, you know, uh for example, identify the problem. That's

the description of the task, right? It's

a string.

And that's how you do it. Now, we can actually build a workflow.

Identify problem. It's a sub graph with task from string to account issue summary with a limited set of tools. And

I'm using GPT-5 model because in my experience, GPT are a little bit better when it comes to user conversations.

Next step, fixing problem. Same

component, different types, different tools, and also different models. I'm

switching to Sonnet on a fly. And this

step will be working with Sonnet because, well, again, in my experience, it's a little bit more precise when it comes to tool calling. And here, we need to do a lot of database modifications.

Next one, verify solution. Almost the

same, just a little bit different component. It's called sub graph with

component. It's called sub graph with verification. It's also out of the box

verification. It's also out of the box component, and it just makes sure that the critic result and the final LLM judgment is performed before LLM exits this step.

Um and also um well, I'm using GPT Oh, sorry. Yeah, GPT-03 model. Why? Uh

sorry. Yeah, GPT-03 model. Why? Uh

because we need a reasoning model here. We need

to reason about the output.

So, I'm switching the model on a fly again.

Last but not least, adjustment step.

Same component, different types, different tools, different model, different task. That's it.

different task. That's it.

What's left? Just to glue all of that together. For that, I'm going to define

together. For that, I'm going to define the edges for my graph.

Um first three edges are pretty primitive, uh from node forward to node.

The other ones though, they need some branching here. So, let's define that.

branching here. So, let's define that.

Verify solution forwards to node finish on condition it is successful.

But, that one will not compile because verify solution provides a critic result class.

Finish step expects the final solution and critic result is not a solution. So,

I will have to transform the type on a fly while going through this edge. For

that, I'm going to write transformed to the input.

Same way, uh and yeah, it's all compile time checked. You'll not be able to deliver

checked. You'll not be able to deliver such agent that is not type safe, right?

So, and then the same way the other edge, uh verify solution forwards to adjustment step on condition it is not successful transformed to a feedback.

Why? Because adjustment step requires a string.

And uh the edge back the same way.

And that's it.

And please note that uh I never said anything about message history or the context management because all of that happens out of the box under the hood.

Moreover, you don't even have to care about how you explain what was before with a new model with the previous model to the next one because because Cook does that all also out of the box.

I can illustrate that.

Remember, identify problem was working on the GPT model and it has the access to conversation tools and reading tools.

The next step, fixing problem, was working with a Sonnet.

Reading tools are still there, but the problem is that conversation tools are no longer available, but they are already in the message history.

So, if you just copy the message history and continue from there, lots of models would just break.

To solve this problem, we automatically re-explain the model like the tool calls to the model either with a plain text or JSON, or you can also customize adjust an interface. You can inherit how you

an interface. You can inherit how you describe uh an existing tool calls to the next model.

But, it just going to work with the default solution. So, you don't have to

default solution. So, you don't have to even know about that.

And that's when fixing problem continues with other tool calls and goes further.

That's how it works.

Um moreover, of course, sometimes you need some custom actions, not just LLM subgraphs or loops.

You I have to be able to put some actions and guardrails.

Uh that's pretty easy. Uh you can define custom node from any type to any other type. Here, I'm defining the ask user

type. Here, I'm defining the ask user confirmation node that just ask something from the user and comes back with a typed user response. You can send

SSE event. You can show some, you know,

SSE event. You can show some, you know, UI message and ask the user to fill in the form, and then come back to the LLM with the form filled, or you can do anything inside. You can put some

anything inside. You can put some guardrails and checks.

And that's very much recommended. And

that's exactly what Mercedes-Benz has done. You have already heard the quote

done. You have already heard the quote from the keynote, right?

Um but graphs are not the silver bullet, and we do understand that. And Kook

comes with different types of strategies. So, of course, you can start

strategies. So, of course, you can start always with the basic strategy, the simple loop. You can also develop

simple loop. You can also develop graph-based strategies, and that's what we recommend, but you can also develop functional strategies almost in plain code. Indeed, all these edges and

code. Indeed, all these edges and conditions and transformations can be expressed with just Kotlin.

For, if, while, that's it.

The only problem is that you're going to miss some observability and persistence.

I'm going to talk about it, but it's a good start. You can prototype and write

good start. You can prototype and write functional strategy fully in Kotlin, still with most of the Cook's features available and supported, including the automatic context management, but

before deploying to production, just refactor it once to a graph strategy, and that's it.

You can also use multi-agentic systems and build planner-based strategies.

Whoever heard of GOAP, goal-oriented action planner?

Okay, just a few people. I'll explain.

So, functional strategies, again, as I said, supports most of the things from Cook.

You can code with normal Kotlin, no persistence, and no observability.

Planner strategies, this is a pretty novel approach, originally pioneered with by by Rod Johnson, and the approach comes from gaming. So, essentially, you just define the nodes or like the

actions, and then you also describe the preconditions and beliefs, like what has to be true before the node is executed, and what's the belief or side effect of

the node. And then the planner will plan

the node. And then the planner will plan the optimal path, and replan if needed.

So, it's more dynamic, and it can kind of draw the edges for you.

And you can also do that in Cook. I'm

not going to dive too deep into that.

So, we have just developed a custom strategy that is so great, so good, and so resilient. Any problems? Well,

so resilient. Any problems? Well,

problem number two, runtime crashes.

What if your agent just crashes in the middle?

Well, LLMs take time, use a conversation, and tool calls take even more time.

What if user comes back with an email response the next day or the next year?

So, suddenly, your JVM might crash, or if you are using AWS EC2 spot instances for obvious cost optimization reasons,

well, your Amazon provider like might just preempt your instance and rerun your process from another machine.

And you will lose all your progress and all your money you paid for LLMs because you have to start from the beginning.

Moreover, you can start from the inconsistent state of the database just in the middle of the execution.

Not a good idea to to deploy such system into production.

And there is a solution. It's called

Cook persistence.

You just install it to your agent, any agent, but it has to be a graph one.

Why? Because it's not possible to persist any place of the execution just in the middle of your Kotlin code. You

cannot serialize Kotlin function and just rerun from there. But you can do that with graphs because there is a structure.

You only need to provide the storage where to store the end point, S3, database, file system, memory, somewhere.

And enable automatic persistence so that it will checkpoint after each step.

So, how it will work? Well, it will just checkpoint the exact node of the workflow plus the state including the message history, storage, meta information, everything.

Let's just illustrate that. Let's zoom

in and zoom in again. Let's imagine that our agent is somewhere inside this verify solution step, for example, calling LLM or doing any other inner

atomic action. It's also a node. So,

atomic action. It's also a node. So,

what will happen is Cook will checkpoint the place of the algorithm, the node, and also all of the meta information including the system messages. And by

the way, there is a to-do. Why? Because

you never finish your presentation in time. I'm joking. Um what will happen is

time. I'm joking. Um what will happen is the same agent will recover after a crash exactly from where it stopped.

Um you only need to connect with the storage provider. A persistence uh

storage provider. A persistence uh storage provider with JDBC is supported, Exposed is supported, many databases are also supported, and you just connect to

the storage provider, and it will just work.

Moreover, you can also do it with Spring. You can use Spring data source

Spring. You can use Spring data source or Spring AI or Spring Boot, whatever you want, and the bean will be just injected, and you can use it here.

So next you can adjust uh proceed without debugging and monitoring.

No system is perfect. You have to observe what's happening.

For that, we're going to integrate with observability tools. Here, I'm

observability tools. Here, I'm installing OpenTelemetry. It's also a

installing OpenTelemetry. It's also a feature in Kog. You just install it to any agent, and you can connect it to any span exporters on and span processors if you already have some

OpenTelemetry-compatible tooling in your system. Or if not, you can also use LangFuse integration.

LangFuse is pretty popular AI observability specialized provider.

There is out-of-the-box integration.

Just provide the credentials to your LangFuse console, open it, and you will see all the graph nodes, how much money you spent, messages, tool calls, tool responses, all of that.

Same way, you can integrate with Weights & Biases, with another pretty popular tool.

Uh just add the the the Weights & Biases credentials, and you can see the same exact information in Weights & Biases console.

Or the DataDog, same thing.

Um but, there is one more problem that I promised to solve, which we all, I hope, love, which is money.

Agents has to be has to be has to be cost-efficient.

So, the first problem is pretty obvious.

Well, long-running agents take a lot of money from you.

Uh for instance, we've been trying and running and evaluating different coding agents in JetBrains.

And the thing is one single run through the whole SV bench benchmark costs around several hundreds of dollars, like 5 to 700 of dollars, just

one single iteration over the whole benchmark.

And that can come with a lot of cost because if you deploy such agent for millions of users, not in the IDE, but on the backend, that's going to be very expensive.

Moreover, a level have to read the whole history of the conversation from the very beginning in order to come back with every single next message, whether it's

a assistant message or a tool call.

It will have to read the whole history of the tool calls and conversations in order to guess the next action.

Um obviously, it's very expensive.

Moreover, the more messages are in this book of history uh that the that the model has to read, the worse it works.

Precision drops drastically. Moreover,

latency increases. You have to wait longer and longer and longer times before you get each next message.

But that's not the only problem.

Moreover, uh you can easily hit the context limit.

That's especially true if you're running on local models, like on device, or if you're just deploying smaller LLMs. But even with the frontier models,

I once made a uh personal agent just for my myself.

I used a frontier model from popular provider, which has 2 billion token context limit, which is the largest available today.

And I asked it to perform a very simple task. You know, just look through my

task. You know, just look through my documents. Here is my passport, 40 PDF

documents. Here is my passport, 40 PDF pages. Here is my job contract,

pages. Here is my job contract, 72 pages.

And some other documents.

Well, pretty simple task. And I asked like, "Please check, do I have all the required documents for the UK visa?" I

was applying for the visa that time.

And after several of iterations reading through my documents, it just crashed with an exception. Sorry, I hit the context limit.

Imagine scaling some the same agent for millions of users on your backend if it cannot solve small problem for just me, a single person.

That's not going to scale. And you have to solve this problem and there is a solution, it's called history compression.

So, it's also a node in the graph workflow. It's called node LLM compress

workflow. It's called node LLM compress history.

And it has just a one type because it doesn't do anything with the value. It

just passes it further.

But what it does, it actually composes a TLDR message. Instead of the whole

TLDR message. Instead of the whole history, LLM will just see a TLDR. What has been done before this point in time.

And you can, of course, put it anywhere in your strategy. For example, after between adjust and step and verify solution step, if the history is too long, I'm putting the compress history

node.

And that's how I how I do it. So,

adjustment solution goes to compress history on condition history is too long.

And it goes to just verify solution if it's not too long.

That's it.

Um Yeah, and history is too long is just an extension method that you can define if the token usage is uh huge. That's it.

uh huge. That's it.

The problem is this.

Now, LLM doesn't have enough information to come back with the right decision.

And we also tested that and hit into that problem.

For example, if you are doing coding agent or banking agent and the agent has already explored uh the repository or the user account, and then you can press the history, of

course, it will get the TLDR, but it will lose important information about, for example, project project dependencies or some transactions throughout your account. And what it

will do next in the next step, it will start to explore your system again from the beginning.

And you wanted to save money, but you will actually lose money because it will cost you more tokens.

Of course, we know about the this problem, and we also solved it as well.

Because we came up with different strategies for history compression that you now also can use.

The first one that we tried was chunking.

Um inside this uh history compression notes, there is an optional parameter called strategy.

And there's a bunch of out-of-the-box strategies uh that you can use, and it's also an interface you can implement your own if you want.

So, chunking is pretty simple. Here, I'm

just asking the model to chop the history into pieces of 20, compress each of them independently, and then instead of reading the whole book, LLM will read

the plot of the conversation.

Works much better, but still not good enough.

And the final thing that we came up with was uh fact retrieval. And by the way, these strategies are not available in any other frameworks, including Python

frameworks. So, it's only in Cook.

frameworks. So, it's only in Cook.

So, fact fact retrieval works like that.

As a developer, you can specify the concepts, what specific things you want to persist in the first in in the history.

Like here in our banking assistant, I'm asking to save the steps that were important, and also the banking operations that were suspicious. And you

can also come up with your own list.

So, instead of just blindly persisting the plot, it will search for the specific facts, drop the history, and then compose a new

one that naturally mentions these facts.

And this this strategy works pretty well, and it gained us gained us gained us around like 5 to 8% improvement on the benchmark.

So, that's the illustration is instead of reading the whole book, LLM will be left with just a few important pieces of paper on the table.

Uh you can also scan the QR code to view the full example that we just cooked.

Yeah.

Can I just leave this slide for a while?

And uh just to recap, so we started with a simple agent today. We programmed

tools in Kotlin. We connected to Spring application. We added sessions with chat

application. We added sessions with chat memory. We also tailored specific

memory. We also tailored specific strategy for our tasks task using graph strategy to get the most reliability and control. We enabled fault recovery with

control. We enabled fault recovery with with persistence feature. We connected

to different observability tools, and we cost optimized our agents using history compression.

Here's the website for Cook. If you have never visited it, please feel free.

And of course, try Cook 1.0 today and cook your production AI tomorrow. Thank

you so much for your attention.