Harnesses in AI: A Deep Dive — Tejas Kumar, IBM

[music] Hello everybody.

Everybody's head turned up. Hello, hi.

How was lunch? Was it good?

You didn't like it, no?

It's like British food. Anyway, hi. I'm Tejas. I'm

British food. Anyway, hi. I'm Tejas. I'm

I'll be your first speaker this afternoon. Tejas, that's pronounced like

afternoon. Tejas, that's pronounced like Tejas.

Don't worry, I'm not Hopefully my my joy in AI is and I've had the privilege of working at a number of different places over my career in one form or the other. It's just been an absolute joy to learn from the best.

Today, I'm a AI developer advocate at IBM where we we do things with AI, believe or not. We train frontier models, we

or not. We train frontier models, we build harnesses. It's really it's a fun

build harnesses. It's really it's a fun lab to work in.

But that's not what I'm here to talk to you about today. Today, I'm here to talk to you about AI harnesses. AI harnesses.

Before I move forward, I would love to just have a show of hands.

How many of you are like confident in your understanding of AI harnesses? Like

you're like I could present this on stage today.

Look around. Look around. No, seriously,

look around. That's why we're doing this talk. Okay, this is my hope. I want you

talk. Okay, this is my hope. I want you to If I ask you this at the end of the talk, right? I want you to be like, oh,

talk, right? I want you to be like, oh, I I I get it now. That's the whole point. I have literally nothing to gain

point. I have literally nothing to gain from this other than I I shared knowledge, okay? Because also this term

knowledge, okay? Because also this term is kind of everywhere. You may have heard it used like 52,000 times today.

And it means different things to different people cuz like in the machine learning world, it means like a glorified test suite for machine learning models. But in the AI in the AI

learning models. But in the AI in the AI world, it means something different. And

so today, we're going to understand this in detail. It's a deep dive, but it's 18

in detail. It's a deep dive, but it's 18 minutes long. So let's let's move

minutes long. So let's let's move forward. Um

forward. Um I want to start by talking about why harness. Like why do we use harnesses?

harness. Like why do we use harnesses?

And the reason for this is because we pay rent to companies that give us compute, give us inference, give us tokens in return. Some of you maybe work for companies that have frontier models

like Anthropic or Google or whatever and you maybe What was the term? Token

billionaires yeah?

Um I'm not that. I am maybe with Watson models, but but the vast majority of us aren't token billionaires. We we pay rent. We literally $20 a month for

rent. We literally $20 a month for Claude Pro. And then you get a context

Claude Pro. And then you get a context window that's limited and you get like, you know, you you don't get the full hog, so to speak. And the model you rent is is a black box. Like they could at

any time, I'm not saying they do, but they could if Opus is somehow not available, they could serve you Sonnet even though it says Opus. You would

never know, right?

And so it's just a big There's too many variables that we cannot control. So why

harness? Because the name of the game with harness is reliability. Um

I really hope I'm not supposed to stand in front of this white line and then I'm just not in the camera. Anyway,

whatever.

It's reliability. It's it's making sure that the agents we build do what they do, period. Irrespective of the black

do, period. Irrespective of the black box model, irrespective of the of the the thing we rent and so on, okay? Now

that we understand why harness, let's talk about what a harness even is from first principles. Like let's let's take

first principles. Like let's let's take it all the way back to harnesses that we know and understand. If you've ever, you know, climbed a mountain or something or you've seen someone This is a harness.

It's like mountain climbers literally will like harness themselves to what? To

a mountain because it's stable. And they

can't go off the rails, literally.

They they anchor themselves in something stable so that they can't drift too far.

Okay, that's that's what a harness is by design. When you have any dog owners

design. When you have any dog owners here? You have dogs? You you walk your

here? You have dogs? You you walk your dog on a harness, okay? That's why?

Because your dog doesn't go and bankrupt you with tokens. Okay?

That's what a harness is.

But the problem is if we think about what harness, there's really two types.

There's one from the machine learning world, which as I mentioned is kind of like a test suite and a test runner.

You would give a model some inputs and you see the quality of the outputs.

That's not This is not ML engineer Europe. We're going to talk today about

Europe. We're going to talk today about the agent harness that is common in AI engineering. Okay, so what what is an

engineering. Okay, so what what is an agent harness?

An agent harness and and this is kind of the money shot here. The agent harness is I'm not making money off this. It's

just an expression. The agent harness is everything around the model that gives it grounding in reality. It's literally

the thing that ties it to a stable environment, okay? An agent So Claude

environment, okay? An agent So Claude code, for example, can be considered an agent harness. And some of you would

agent harness. And some of you would say, oh, no, it's a coding agent.

Absolutely, it's a coding agent. But

it's a harnessed coding agent. An agent

harness has more or less the same typical suspects, moving parts. Number

one, it's got a tool registry. Almost like so Claude code, cursor, codex, they have tools to read from the file system, to write, to execute bash commands, right? They have

a tool registry. They have a model and some of them allow you to choose a model, some of them allow you to not.

They have a model. They have primitives for managing context. Almost every

harnessed agent runtime today will compact its own context, right? That's

that's that's the job of the harness.

Guardrails are another part of a harness. For example, max steps. Anyone

harness. For example, max steps. Anyone

using max steps? Do not do more than five tool calls. That's a guardrail. And

so if if you do that, you just kill the kill the run, right?

An agent loop is another part of an agent harness, which is crazy. This is

what some people I've spoken to preparing this talk will say, wait, isn't a harness just the agent loop? No, it's the stuff around the

loop? No, it's the stuff around the agent loop. In fact, it could be a loop

agent loop. In fact, it could be a loop around your agent loop. It could be an NM loop. And we'll look at that a little

NM loop. And we'll look at that a little bit in some code. And then finally, there's a verify step. This is, for example, in a coding coding agent, after the work is done, a verify step would

be, hey, let's let's run lint, let's run tests, let's make sure nothing broke, right? So almost every I'll use code

right? So almost every I'll use code coding agents as an example, but you could have a harness for anything. And

it's it's amazing cuz it really grounds black box models in a stable environment that you control, okay? I'd like to show you a demo. And what we're going to do

a demo. And what we're going to do together is we're going to build a harness, a bare bone baby's first harness. Let's call it a poor man's AI

harness. Let's call it a poor man's AI harness together so we understand from first principles how this works. We're

going to build a computer use agent that has a job. The job is go to Hacker News and upvote the first post, okay? It's a

computer It's a browser use agent. We're

going to use a really bad model intentionally. We're using GPT-3.5

intentionally. We're using GPT-3.5 Turbo, which is like 2023, right? But

we're going to harness it so that it can actually do the job. And we're going to save money. So let's I've spoken too

save money. So let's I've spoken too much. Let's just get into the demo.

much. Let's just get into the demo.

Uh And and so let's Welcome to my project.

This is my project. Hello everybody.

This is the entry point. Can you see that? Is it too

that? Is it too Yeah? You want it bigger?

Yeah? You want it bigger?

Let's do bigger. Okay. So this is not Actually, this room is too bright. Let's

do light mode. It's I It's not my nature, but sometimes. That's better,

yeah? Okay. So we have we have a model and we're trying an old LG Sorry.

We We shouldn't have seen that. No, we'll

we'll try an old model. And this is the prompt. This is the This is the task.

prompt. This is the This is the task.

This is literally my prompt. Upvote a

story I just described it. For the

purpose of this demo, we will not change the prompt at all.

Because a lot of us think, hey, my agent is not doing what it's supposed to do, so I just need to prompt it harder, right? That's not always true. I need to

right? That's not always true. I need to change the system prompt. We're not

going to touch any prompts here. We're

just going to build a harness and the outcome will change.

We we log some things to the console and then we start a browser session. Okay,

what's a browser session? It's literally

just Playwright. Not Playwright MCP, like Playwright Playwright. Where this

is just a class I made with an open method that launches Chromium and gets a context and makes a page. And then

navigate We're just literally calling the Playwright functions, yeah? This is

This is just traditional engineering. So

we create a session, we open the session, meaning a browser window in a context. And then we create our tools

context. And then we create our tools and we give that browser session to the tools. And we create a context and we

tools. And we create a context and we give the task, meaning the prompt here, to the context. Now, create tools is literally what it sounds like. It's

here. There's just some types and create tools is a function that takes a browser session and gives you like tools. And

these tools are not I didn't invent this. This is from OpenAI's SDK, okay?

this. This is from OpenAI's SDK, okay?

So you have the name, the description parameters, and execute, the way you actually call the tool in your runtime.

And and there's just tools for I made this. It's very easy. Um

this. It's very easy. Um

So that's my tools. And then create context, you may think, whoa, context engineering. Absolutely not. It's This

engineering. Absolutely not. It's This

is my context. There's nothing here.

It's just a system prompt.

Literally, the most basic system prompt and the user's task. This is basic basic. And then we have run loop, which

basic. And then we have run loop, which is just running the agent in a loop. So

what it's doing here, we can actually just look at this, too.

While true, so it is an agent loop. And

we get a response from the agent and we see if the response says stop, meaning if the LLM says, I'm done, then we return the value.

If we get any other response, we don't do anything except add these events into a trace. So we

just push history into a big list of history. Does that make sense? And so

history. Does that make sense? And so

that's all we're doing here. This is

just a loop where we just collect events until we're done, okay? So this is super basic. Now let's see how it works. So

basic. Now let's see how it works. So

I'm going to come over here and I'm going to do Are you okay, sir? Do you

need water? I'm going to npm run agent. Um And so it's going to open Chromium. It's going to Okay,

open Chromium. It's going to Okay, Hacker News, so far so good. Click

upvote. Oh, no. So we we hit a login screen and then it kind of panicked and crashed. But look, it it lies. You see

crashed. But look, it it lies. You see

this?

Um this is a problem. And so what's the solution? Prompt it harder? No. Change

solution? Prompt it harder? No. Change

the system prompt. Always login with these credentials included in the system prompt. No.

prompt. No.

So how do we then solve this? And look,

we because of my logging, we can actually see it just clicks the upvote button and then considers it a success.

It doesn't verify. This is the job of a harness, okay? So now incrementally,

harness, okay? So now incrementally, we're going to slowly start building a harness. Um

harness. Um And so, let's just move I'm not going to write code here. I'm not going to live code because we don't write code anymore. We inspect diffs.

anymore. We inspect diffs.

Right? Anyone write code by hand? You

don't Maybe actually you do belong here.

Anyway, so um I'm kidding. So, this is um

I'm kidding. So, this is um This is the first change we're going to make. This was our index file.

make. This was our index file.

And we have this run loop that I showed you, but now we're going to add one thing to it, which is default guardrails. We're going to create some

guardrails. We're going to create some guardrails okay?

Um what do our guardrails look like?

Well, let's go and look at it in the editor um with guardrails over here. And

so, we have some types, but these are our guardrails. We have two. Max

our guardrails. We have two. Max

iterations, meaning if you do more than six steps, I'mma kill you.

And max messages, meaning if you have more than this many messages, I will compress the context. These are just guardrails, okay? A little utility to

guardrails, okay? A little utility to combine them, and we just we can compose them here. We could do like as many as

them here. We could do like as many as we want. So, now

we want. So, now let's go back to our changes. That's the

guardrails. We if we go back to the agent loop we actually use the guardrails here in this diff. And so, we include the guardrail functions, and we can see that here what we're doing is we're checking how many messages have we

accumulated, and we just like trim the context if it's too much. Um but what I did want to show you is here at the end um we we push context size, which is some more metadata about what we've done

with our guardrails, okay?

Um our context compressor is extremely basic and extremely naive. This is what it does. Um let me actually open this

it does. Um let me actually open this with syntax highlighting to spare Um this is what it does. So, what we're doing is if we always keep the system prompt and the user prompt and the most

recent two messages. So, if the guardrail is triggered, we always remove everything after the system prompt and the user prompt in the middle, and we keep the last two messages. This is

super naive. Don't do There's better ways, but this is where babies first.

We're We're getting there.

So, we we're starting to have a harness, but it's not called a harness, but this is really like a pregnant harness. Like it's almost born, okay? And so, what we're going to

born, okay? And so, what we're going to do is let's just call it a harness now.

So, I'm going to show you another diff where we Here, check this out. Index, we've

deleted almost everything.

Um and we've moved it into this file called harness. Let's go look at our

called harness. Let's go look at our entry point now.

It index, it's it's all gone. So, the

prompt is there. But this is it's like 19 lines of code, and we just have run harness. We've taken all the logic from

harness. We've taken all the logic from here and hidden it in a function called run harness. And as you would expect,

run harness. And as you would expect, run harness does exactly the same thing as we did in the index, okay? Nothing

new is here except maybe like a print function, which is just console log. Is

this clear so far? Yeah, we just moved stuff. Now that we have something called

stuff. Now that we have something called a harness, we can actually use it.

And let's solve the problem of lying first before we solve the problem of logging in as me. Yeah, because it says I I upvoted, it did not. I want to know.

So, what we're going to do is we're going to add some guardrails and and have it tell the truth. Like if you failed, tell me the truth. Um

how might we do that? Well, we'll check it out here. So,

many many things changed. Um

Or not. I don't know. Hang on a second.

Yeah okay.

Did Many many things changed. So, we run harness and we added a third argument here, which is a verify step and max attempts. Max attempts goes to our

attempts. Max attempts goes to our guardrail. So, if if you took more than

guardrail. So, if if you took more than three tries to do this, just give up.

And if we go to the harness, we added a lot of things um that are just manual code. This is not different prompt. This

code. This is not different prompt. This

is my logic. Um the main logic is run harness no longer wraps over the code we moved, but we moved that to a different function called

run harness attempt. So, if we if we come to run harness Let's go here. I

need to check the branch out, sorry.

Yeah. So, now if we go to run harness attempt, we'll collapse this. I'll

collapse this.

I'll collapse all of these. And if we go to run harness attempt, now this is the same thing from our index. We just moved it into a function called run harness attempt because our main run harness is just a loop that runs no more than three

times, okay? Is this clear? So, we're

times, okay? Is this clear? So, we're

just enforcing the max steps, but at the harness level for safety. Um

then we have run harness attempt that calls it. We have this function called

calls it. We have this function called verify successful upvote. I wrote this.

This is deterministic. That's what I want to show you. What does this do?

Well, we see if You remember we were tracing in the agent loop, we're just adding history events. So, we reflect on that, and we see if there was a browser click

on the upvote and if it's successful, but really successful, then we say true.

But there's a huge butt here, which is we have now cases for failed login. If

there's a tool named harness auto login, and if the message starts with failed then we return early and we say no no, this failed. We're We're removing the

this failed. We're We're removing the lie, okay? Similarly, unrecovered login

lie, okay? Similarly, unrecovered login redirect. We look over our agent loops

redirect. We look over our agent loops tools that we've been pushing into.

Um and if we see that the harness auto login didn't run and now we're on the page that is the login URL, then again, we just fail.

Okay?

Um and so, we're what we're doing is we're just adding like if this happened, if this happened, you just just fail.

Return early. Is this clear? This is

what a harness does. And so, let's run this now with the harness.

Uh npm run agent.

And now it's going to go on Hacker News, and we're going to repeat the same cycle.

Okay, it's going to come here, and now it's still failed, but look, it stopped lying because our harness checks the tool history and actually sees what happened. This is what a harness is

happened. This is what a harness is supposed to do. Great. This is already like half the battle won because step one to solving a problem is admitting you have one, okay?

Test-driven development vibes. So, now

that we we're failing correctly, we can succeed. And I'd like to show you that

succeed. And I'd like to show you that in the last diff, and then we'll finish the talk here. So, number four.

Um we have a whole new function. It's

called login handler. Uh I'll add some syntax highlighting here so you don't go blind. Uh but here, create login

blind. Uh but here, create login handler. This is This is all it does. It

handler. This is This is all it does. It

runs every agent loop just before we push to the traces, and it This is what it do It checks the browser session's current URL.

And if we're not on a login page, it just says cool, I don't I return I have nothing for you. This computationally is not costly at all, right? If you're not on the login page, but if you are on the login page then it we fill in

a temporary This can be an environment variable. It can be secure, you get the

variable. It can be secure, you get the idea. But we fill in credentials and

idea. But we fill in credentials and submit the button programmatically from the harness, not from the agent, deterministically and securely because this file has access to any secrets I

want it to, right? And so, this How How is this called? Well, this is called in the agent loop. So, if we go back to our agent loop and notice we were pushing traces, yeah? This is where we push the

traces, yeah? This is where we push the traces.

Just before, if we have a login handler we call the login handler just before this in the agent loop. What does the login handler do? Well, if we're not on the login page, it does nothing.

If we are on a login page, then it quickly will inject credentials and submit the form and then take you back.

It will also add, as we can see here, it pushes a message into the queue saying, "Hey, I'm the harness. I logged in.

You're good now." Is this clear? Yeah?

So, the the harness is is is literally harnessing the agent to something stable, something deterministic. That's

what it's for, okay?

Let's run this now and see what happens.

So, npm run agent.

It's going to open Hacker News, and when it gets to the login, now that harness step, it logged in and it upvoted the first one, and it closed.

Amazing. So, successfully upvoted a little snitch for nilux, uh rank two, uh succeeded after six iterations, and I can click this and go into Hacker News and actually see indeed it was upvoted,

um and it I can unvote now, which means it was upvoted, right? So, um the agent used the computer, logged in as me with my harness that I just made here on stage. That's the purpose. Is this clear

stage. That's the purpose. Is this clear so far? Do you understand the role of a

so far? Do you understand the role of a harness? Look at you nodding. This is

harness? Look at you nodding. This is

music to my ears. Fantastic.

Something to my eyes. I don't know the It's beauty to my eyes, kind of weird.

We don't have a expression for that.

Let's land the plane. I'm done. I think

my work here is done. What does this look like in practice? Why Why do I care so much about harnesses? Because they

run the world. Models are

non-deterministic. And you want to do more with less. You want to use a cheap model. Use like Quinn or something, or

model. Use like Quinn or something, or even something smaller. Use GPT-OSS.

It's free. And with a great harness, you can go very far. That's why. At IBM, we create a open-source project that we deploy in the enterprise that allows very large companies, huge companies, in

their private like data-sensitive areas to perform rag operations on all kinds of things, teams, calls, and PDFs, and invoices. Um We We build It's called

invoices. Um We We build It's called open rag, and it's it's rag I don't know if rag is cool or not anymore, but open rag has a hell of a harness that provides enterprise-level security to

like asking questions with internal very very siloed data. And And that's kind of where the harness engineering comes in.

So, let's summarize. We covered a lot of content. Was it a deep I think it was a

content. Was it a deep I think it was a deep dive. It was a deep dive in like 18

deep dive. It was a deep dive in like 18 minutes or so. Um

we went pretty far. I

It's not It should not be lost on you that I did not touch the prompt once.

I did not change the system prompt. We

just built a harness and the outcome radically changed. And

of course, we can add secrets, we can add tokens. Um yeah, we did a lot. In

add tokens. Um yeah, we did a lot. In

the end, I hope you understand what a harness is, the value it can present, and how you can use it. What's next? Um

Look I I don't have a crystal ball like everyone else here, um but it's not lost on me that 2025 was the year of agents.

Yes? Uh 2026

is the year of harnesses, I'm pretty sure. Everybody How many times is this

sure. Everybody How many times is this word used here? Um I think I would hope I think it'd be pretty cool if 2027 was the year of dynamic

on-the-fly generated harnesses. How cool

would that Like you tell an agent, "Hey, do this for me. Buy me a flight ticket."

Whatever it may be. And then before doing the work, the agent creates a harness. This similar to plan mode. Any

harness. This similar to plan mode. Any

of you using plan mode? But But on steroids. The The agent creates an

steroids. The The agent creates an actual harness, self-aware. It knows,

"Oh, I can maybe hallucinate here. I can

maybe" Creates a harness, does the job, and returns back to you, guardrailed and everything. That is so cool. Dynamic

everything. That is so cool. Dynamic

on-the-fly harnesses. I would I I this is honestly the next logical step towards AGI, and I would love to see it.

I don't know if this is just me being uh you know weird guy with ideas, but um I think that's kind of the direction. So, with

that um I'm almost out of time. I would

be really remiss if I didn't spend the last like 30 seconds saying thank you so much. The slides are on GitHub uh as uh

much. The slides are on GitHub uh as uh am I uh and so I'd love to chat more.

Thank you.

[music]