Context Engineering for Agents

Hey, this is Lance from Langchain. You

might have heard this term context

engineering recently. It's a nice way to

capture many of the different things

that we do when we're building agents.

Of course, agents need context. They

need instructions. They need external

knowledge. They also will use feedback

from tool calls.

Context engineering is just the art and

science of filling the context window

with just the right information at each

step of the agent's trajectory.

Now I want to talk about a few different

strategies for context engineering which

we can group into writing context,

selecting context, compressing context,

and isolating context. I'll walk through

some interesting examples of each one

with popular agents that we use

frequently in our day-to-day work. And

I'll also talk about how Langraph is

designed to support all of these. But

first, what is context engineering?

Where did this come from? Well, Toby

from Shopify had an interesting post

here saying he likes the term contact

engineering. Karpathy followed up on

this and offered a good definition.

Contact engineering is the delicate art

and science of filling the context

window with just the right information

for the next step. And Carpathies

recently highlighted an interesting

analogy between LLMs and operating

systems like the LLM is a CPU context

windows like RAM or its working memory.

And importantly, it has limited capacity

to handle context.

And so just like an operating system

curates what fits in RAM, context

engineering you can think of as the

discipline or the art and science of

deciding what needs to fit in context.

Oh, for example, each step of an agent

trajectory. Now, what are the types of

context we're talking about? Well, we

can think about this as an umbrella over

a few different themes. So, one is

instructions. You've heard a lot about

prompt engineering, and that's just a

subset of this. There's of course things

like memories. There's few shot

examples. There's tool descriptions.

There's also knowledge that could be

facts. That could be memories. And

there's tools which could be feedback

from the environment for example using

APIs or calculator or other tools. And

so you have all these sources of context

that are flowing in to LLM when you're

building applications. Now why is this a

bit trickier for agents in particular?

Well, agents have at least two

properties. They often handle longer

running tasks or higher complexity tasks

and they also utilize tool calling. Now

both of these things result in larger

context utilization. For example, the

feedback from tool calls can accumulate

in the context window or just very long

running tasks can accumulate lots of

token usage over many turns. Here's kind

of an example showing turn one you call

a tool. Turn two you call another tool.

If you have a large number of turns,

that tool feedback just grows and grows

and grows. Now, what's the problem with

that? This blog post from Drew Brun

nicely outlines a number of specific

context failures, context poisoning,

distraction, curation, and clash. I

encourage you to read that post. It's

really interesting, but it's kind of

intuitive. As the context grows longer,

there's more information for an LM to

process and there's more opportunities

for an LM to get, for example, confused

due to conflicting information or

injection of a hallucination which

influences the response in an adverse

way. And so for these reasons,

contexting is particularly critical when

building agents because they typically

have to handle longer context for the

reasons mentioned above. Now, cognition

highlighted this pretty nicely in a

recent blog post saying context

engineering is effectively the number

one job of engineers building AI agents.

So, what can we do about this? Well,

I've had a look at many different

popular agents that many of us use

today. Thought about this a lot,

reflected on my own experience. You can

kind of distill down approaches into

four bins. writing context. Saving

outside the context window to help an

agent perform a task. Selecting context,

selectively pulling context into the

context window to help an agent perform

a task. Compressing, retaining only the

most relevant tokens. And isolating,

splitting context up again to help an

agent perform a task.

And now I'll talk about some examples of

each of these categories.

So writing context. Writing context

means saving outside the context window

help an agent perform a task. When

humans solve tasks, we take notes and we

remember things for future related

tasks. Well, agents can do those same

two things.

For note-taking, agents can use a

scratch pad. And for remembering things,

agents can use memory.

So, you can think about scratch pads as

kind of a term that captures the idea of

persisting information while an agent is

performing a task. And I'll give a good

example of this. Anthropic's recent

multi-agent researcher. The lead

researcher begins by thinking through

the approach and saving that plan to

memory to persist it. And this is a

great point. You want to keep the plan

around. The context window might exceed

the limit of 200,000 tokens, but the

plan can always be retrieved and

retained. Very intuitive

example of taking a note and saving it

in a scratchpad. Now, I do want to make

a subtle point. The implementation for

your scratch pad can differ. So, in

their case, they just save it to a file.

But you can also for example save it to

a runtime state object depending on what

agent library you're using. But the

intuition is really that you want to be

able to write information while the

agent is solving a task so the agent can

then recall that information later if it

needs. Now memories are a bit different.

Sometimes we want to save information

across many different sessions with an

agent. So typically scratch pads are

relevant only within a single agent

session. An agent's trying to solve a

problem. It'll use a scratch pad to

solve the problem and then scratch pad's

not relevant anymore. Memories are

things that you want the agent to retain

over time over many sessions. So there's

some fun examples from the literature.

Gener agents for example synthesize

memories from collections of past agent

feedback. And you've seen this chatbt

has a great memory feature. Cursor

windsurf also will autogenerate memories

based on user agent interactions. So

this pattern is certainly emerging with

popular AI products. And again the

intuition is pretty clear here. You have

some new context. You have some existing

memories. And you can update memories

with new information

dynamically as the agent is interacting

with the user. Now, we covered writing

context and talked about a few examples

of that. Now, let's talk about selecting

context. So, selection means pulling

context into the context window to help

an agent perform a task. Now, we kind of

talked about this previously with

scratch pads. Of course, an agent can

reference what it wrote previously. This

could be via a tool call. This could be

by reading directly from a state object.

Now, memories are a bit more interesting

and a bit more subtle.

There's different types of memories you

might want to pull into context

depending on the problem you're trying

to solve. Could be fot examples that

provide specific instructions for a

desired behavior. It could be a

selectively chosen prompt for a given

task. could be facts. And these are

different memory types. Semantic

memories in humans, for example, are

things like facts, things that I learned

in school. Episodic memories are more

analogous to fot examples, past

experiences. Procedural memories are

like instructions, instincts, motor

skills. These are all things that we may

want to selectively pull into context to

help an agent solve problems. Now, where

do these come up practically different

agents we work with today? Well, you

think about instructions or procedural

memories are typically captured in

things like rules files or like cloud MD

when you're working with the code

agents. This is typically a file that

has like style guidelines or general

instructions for tools to use with a

given project. And often times these are

all pulled into context. For example,

when you start cloud code, it'll just

pull in all the cla files that you have

in your project and in your

organization.

Now, facts are a bit more subtle.

Oftentimes we want to selectively pull

in facts from a large collection. And

this is where it's common to think about

things like embedding based similarity

search or graph databases to actually

house collections of memories

in order to better control their

retrieval and ensure that only relevant

memories from a large collection are

pulled in at the right time.

Now tools are another very interesting

thing we often want to pull into

context. Now, one of the problems is

that agents have difficulty with

handling large collections of tools.

This paper I link here has some

interesting results on that showing

degradation after around 30 tools and

complete failure at around 100 tools.

And so they propose using rag over tool

descriptions. So that's just basically

embedding tool descriptions and using

retrieval based on semantic similarity

to fetch out relevant tools for a given

task. And this can improve performance

significantly. So it's a nice trick for

pulling selectively only relevant tools

into context.

Now finally I want to talk about

knowledge. Rag is a huge topic and you

can kind of think about memories as a

subset of rag but rag is of course much

broader. We often want to augment the

knowledge base of an LLM with for

example private tokens. The code agents

are some of the largest scale rag apps

currently and I actually thought this

post from Verun the CEO of Windsurf was

quite interesting. He talks a bit about

the approaches that they use for

retrieval. And the real point is it's

quite non-trivial. So of course you're

using indexing

and embedding based similarity search as

a core rack technique.

But then you have to think about

chunking. So they for example do a

parsing to chunk along semantically

meaningful code boundaries, not just

random blocks of code. So that's kind of

point one. But he also mentions that

pure embedding based search can become

unreliable. So they use a combination of

techniques like GP file search knowledge

graphs

and they use an LLM based ranking on top

of all of these. So when you look at

knowledge selection in popular agents

for example like the code agents this is

highly non-trivial and the huge amount

of context engineering goes into

knowledge selection.

So we talked about writing context we

talked about selecting context. Now

let's talk about compressing context.

This really involves retaining only the

tokens required to perform a task. So a

common idea here is summarization. If

you used cloud code for example, you

might have noticed

that it'll call autoco compact once the

session reaches 95% of the context

window, 200,000 tokens for the cloud

series. And so that's an example of

applying summarization across a full

kind of agent user trajectory.

But what's kind of interesting is you

can also apply summarization a bit more

narrowly. Like anthropic's recent multi-

aent research paper talked about

applying summarization only to completed

work sections. Cognition's post makes a

similar point, but they apply

summarization in this example at the

interface between different agents and

sub aents. So they kind of use

summarization to compress context such

that in this case sub aent one has a

compression of the context that the

initial agent was using. So it's kind of

a means of information handoff in this

case between linear sub aents. But the

principle is the same in all cases.

Summarization is a very useful technique

for compressing the context in order to

manage overall token bloat when working

with agents. Now it's also worth calling

out that you can use trimming as well.

And you can think kind of think about

this as more selective removal of tokens

that you know are relevant. So you can

use for example heruristics to keep only

recent messages. That's kind of a simple

approach. Or you can use learned

approaches

and this is an LLM based approach for

trimming or context pruning. So we

covered writing, selecting compressing

context. Now let's talk about the final

category of isolating context.

So isolating context involves splitting

up to help an agent perform a task. Now

multi- aent is the most intuitive

example here. So the swarm library from

open AI was designed based upon

separation of concerns where a team of

agents can all have their own context

window and tools and instructions.

Anthropic made this a bit more explicit

in their recent multi-agent researcher

post. They mentioned that the sub agents

operate in parallel with their own

context windows exploring different

aspects of the question simultaneously.

And one of the key points from their

blog post is really that the ability to

use multi- aent expands the number of

tokens that the overall system can

process because each agent has its own

context window and can independently

research a subtopic

and that allows for richer generation of

reports because the system was able to

process more tokens across these various

sub aents. Now there's some other

techniques for context isolation that I

want to call out. I thought hugging

faces open deep research gives an

interesting example. So they use a code

agent. A code agent uses an LM to

generate executable code that contains

whatever tool calls that the agent wants

to run. And the code is just executed in

a sandbox

which will then run all the tools and

selective information can then be passed

back to the LLM

return values standard out variable

names and so forth that the Elm can

reason about. The key point they make is

that this sandbox can actually persist

state over multiple turns. So you don't

have to dump back all the context to the

LLM when you're doing multiple turns of

an agent. this environment can kind of

house a lot of tokenheavy information

like images or audio files so they never

expose to the LM's context window. And

this is another nice trick for isolating

tokenheavy objects from the LM context

window and only selectively passing back

things that you know that the LM will

need to make the next decision.

And I do want to call out that a runtime

state object is another kind of obvious

way to per perform context isolation.

And it's kind of common to create like

for example a data model like a pedantic

model for your state which has different

fields and those fields are just like

different buckets that you can dump

context into and you can selectively

decide what you want to fish out at what

point and then passed to the LM at a

certain stage in your agent. So it's

another very nice and intuitive way to

isolate context.

So we've covered these four categories

writing selecting compressing and

isolating context and talked about a

bunch of examples from popular agents.

Now I want to talk about how langraph

enables all of these.

So first as a preface before you

undertake on contact engineering it's

useful to have at least two things. One

is the ability to actually track tokens

that can be achieved through tracing and

observability. Langmith is a great way

to do that. Also ideally evaluation so

some way to measure the effect of

context engineering effort. Like here's

a good example. Let's say you're using

some kind of context compression. You

want to make sure you didn't actually

degrade the agents behavior. And so

simple evaluation

using lang is a great way to ensure of

that. So those are kind of table stakes

before actually undertake any context

engineering effort.

So let's talk about this idea of writing

context in line graph. So lang graph is

a low-level orchestration framework for

building agents. You can lay agents out

as a set of nodes and edges connecting

those nodes. Like for example, with the

typical tool calling agent, you'll have

one node that makes an LM call, one node

that just executes the tools, and you'll

just bounce between those two. Super

simple layout for a classic tool calling

agent. Now, this notion of a scratch bed

is actually very nicely supported in

Langraph because Lang graph is designed

around the idea of a state object. So,

what happens is in every node, this

state object is accessible and you can

fish anything you want from the state

object and write anything back to it.

The state object is typically defined up

front when you lay out your graph. It

can be for example a dictionary, a type

dict, a padantic model. It's just a data

model. You define it and it's accessible

to you. It's perfect for this notion of

scratchpad

in for example the LM node of your

agent. The agent can take notes. Those

notes can be written to state.

And now that state is checkpointed

across the lifetime of your agent. And

so you can access that state at any node

at any point within the session. And so

it's available to you for example at

future turns. And that's exactly the

intuition around why this idea of a

scratchman is so useful. Agents can

write down things fetch them later. Now

for memory lang is actually designed

with long-term memory as a first class

component. Long-term memory is

accessible in every node of your graph

and you can very easily write to it.

So the key point is within a session,

Langraph has persistence via

checkpointing where agent state is

accessible at every node. If you want to

save things across different

agent sessions, that's also achievable

in Langraph using Langraph's native

built-in long-term memory. Now, I talked

a bit about this previously. How about

selecting context? Well, within

Langraph, you can select, for example,

from state in any node, and that can

serve as a scratch pad. You can also

retrieve from long-term memory in any

node. And what's interesting is

long-term memory can store different

memory types. You can s you can store

simple files. You can also store

collections and use embedding based

similarity search as an example. So you

can see these two resources. Check out

this course from deep learning AI if you

want to learn a lot about the different

memory types. And check out this recent

course on ambient agents if you want a

simple crisp example of long-term memory

in a

longunning email assistant agent. And

the nice thing about that example is the

agent updates memory based on human

feedback. So it really shows that

feedback between human and loop memory

updating and then persisting long-term

memories over time to govern and improve

the behavior of agent in a kind of

virtuous cycle as you give it more

feedback.

Now this pre-built langraph big tool is

actually a really neat example of tool

selection in Langraph. So it uses

exactly the principle that was mentioned

previously

embedding based semantic similarity

search across tool descriptions and you

can see it all right here in this repo

but it's quite an effective way to

select across large collections of tools

and for rag lang graph is very low-level

framework you can implement many

different rag techniques using langraph

I linked to some tutorials that we have

we also have a lot of popular videos on

building rag workflows or agents with

langraph

now How about context compression?

Lapraph has a few useful utilities for

summarizing and trimming message history

when you're building agents which can be

used out of the box. But it's also of

course a low-level framework. So you

flexibility to define logic within each

node of your agent. And one thing I do

frequently is I'll actually have some

logic to post-process certain tokenheavy

tool calls just inside my tool node. So

you can kind of look at what tools

called and then kick off a little

post-processing step depending upon the

tool that was selected by the LLM.

So it's very easy to augment the logic

of your agent in lang graph to

incorporate things like post-processing

because it's just a low-level framework

and you control all the logic within

each node of your agent graph. I can I

show a little example here in our open

deep research.

Now context isolation we actually have a

lot of we've actually done a lot of work

on multi- aent. We have implementations

for both supervisor and swarm. They're

popular open source implementations.

their popular multi-agent

implementations and a bunch of different

videos here that show how to get

started. So multi-agent is something

that has been well supported in Langraph

for a while. Langraph also works nicely

with different environments and

sandboxes. This is a cool repo by Jacob.

You can check out it uses E2B within a

Langraph node to actually do code

execution. And this video here talks

about using a sandbox with langraph but

in that case with state persistence. So

the states persisted within that sandbox

across different turns of the agent

which is exactly what we saw in the

hugging face example and this could be a

very nice way to isolate context within

an environment and prevent it from

flooding back into the context window of

your agent. And finally that you have a

state object. So state objects are are

central in line graph as mentioned

they're available to you within each

node of your graph. You can read from it

you can write to it. You can design the

state object to have a schema. For

example, it can be a pyantic model and

it can have multiple fields in that

schema. So for example, you can have one

field like messages and that's always

exposed to the LM at each turn of your

agent, but you can have other field that

just saves it any other information you

want to keep around but only use at

specific points like maybe towards the

end of your agent trajectory. So it's

very easy to organize and isolate

context in your state object just by

defining a simple schema. So just to

summarize, there's at least four overall

categories for context engineering,

though we've seen across many popular

agents. Writing, selecting, compressing,

and isolating context. Writing context

typically means saving it outside the

context window to help an agent perform

a task. And usually this is so that the

agent can retrieve that context at a

later point in time. Could be a scratch

pad, could be just writing it to a state

object, could be writing to long-term

memories. selecting context. It could be

retrieving tools. Could be retrieving

information from a scratch pad that an

agent is using to accomplish a task

within a given session. Could be

retrieving long-term memories that

provide guidance for an agent based upon

past interactions. It could be just

retrieving relevant knowledge. And that

of course gets into the entire topic of

rag which is a very deep one. Then

there's compressing context summarizing

it, trimming it, basically trying to

retain only the most relevant tokens

required to perform a task and isolating

it. A great way and simple way to do

this is just partitioning context in a

state object. Sandboxing is an

interesting approach to isolate context

from the LM. And of course, there's

multi- aent which involves splitting

contexts up between different sub aents

that perform independent tasks, but they

collectively can increase the overall

number of tokens that the system

processes in order to accomplish a task.

So these are just a few of the

categories. This is very much a moving

and emerging field. This is not a

complete list, but at least it's my

attempt to organize the space, talk

about some interesting examples, and

hopefully give some references about how

to do each of these things in line

graph. Thanks a lot.