LLM Agents MOOC | UC Berkeley CS294-196 Fall 2024 | LLM Agents: History & Overview by Shunyu Yao

SHUNYU YAO: Cool. Hi, my name is Shunyu Yao. Super glad to be here and talk to you about LLM agents. Brief history and overview. So today's plan is very straightforward. I want to talk about three things. So first, what is LLM agents, to start with? And then second, I want to talk about a brief history of LLM agents, both in the context of LLM and in the context of agents. And lastly, I want to share some ideas on some future directions of agents. So as you know, this field is a moving piece. And it's very big and messy, so it's impossible to cover everything in LLM agents. So I just try to do whatever is in the best of my [INAUDIBLE]. And you can see there's a QR code, and you can scan it and give me feedback. And I can improve the talk accordingly. OK, so let's get started. First, what is LLM agent? Do anyone know-- do anyone think they know the answer? If so, raise a hand. Do have a definition for what is LLM agent? OK, there are maybe three people with it.

So that means this field is really a moving piece. So I think, if we want to define, what is LLM agent, we want to first define the two components. What is LLM? And what is agents? And does everyone know what is LLM? OK. So really, what's left is, we need to define, what's agent? And if you search on Google image, this is agent. But in the context of AI, obviously, we know it's a notoriously broad term. It can refer to a lot of different things, from autonomous car to playing Go, to playing video game to chat bot. So first, what exactly is an agent? So my definition is that it is an intelligent system that can interact with some environment. And depending on different elements, you can have different agents. You can have physical environments, such as robot or autonomous one. And you can have agents that interact with digital environments, such as video games or iPhone. And if you count humans in environment, then chat bot is also some kind of agent. And if you want to define "agent,"

you really need to define, what is "intelligent," and what is "environment?" And what's really interesting is that throughout the history of AI, the definition of what's "intelligent" often changes across time. So like 60 years ago, if you have a very basic chat bot using three lines of rule, then it can be seen intelligent. But right now, even ChatGPT is not surprising anymore. So I think a good question for you all is like, how do you even define "intelligent?" OK, so let's say we have some definition of "agent." Then, what is "LLM agent?" So I really think there are three categories or three concepts. So I think the first level of concept is, what is a text agent? And I think text agent is defined as-- so you have this agent interacting with the environment. And if both the text and the observation is in language, then it's a text. So obviously, you can have text agents that's not using LLMs. And in fact, we have text agents from the beginning of AI, like, several decades ago.

And I think the second level of definition is LLM agent, which is agents that are text agent that are also used LLMs to act. And I think, the last level is what I call reasoning agent. And the idea is those agents use LLMs to reason to act. So right now, you might be confused. What is the difference between the second level and the third level? which I will explain later. So like I said, people have been developing text agent from the beginning of AI. So for example, back in 1960s, there has already been chat bots. So ELIZA is one of the earliest chat bots, and the idea is really simple. You just have a bunch of rules. And what's really interesting is that using a bunch of rules, you can already make a chat bot that's quite human. And what it does is it keeps asking you questions or repeating what you said, and people find it very human.

But obviously, there are limitations to those kind of rule-based agents. As you can see, if you want to design rules, then it often is very task-specific. And for each new domain, you need to develop some new rules. And lastly, those rules not really work beyond the simple domain. Suppose you write many rules to build a chat bot, but then you need to write many rules for a video game agent and so on and so forth. So before LLMs, there was another very popular paradigm, which is to use RL to build text agent. And the idea is, I'm sure everybody has seen video games. So you can imagine text games where, instead of pixels and a keyboard, you're using text as observation and action. And you similarly have rewards. You can similarly use reinforcement learning to optimize reward. And the idea is you can just optimize the reward, and you exhibit some kind of language intelligence. But again, this method are pretty domain-specific. For each new domain, you need to train a new agent.

And it really requires you to have scalable reward signal for the task at hand, which many of the tasks don't. And lastly, it takes extensive training, which is a feature of RL. So really, if you think about the promise of LLMs to revolutionize text agent, this LLMs are really just trained on next-token prediction on massive text corpora. Yet, during inference time, it could be prompted to solve various new tasks. So this kind of generality and feature learning phenomena will be really exciting to build agents. So next, I want to give a brief overview of LLM agents. And it's also like a historical view, and it's obviously very simplified. So I think what's happening is-- so first, we have something like LLM in 2020. I think the beginning of LLM is GPT 3. And then people start to explore that across different tasks, and some tasks happen to be reasoning tasks, such as symbolic question answering and so on. And some tasks happen to be what I call acting tasks.

You can think of games or robotics and so on and so forth. And then we find that this paradigm of reasoning and paradigm of acting start to converge, and we start to build what I call reasoning agent. That's actually quite different from all the previous agents. And from reasoning agent, we start to explore, on one hand, more interesting applications and tasks and domains, such as web interaction or software engineering, or even scientific discovery and so on. And on the other hand, we start to explore new methods, such as memory or learning or planning or multi-agent and so on. So first, I want to introduce what I mean by the paradigm of reasoning and what I mean by the paradigm of acting and how they converge, and what is this paradigm of reasoning agent? And history is always messy. So for now, let's just assume-- let's focus on one task, which is question answering, which can simplify our history discussion a little bit. And then we'll come to more tasks.

So question answering is a very intuitive task. So if you ask a language model, what is 1 plus 2, it will tell you 3. That's question answering. It's very intuitive. So it also happens to be one of the most useful tasks in NLP. So obviously, people try to use language models to do question answering. And then people find a lot of questions, a lot of problems when you try to answer questions. So if you have some question like this, it will be very hard for the transformer language model to just output the answer directly. So it turns out you need some reasoning, and then covering the last talk, like chain of thought reasoning, and so on and so forth. There has been a lot of people investigating how to do better reasoning with language models.

You can also imagine language model trying to answer something like this. And it will probably give the answer wrong because, for example, if a language model is trained before 2024, and the prime minister of UK changed often, as you know, so it might get the answer wrong. So in that case, you need new knowledge. And people are working on that. And for now, example, you can ask something that's really mathematical and really hard. And in that case, you cannot really expect transformers to give the answer. So in some sense, you need some way of doing computation beyond the naive autoregression of transformer. So as you can see, there are many types of question answering tasks. And people find many problems when using language model to answer those questions. And then people come up with various solutions. So for example, if you're trying to solve the problem of computation, what you can do is you can first use a language model to generate a program.

And then this program will run and give you a result. That's the way you can answer a question about prime factorization, or what's a Fibonacci number? So for the problem of knowledge, there is this paradigm of Retrieval-Augmented Generation, RAG. And the idea is very simple. You assume you have some extra corpora, for example, a Wikipedia or this corpora of this company, for example. And then you have a retriever, whether it's a BM25 or DPR, or so on and so forth. You can think of retrieval as a search engine. So what it does is given a question. This retriever will actually just pull the relevant information from the corpora and then append that to the context of the language model so that it's much easier for the language model to answer the question.

So this is a very good paradigm. However, what if there is no copyright for the knowledge or information that you care about? For example, if I care about today's weather in San Francisco, it's very hard to expect any existing copyright to have that. So people also find this solution called tool use. And the idea is you have this natural form of generation, which is to generate a sentence. But then you can introduce some special tokens so that it can invoke two calls. For example, you have a special token for calculator or a special token for a Wikipedia search or a special token for calling a weather API.

So this is very powerful. Obviously, you can augment language models with a lot of different knowledge, information, and even computation. But if you look at this, this is not really a very natural format of text. There's no blog post or a Wikipedia passage on the internet that looks like this. So if you want a language model to generate something like this, you have to fine tune that in this very specific format. And it turns out very hard to call that more than once across the text. So another natural question comes. What if you need both reasoning and knowledge? And people actually come up with a bunch of solutions for different tasks. For example, you can imagine interleaving the chain of thought and the retrieval or generate follow-up questions, and so on and so forth. But We don't need to get into the details of all the methods. I just want to point out, like, the situation at the time was a little scattered. So you have this single task called QA,

but it turns out to be more than a single task. You actually have tens of different benchmarks, and they happen to challenge language models in very different ways. And people come up with solutions for each of the benchmark. So it feels very peaceful, at least for me. And at least for me at the time, the question is, can we really have a very simple and unifying solution? And I think if we want to do that, we really need abstraction beyond individual tasks or methods. We need a higher level abstraction over what's happening. So the abstraction that I find, at least for myself, is the abstraction of reasoning and acting. So what is reasoning? Hope you already know that from Danny's talk last time. So chain of thought-- it's very intuitive, and it's just a very flexible and general way to augment test-time compute and to think for longer during inference time to solve more complex questions. However, if you only do chain of thought, you don't have any external knowledge or tools.

Even the biggest, smartest model in the world does not know the weather in San Francisco today. So if you want to know that, you need an external environment, knowledge, and tools. And what we have to [INAUDIBLE], it's RAG or retrieval or code or tool use, and so on and so forth. In some sense, it's just a paradigm of acting because you're just assuming you're having an agent. And you have various environments, whether it's retrieval, search engine, calculator, API, or Python.

And the benefit of interacting with an external environment is that it's like a flexible and general way to augment knowledge and computation and feedback and so on and so forth. However, it doesn't have reasoning, and we will see later why that's troublesome. So the idea of this word called ReAct is actually very simple. So you have this two paradigm, reasoning and acting. And before we act, language models are either generating reasoning or acting. And for ReAct, the idea is to just generate both. And we will see that it's actually a great way to synergize both in the sense that reasoning can help acting, and acting can help reasoning. And it's actually quite simple, intuitive. You will see later is actually-- you can argue that's how I solved the task or you solve the task. It's a very human way to solve the task, and it's very general across the domain. So the idea of ReAct is very simple. Suppose you want to solve a task. And what you do is you write a prompt,

and the prompt consists of a trajectory that looks like this. So you give an example task. And as a human, you just write down how you think and what you do to solve the task, along with the observation along the way. So if you're trying to answer a question using Google Earth Engine, you just think about some stuff and do some search and then you write down that. And they also write down the result with Google, and you keep doing that until you solve the task. You can give this one example, and then you can give a new task. And given this prompt, the language model will generate a thought and action. And this action is parsed and fed into the external environment, and then that will trigger some observation. And then the thought, action, observation is appended to the context of the language model and then the language model generate the new thought and new action, and so on and so forth. So obviously, you can do that using a single example that's called one-shot prompting.

You can do that with a few examples. That's called few-shot prompting. If you have many, many examples, you can also fine-tune the model to do that. So it's really about a way to use language model, rather than prompting or fine-tuning. So as a concrete example, let's say you want to answer a question. If I have $7 trillion, can I buy Apple, NVIDIA, and Microsoft? I made this slide back in March, and that was a trendy topic at the time. So you can write down a prompt like that. You just say, OK, language model, now you're agent. And you can do two types of actions. You can either Google, or you can finish with the answer. And you just need to write down the thought and action. OK, so that's very intuitive. And let's see what language model do. So this is what GPT-4 does back in March. So we first generate a thought. So first, I need to understand. I need to find what is the market cap of these companies and then add them together so that I

can determine if $7 trillion can buy all three companies. And then this triggers this action to search on Google, and this Google search returns this snippet as a result. Unfortunately, like, it just contains all the market caps you need. So the ReAct acknowledges that. And now I have the market cap, all I just need to do is to add them together. So it uses search engine as a calculator, add them together, and get a result. And it will think, OK, so $7 trillion is not enough. So you need additional money to buy that. I think if asked today, then it's even more money because NVIDIA is much higher now. Yeah, so that's how ReAct solves the task. And you can see it's a very intuitive way, very similar to how humans solve the task. You think about a situation. You do something to get some more knowledge or information. And then based on the information, you think more.

And then I try to be a little more adversarial. So instead of finding all the market caps, I inject this adversarial observation. Nothing is fine. Now, here comes the power of reasoning. So reasoning actually finds a way to adjust the plan and get the action to adapt to the situation. Because the research is not-- result is not found, maybe I can search for individual market cap. So I can just search for the market cap of Apple. And then I try to be adversarial again. I give the stock price instead of the market cap. And here reasoning helps again. Based on the common sense, it figures out, this is probably the price, not the market cap. So if you cannot find the market cap, what you can do is you can find a number of shares. And then you can multiply the number of shares and the stock price to get the market cap. And then you can do that for all three companies, and then you can solve the task. So from the example, you can see that it's not really acting helping reasoning.

Obviously, acting is helping reasoning to get real-time information or doing calculation in this case. But also, recently is constantly guiding the acting to plan the situation and replan the situation based on exceptions. So you can imagine something like this to solve various engineering tasks. All you need to do is to provide different examples and provide different tools. So OK, this is good. We're making progress. But I think what's really cool is. Like this paradigm goes beyond QA. So if you think about it, you can literally use it to solve any task. And to realize this, all you need to realize is many tasks can be turned into a text game. So imagine if you have a video game. What you can do is you can assume you have a video captioning or image captioning model, and you can have some controller that can turn language action into a keyboard action. And then you can literally turn many of the tasks into a text game, and then you can literally use ReAct to solve them.

So it goes well beyond question answering. So after the invention of language model, obviously another part of the history is there are people from reinforcement learning robotics, video games, so on and so forth. They're trying to apply this technique. And there are many works, and I'm only listing one, for example. And the idea is very intuitive, like I said. You can try to turn all the observations into text observation, and then you can try to use a language model to generate a text action. And then you turn the text action into some original format of action, and then you solve the task. But what's the issue of this? So this is an example from a video game where you're trying to do some household tasks in a kitchen. And the problem really is sometimes, it's really hard to directly map observation into the action because, for one, you may have never seen the domain. Second, to process-- from observation to action, you need to think. But if you don't have the thinking paradigm,

all you're doing is just trying to imitate the observation-to-action mapping from the prompt or from the future example. So in this case, in the sink basin one, there is no pepper shaker. So nothing happens. But because it doesn't have the capacity to sink, it will just keep doing that and keep filling because it's like a language model. It's just trying to imitate. So it's not really trained to solve the task, like agent. So what ReAct is, is actually something very simple. You are literally just adding another type of action called thinking. And thinking it's a very interesting action because you can think about anything. So in this video game, you might only be able to go somewhere or pick up something. That's the action space defined by the environment. But you can think about anything. And you can see that the thinking action is very useful because it helps you plan the situation. It helps you keep track of the situation and help you replan if something wrong happens.

So as you can see, ReAct is a general pattern that helps across various tasks. And it's systematically better than if you only do reasoning or only do action. So this is interesting. And I just want to point out why this is interesting from a more theoretic perspective. So again abstraction. So if you think about all the agents that you have, everything from video game to AlphaGo to autonomous car, or whatever, like, all the agents, one common feature is that you have an action space that's defined by the environment. So assume you're solving video game, say, Atari game. Then your action space is left, right, up, down. You can be very good. You can be very bad, but your action space is fixed. And what's really different for language agent or LLM agent or reasoning agent is that you have this augmented action called reasoning. And what's really interesting about this augmented action is that it could be any language. You can think about anything. It's an infinite space.

You can think about a paragraph. You can think about a sentence. You can think about a word. You can think about 10 million tokens. And it doesn't do anything to the word. No matter what you think, it doesn't really change the Earth or the video game you're playing. All it does is it changes your own context. It changes your memory. And then based on that, it changes your follow-up actions.

So that's why I think this new paradigm of reasoning management is different. It's different because reasoning is an internal action for agents, and reasoning has a very special property because it's an infinite space of language.

Cool.

So we've covered the most important part of the talk. I think the history goes on. So from now on, we have the paradigm of reasoning agent. And then we have more methods, more task. And there's a lot of progress, obviously, and I cannot cover everything. So on the methodological side, I just want to cover one thing today, which is long-term memory.

So we just talk about what this reasoning agent. And the idea is you have an external environment, be it video game or Google search engine or your car or whatever. And we just talk about the difference of reasoning agent is that the agent can also think. Another way to think about this is you have an agent that has short-term memory, which is the context window of the language model. And it's interesting that you can append interesting thoughts and actions and observations to this context. But if you look at this context window of the language model, first is append-only. So you can only append new tokens to the context. And you have limited context. So it could be a thousand token two years ago. It could be a million token. Now it could be 10 million tokens next year. But you have a limited size of context. And even let's say, we have a 10 million token window. You might have limited attention. So you can have a lot of distracting things if you're doing a long horizon thing.

And lastly, it is a short-term memory because this kind of memory does not persist over time or over new tasks. So you can imagine, let's say this agent solved Riemann hypothesis today, which is really good. But then unfortunately, if you don't fine-tune a language model, it doesn't change. So next time, you have to solve from scratch again. And there's no guarantee whether it will solve tomorrow. So I think, another one I want to make is it's like a golden fish. So folk wisdom is, a golden fish only has three seconds of memory. So you can solve something remarkable. But if you cannot remember it, then you have to solve it again. And it's really a shame. So hope that's motivating enough to introduce this concept of long-term memory. So it's just like, as a human, you cannot remember every detail every day. But maybe, you may write a diary. That's like a long-term memory. You read and write important stuff for your life, for your future life, important experience, important knowledge,

or important skills. And hopefully, that should persist over a new experience. So you can also imagine a mathematician writing a paper how to prove Riemann hypothesis. That's like a long-term memory, because then you can just read the paper. And you can prove it. You don't have to solve it again. So let's look at a very, very, very simple form of long-term memory in this work called Reflexion, which is a very simple follow-up from ReAct. So let's say you're trying to solve a coding task. This is a task, and you can imagine you can write some program. You can run the program. You can reason. You can do whatever.

But at the end of the day, you test it. And let's say it doesn't work. Some tests failed. So if you do have a lot of memory, then you just have to try again.

But what's different now is if you have a long-term memory, what you can do is you can reflect on your experience. So if you read a program and it failed some test, you can think about it. It's like, oh, I failed this task because I forgot about this corner case. So if I write this program again, I should remember this. And we can do this. You can persist this piece of information over time. Like, when you write this program again, you can literally read this long-term memory. And then you can try to be better next time.

And hopefully, it will improve. So this turns out to be working really well for various tasks, but in particular, coding because for coding, you have great feedback, which is the unit test result. And you can just keep reflecting on your failure or success, and then you can keep track of the experience as a long-term memory. And then you can get better.

Another way to think about this is it's really a new way of doing learning. So if you think about the traditional form of reinforcement learning, so you do something. And then you get a scalar reward. And what you do is essentially trying to backpropagate the reward to update the weights of your policy. And there are many, many algorithms to do that. If you think about a Reflexion, it's really a different way of doing learning because first, you're not using scalar reward. You can use anything. You can use a code execution result. You can use a compiler arrow You can use the feedback from your teacher, which is in text, so on and so forth. And it's not doing learning by gradient descent. It's learning by updating the language. By language, I mean a lot of memory of task knowledge. And then you can think of this language as affecting the future behavior of the policy. So this is only a very simple way, simple form of long-term memory.

And I think follow-up work did more complicated stuff. You will hear about VOYAGER from Jim later, I guess, where you have a memory of code-based skills. And the idea is, for example, you're trying to play Minecraft, and you learn how to build a sword in this API code. Then you can try to remember it. And next time, if you want to kill a zombie, you can first pull the skill of building a sword. You don't have to try it from scratch. And for example, in this work of generative agents, the idea is you have 20 human-like agents in this small town trying to be human. They have jobs. They have life. They have social interaction, and so on and so forth.

You have this episodic form of long-term memory, where you literally-- each agent keeps a log of all the events that's happened every hour. That's like a most detailed possible diary you can possibly have. And you can imagine, later, if you want to do something, you can try to look at the log to decide what to work on. Because if you drop off your kid at this place, you want to retrieve that piece of information. Then you pick it up. You can also have this form of semantic memory where you can-- you can look at your diary, and you can draw some conclusions about other people and yourself. You can realize. You can reflect on that. And you can say, OK, OK, Jim is actually a very curious guy, and I actually like video game. And this kind of knowledge can actually affect your behavior later.

Yeah, so this is long-term memory.

And I think the final step to finishing this part is to realize that you can actually also think of the language model as a form of long-term memory. So you can learn. By learn, I mean improve. You can improve yourself. Or you can say, you can change yourself by either changing your parameters of the neural network, which is to fine-tune your language model, or you can write some piece of code or language or whatever in your long-term memory. And then you can retrieve from it later. So that's just two ways of learning. But if you think of both the neural network and whatever text corpora as both a form of long-term memory, then you have a unified abstraction of learning. And then you have an agent that has this power of reasoning over a special form of short-term memory called context of the language model. And then you can have various form of long-term memory. And in fact, you can show that this is almost just sufficient to express any agent.

So I have this paper called CoALA, which I don't have time to cover today but I encourage you to check out, where the statement is that you can literally just express any agent by the memory, which is where the information is stored, the action space, like, what the agent can do, and the decision-making procedure, basically, given the space of actions, which action you want to take. You can literally express any agent with these three parts, so this is a very clean and sufficient way of thinking about any agent. And I want to leave two questions for you to think, and I have answer in this paper that you can try to retrieve. So the first question is, what makes external environment different from internal memory? So imagine if the agent opens up Google Doc and writes something there. Is that a form of long-term memory? Or is that some kind of action to change the external environment? Or like, imagine if the agent has an archive of internet,

and it tries to retrieve some knowledge from there. Is that a kind of action, or is that a kind of retrieval from long-term memory? I think this question is interesting because if you think about physical agents, like humans or autonomous cars, it's very easy to define what is external and what is internal because for us, what's outside our skin is external. What's inside our skin is internal. It's very easy to define. But I want you to think about for digital agents, how can you even define that? And lastly, how do you even define long-term memory versus short-term memory? Suppose you have a language model context of 10 million tokens. Can that still be called a long-term memory? Note that those terms are defined from human psychology and neuroscience. And think about these two questions.

OK, so we have covered some brief history of LLM agents. I also want to talk about the history of LLM agents in the broader context of agents. We have talked about how we start from LLM to derive various things and all the developments of language agents. But if you look at more agent history, how is reasoning agents different from all the previous paradigms of agents? So here I want to give a very, very minimal history of agents, and it's definitely wrong. So it's just for illustration. Don't take that too seriously. But I think if you want to write a very minimal history of agents in one slide, at the beginning of AI, the paradigm is called symbolic AI. And you have symbolic AI agents. And the idea is like programming. You can program all the rules to interact with all the different kind of environments, and you can have expert system and stuff. And then you have this period of AI winter, and then you have deep learning. And you have this very powerful paradigm of RL agent,

and it's usually deep RL agent where you have a lot of amazing miracles, from Atari to AlphaGo, so on and so forth. And only very recently, we have LLM agents. So this is obviously wrong. But if I have to put things in my slide, this is the perspective. And remember the examples we looked at the beginning of the talk? This is a very typical example of a symbolic AI agent. And LSTM-DQN is very typical example of a deep RL agent in the text domain.

And I think one way to think about the difference between these three paradigms of agents is the problem is the same. So you have some observation from the environment, and you want to create action. You want to take some action. And the difference is, what kind of representation or kind of language do you use to process from the observation to the action? So if you think about symbolic agents, essentially, you're first mapping all the observation into some symbolic state. And then you're trying to use a symbolic state to derive some action. You can think of if else rule. Essentially, you're just trying to map all the possible complex observations into a set of logical expressions. And if you think about all the deep RL agents, a very abstract way of thinking of this is you have many different possible forms of observations. It could be pixel. It could be text. It could be anything. But from a deep RL perspective, it doesn't really matter, because it's mapped

into some kind of embedding. It's processed by a neural network to some vectors or matrices, and then use that to derive some actions.

And in some sense, was what's different for language agent or reasoning agent is that you are literally using language as the intermediate representation to process observation to action. Like, instead of this new neural embedding or this kind of symbolic state, you're literally thinking, language, which is the human way of doing things. And the problem with symbolic state or neural embedding is that, if you think about it, it takes intensive efforts to design those kind of symbolic agents. If you think about how Waymo is built as an autonomous car, you probably write millions of lines of rules and code. And if you think about all those deep RL agents, most of them, it takes millions of steps to train them. And the problem is, those are task-specific. If you write millions of lines of code for autonomous car, you cannot really reuse that for playing video game. Similarly, if you train an agent using deel RL, use a million steps, to play a video game. You cannot use that to drive cars.

Language is very different because first, you don't have to do too much, because you already have rich priors from LLMs. That's why you can prompt to build LLM agents. It's really convenient, and it's very general. You can think about, how do you drive a car. You can think about, how do you play a video game? You can think about, which house to buy, considering mortgage rate and stuff. And thinking is very different from symbolic state and deep RL because the symbolic state and the deep RL vector, they usually have a fixed size. But you can think arbitrarily long. You can think about a paragraph. You can think about a sentence, and that brings this whole new dimension of inference-time scaling. And that's why, fundamentally, reasoning agent is different.

OK, so I just realized I just covered the latter half of the brief history of LLM agents, where we talk about long-term memory and why the methodology is fundamentally different from the previous agents. I also want to briefly talk about the new applications and tasks that LLM agents enabled. So as you can see in the beginning of my talk, the examples are basically question answering and playing games. And that's pretty much the-- if you think about it, that's pretty much the predominant paradigm of NLP and RL. But I think what's really cool about language agents is that it really enables much more applications, and in particular, what I call digital automation. So what I mean by digital automation is imagine if you have an assistant that can help you file reimbursement reports or help you write code, debug, run experiments, help you find relevant papers, help you review paper, help you find papers that are relevant. If all of them can be achieved, then everybody

can graduate undergrad in two years or PhD in three years, or get tenure in three years. Everything can be sped up. But if you think about it, before ChatGPT, there's literally no progress. If you think about Siri, which is the state-of-art digital agent before ChatGPT, it literally can do nothing. And why is that? I think the reason is that you really need to reason over real-world language. If you want to write a code, this paradigm of sequence-to-sequence mapping is not enough. You have to think about what to write and why you write it, and you have to make decision over open-ended actions over long horizon.

But unfortunately, if you think about-- if you look at all the agents benchmarks before the existence of LLMs or LLM agents, they often look something like this. So they are usually very synthetic task, very small scale, and not practical at all. And that's been limiting for the history of LLM agents. Because even if you have the best agents in the world, if you don't have a good task, how can you even show progress? Because, let's say we solve this brain game with 100% accuracy. Then what does it mean? So I think the history of LLM agents on one side is all the math is getting better and better. And equally, if not more important side of the history is we're getting more practical and more scalable tasks. So to have a flavor, this task is called WebShop. And I created it with my coauthors in 2021, 2022. And the idea is you can imagine agents to help you do online shopping. So you give the agent an extraction to find a particular type of product.

It could just browse the web like a human. It could click links. It could type search queries. It could check different products and go back and search again. And if it has to search again, it has to explore different items or think about how to reformulate the query. You can immediately notice the environment is much more practical and much more open-ended than GridWorld. And let's say you find a good product. You can just click all the customization options, and you can click Buy Now. And you can also give a reward from 0 to 1, indicating how good you are solving the task. So it's really like a very standard paradigm of reinforcement learning environment, except that the observation and action is in text. And it turns out to be a very practical environment. And WebShop is interesting because it's the first time people build large-scale, complex environment based on large-scale real internet data. So at the time, we script more than a million Amazon products.

And we build this website. And we build some automatic reward system to tell, if you find a product and here is the instruction, then how can you give a reward to indicate, how matching are the two things? And you can clearly see it's perhaps harder than the GridWorld task because you need to understand not only the images and language in real-world domains, but you also need to make decisions over the long horizon. You have to maybe look, explore 10 different products or in different search queries to find the perfect match.

And for example, on this direction of web interaction, pull-up work has made great progress. Beyond shopping, you can actually solve various tasks on the web. And you can also try to solve other practical tasks, for example, software engineering. So in this example, SWE-Bench is a task where you are given a GitHub repo and an issue. So you are given a bunch of files in a repository. And you are given an issue. This thing doesn't work. Help me fix it. And you're supposed to output-- you're supposed to output a file diff that can resolve the issue. So it's a very clean definition of the task, but it's very hard to solve because if you want to solve it, you have to interact with your code base. You have to create unit tests. You have to run it, and you have to-- you have to try various things, just like a self-engineer.

So another example that I think is really cool is I think the current progress is well beyond digital automation. So in this example from ChemCrow, a work that I really like, the idea is they're using reasoning LLM agents to try to find new chromosomes. And what's really cool is that you give the agent a bunch of data about some chemicals, and you give them access to use tools like Python or internet or whatever. And they could do some analysis and try to propose some kind of new chemical. And also, the action space of the agent is somehow extended into the physical space because the action or the suggestion from the agent is then synthesized in the web lab. And then you can imagine you can get feedback from the web lab. And then you can use that to improve yourself and stuff like that. So I think it's really exciting that you can think of language agent not only as operating in the digital domain, but also in the physical domain,

not only in solving tedious tasks, like book me a DoorDash, but also more intelligent or creative tasks, like self-engineering or scientific discovery. OK, so great. So we have covered this slide finally. So in summary, I have talked about how we start from LLM. We have this paradigm of reasoning. We have this paradigm of acting the converge, and that brings up more diverse tasks and methods. And we have also covered a more broader timescale, the paradigms of agents, and why this time, it's different, and also from a task perspective. So the previous paradigm of task-- if you think about AI, you can think of games. You can think of simulations. You can think of robotics. But really, LLM agents bring up this new dimension of task, which is to automate various things in the digital world. So we have covered a lot of history, and I just want to summarize a little bit in terms of lessons for doing research. So I think, personally, as you can see,

it turns out some of the most important work, sometimes the most simple work. You can argue, a train of thought is incredibly simple. And ReAct is incredibly simple. And simple is good because simple means general. If you have something extremely simple, then you have probably something extremely general. And that probably is the best research.

But it's hard. It's hard to be simple. So if you want to be simple in general, you need to both have the ability to think in abstraction. So you have to jump out of individual tasks or data points. You have to think in a higher level. But you also need to be very familiar with the individual task, the data, the problem you're trying to solve. So note that you can actually-- it could actually be distracting to be very familiar with all the task-specific methods. So remember, in the history of QA, I covered all this-- a lot of task-specific methods. If you are very familiar with them, then you might end up trying to create an incremental solution after that. But if you're familiar with not only QA, but a lot of different tasks and you can think in abstraction, then you can propose something simpler and more general. And in this case, I think really learning the history helps. And learning other subjects helps because they provide you some practice for how

to build the abstraction, and they provide ways to think in abstraction.

OK, so this is mostly the talk. I think I will just briefly talk about some thoughts on the future of LLM agents. So everything before this slide is history, and everything after this slide is the state of art or the future.

Obviously, the future is very multi-dimensional. There are many directions that are very exciting to work on. I talk about-- I want to talk about five key words that I think are truly exciting topics that are, first, very new in the sense that if you get to work on this now, there might be a lot of low-hanging fruit. Or you might have a chance to create some very fundamental results. The second is somehow doable in the academia setup, so you don't have to be OpenAI to do this. But it's still good to be OpenAI.

So these five topics actually correspond to three recent work that I did, and I will only cover them briefly. And if you have more interest, you should check out those papers yourself. So the topic is first training. How can we train models for agents? Where can we have the data? Second interface. How can we build an environment for our agents? Third robustness. How can we make sure things actually work in real life? Fourth human. How can we make sure things actually work in real life with human? And lastly, benchmark. How can we build good benchmarks? So first, training. So I think it's interesting to note that up until this year, language model and agent are kind of disentangled in the sense that the people that are training models and the people building agents are the different people. And the paradigm is that the model-building people build some model. And then the agent-building people build some agents on top of it using some fine-tuning or some prompting, mostly prompting.

However, these models are not trained for agents. So if you think about the historical route of language model, it's just a model that's trained to predict text. What people can never imagine, is when they use to solve chemical discovery or self-engineering. So that brings the issue of discrepancy of the data. So it's not trained to do those things, but then it's prompted to do those things. So the performance is not optimal. And one solution to do that is-- one solution to fix this is you should train models, probability for agents. And once you can do this, you can use those prompted agents to generate a lot of data. And then you can use those data to fine-tune the model to be better at agents. And this is really good because first, you can improve all the agent capabilities not covered in internet. So you can imagine, on the internet, which is the predominant source of language model training, there is not a lot of self-evaluation kind of data.

People only give you a well-written blog post, but no one really releases all the thought process and action process of how to write the blog post. But that's actually what matters for agent training. So you can actually prompt agents to have those trajectories, and you can train models on those things. And that's, I think, really one way to fix the data problem, because we all know internet data is running out. And how can we have the next trillion dollars to train models? This is very exciting and, I think, a very-- like, maybe not best analog is you can think of the synergy between GPU and deep learning. Because GPU first was not designed for deep learning. It was first designed to play games. And then people explore the usage and find, oh, it's very good for deep learning. And then what happens is that not only people use existing GPUs to build better deep learning algorithms, but also the GPU builders create better GPUs to fit the deep learning algorithms.

You can build a GPU specifically for transformer or so on and so forth. I think we should also establish the synergy between model and agent. The second topic is interface. And in fact, human-computer interface has been a subject for decades. It has been a great topic in computer science. And really, the idea is if you really cannot optimize the agent, you can optimize the environment. Because if you're trying to write a code, even if you're the same person, it makes a difference whether you're doing that in the text editor interface or the VS Code interface. You're still the same you. You're not being more smart. But if you have a better environment, then you can solve the task better. I think the same thing happens for agents. So it's a very concrete example. You can imagine, how can the agent search files in OS? So the human interface in terminal, as we all know, is to use LS and CD, and so on and so forth. It works for humans, but it's not the best interface for agents.

You can also do something like, you can define a new command called Search. And then it will give a result. And then you can use this action called Next to get the next result. But it's probably still not the best for language model. So in this research called SWE-agent, what we find is that what turns out to be the best way to help agents search files is to have this specific command called search. And instead of giving one result at a time, you can just give 10 result at a time and then use-- let the agent decide, what's the best file to look at? And you can actually do experiments and show-- like, you can use the same language model. You can use the same agent prompt, but the interface matters for downstream tasks. So I think this is a very, very exciting topic, and it's only getting started. And it's a great research topic for academia. You don't need to have a lot of GPUs to do that. And it's interesting because models and humans are different, so should their interfaces.

You cannot expect language models to use VS Code to be the best code interface. So there must be something different, and we need to explore that. And in this case, you can think of the difference as being that human-- we just have a smaller short-term memory. If I give you 10 results at the same time, you cannot just read them. That's why, for human interface, you have to design that in an intuitive way. You have a Next button. If you do Control-F, you can only read one thing at a time. But actually, for models, it's worse because models have a longer context window. So if you Control-F, you should just probably give everything to the model. So if you design better interface, it could help you solve tasks better with agents. It can also help you understand agents better. It can help you understand some of the fundamental differences of humans and models.

Lastly, I want to point out this topic of human in the loop and robustness. So I just want to point out, there is this very big discrepancy between existing benchmarking and what people really care in the real world. So you can think of a very typical agent task or AI task, say coding with unit test. And this is a plot from AlphaCode 2. And basically, the idea is, if you sample more times, then the chance that you have a right submission increases. That's very intuitive. And if you have a unit test, then you can sample many, many times, obviously. And what you really care is what we call pass at K. What you really care about is, can I solve it one time out of 10,000 times or 1,000 times or a million times? It's like solving Riemann hypothesis. You just need to do it once. What you care about is, if you sample 10 million times, can you solve it once? But if you think about most of the jobs in the real world, it's more about robustness. So suppose you're trying to do customer service.

Like, LLM agents are deployed for customer service, but sometimes they're elusive. And there are consequences. And if it does something, the company might have compensation and stuff like that. So arguably, customer service is much easier than coding or proving Riemann hypothesis, at least for human. But here it really presents a different challenge because what you care about is not how you solve it one time out of a thousand times. What you care about is, can you solve it a thousand times out of a thousand times? You care about, will I fail it one time out of a thousand times? Because if you fail one time, then you might lose a customer. So it's more about getting simple things done reliably. So I think, really, that calls for a different way of doing benchmarking. And we have this recent work called Tau-bench. And the idea is first, you have a very practical task, which is to do customer service. Second, the agent is not only interacting

with some kind of environment, like digital environment. It's also interacting with a human, but it's a simulated human. So the idea is the customer service agent, just like a human customer service agent, needs to interact with both the backend API of the company, and also some kind of user. And the agent really needs to interact with both to solve the task, and the trajectory might look something like this. So the human might not give you all the information at the beginning, which is a predominant paradigm of all the tasks right now, if you think about self-engineering, and so on and so forth. Imagine seeing something like change flight. And then you might need to actually prompt the user, oh, can you tell me, which flight are you changing? And you need to interact with the user over multiple turns to figure out what they need and to help them. And this is very different. And also, it makes the metric that you care about. So you can imagine, for the same task,

you can sample the trajectory multiple times with the same user simulation. So if you look at the dashed line, it's called pass@k, which is measuring-- if you sample 10 times, can you solve it at least one time? And obviously, as you sample more, the chance that you solve at least one time increases. But here, you don't care about whether you can solve it one time out of 10 times. You care about whether you can solve it 10 times out of 10 times because otherwise, you will lose client. You might lose customers. So the solid line measures, as you sample more, what's the chance you can always solve the task across all the possible task? And what we see in today's language model is that obviously, they have different starting points, meaning, obviously, they have different capabilities. But what's really concerning is they all have this decreasing trend. So if you sample more, the robustness always go down. Like, from the small model to big model, they all have the similar trend.

The ideal trend should be something more flat. So if you can solve something, you should be more reliable to solve the same thing over time. So I just want to point out that I think we also need some more efforts in taking more real-world elements into benchmarking, and that requires new settings and metrics. So we have this blog post that talks about some thoughts on the future of language agents, and one way to think about that is you want to think about what kind of jobs they can replace. And if you think about it, maybe the first type of task is not that intelligent, but really require robustness, if you think about simple debugging or doing customer service or doing simple assistant over time and time. And second, you need to collaborate with humans. And third, you might need to do very hard task. You might need to write a survey from scratch or discover a new chromosome. And that requires some new type of ways for the agents to explore on its own.

But I think, it's in general, very useful to think about what jobs they can replace and why they're not replacing those human jobs yet. And what are missing, and how can we improve it?

Lastly, lastly, this is a limited time, and we're going to have an EMNLP tutorial on language agents in November. And it will be three hours, so hopefully, it will be more comprehensive than this.