Agents on the Canvas in tldraw — Steve Ruiz, tldraw

Hello. Hey, I'm gonna I'm gonna kick off. Uh, sorry, we're starting a little

off. Uh, sorry, we're starting a little bit late here.

I am Steve Ruiz from Teal Draw. Does anyone know Teal Draw?

Yes.

Hey. All right. fans fans in the room.

Uh if you don't know Teal Draw, Teal Draw is kind of kind of a couple of things. Tildraw is a online whiteboard.

things. Tildraw is a online whiteboard.

You can go to it. It's free. It's really

nice. Um

I'm going to be using it for my slides.

Uh Telra is also a startup. We're based

here in London. And Telra is also an SDK, something that you can use to build other products. So, if you've used uh

other products. So, if you've used uh Replet's new um uh agent canvas, then that's built with

our canvas. If you've used uh um Luba

our canvas. If you've used uh um Luba AI's new canvas, that's built with our canvas. If you've used Stitch's new

canvas. If you've used Stitch's new canvas, that's not built with our canvas.

But if you go into this like kind of uh um annotate mode, this actually is our canvas. So we're in

there somewhere. It's a Angular app, you

there somewhere. It's a Angular app, you know. Uh so anyway, um Deal Draw is

know. Uh so anyway, um Deal Draw is again company makes a whiteboard makes a whiteboard SDK. Part of the idea with

whiteboard SDK. Part of the idea with the SDK is that you can build cool things with uh with the SDK, which means it's part of my job to build cool things with the SDK to prove it. Um and a lot

of those things recently have have involved AI, right? hackable canvas

runtime built with React. In fact, the canvas is also just React components.

So, component component component. Um,

which means you can do some pretty cool stuff. Uh, first one, I'm going to be at

stuff. Uh, first one, I'm going to be at the mercy of the demo gods here in the the internet and and other things. So,

uh, bear with me here. Um, does anyone remember this app, Make Real? Maybe

maybe this tweet uh about Make Real. Did

anyone do anyone remember seeing this tweet back in 2023 when the vision models came out? All right, cool. Uh,

this was one of the first projects to kind of like break containment uh in um in AI.

Don't we're going to go to my phone.

[ __ ] this.

Um, which which may may itself be a disaster. Uh so the the basic idea with

disaster. Uh so the the basic idea with make real was that you could um use a canvas draw this and then send that to a a model and and have it make it into a

working prototype which sounds very very quaint in 2026 but in 2023 it was all the rage because there was no lovable there was no kind of vibe coding hadn't been termed or coined as a as a term. Uh

and so this was one of the first projects where nontechnical people could make technical stuff without having to code or to look at code.

And so again, we are at the mercy of the the several [ __ ] various internet uh internet gods. So So we'll see. I'm

internet gods. So So we'll see. I'm

gonna I'm going to let this go for a minute. Um but at the risk of leaking my

minute. Um but at the risk of leaking my API keys, I will try and switch to a faster model. Oh no, it's coming. All

faster model. Oh no, it's coming. All

right. Maybe I just gave it something hard to uh hard to work on. We'll see.

Hang on a second.

Basic idea. Something like that. Uh

yeah, there there there's my API keys. I

always got to rotate them every time I uh do this demo, sadly. Uh we'll see how much how much gets spent before I get done with the talk. Um but it's a very

simple prompt. It's just like go make

simple prompt. It's just like go make this interactive. This is uh this is not

this interactive. This is uh this is not what I asked for, but we'll see. All

right, good job. Cool. And this is a this is a working working thing, which is great. It's like a real little bit of

is great. It's like a real little bit of HTML, but you could also annotate on top of it. You could say like, hey, uh

of it. You could say like, hey, uh actually make this green, and why don't we use these colors? I'm going to make

this red and uh and and black. Um use

these colors, right? And so you're kind of constructing a prompt here, even the prompt that includes the old uh website.

And uh there's not many apps that have actually like kind of used this only now. Only like the the Google Stitch

now. Only like the the Google Stitch that I mentioned before. Uh this idea of well, it didn't do anything that I asked it for. Terrible demo. We're gonna we're

it for. Terrible demo. We're gonna we're going to move on. Things have changed in the last couple years. Anyway,

uh there was another one, teal draw computer, which I'm actually not going to go into, but this was uh a couple of like chains of prompts, but eventually we we were like, "Hey, AI on the canvas

actually might be pretty cool. Maybe we

should um have the AI kind of work as a collaborator, like work with you on the canvas." So, the first one of those was

canvas." So, the first one of those was pretty pretty straightforward idea where you could say like, you know, draw a cat.

You could do anything. you could say um draw me a diagram you know finish my slides complete this graph that I'm working on things like that uh and unlike maybe image models uh like

diffusion models and things like that it's not building an image it is using text structured outputs to to make the same things that I could make right so I

have I have tools I have like circles and shapes and like this uh and it's funny to see how these things have changed over to oh it's sad Um

uh but uh the uh you know the fun of of this is as a way of exploring the model and and what the model knows and how it can comport all that stuff. Uh make the

cat blow out the candle. Um it's pretty cool. You can also do something like

cool. You can also do something like draw a mouse. So multiple um multiple prompts at the same time on different parts. And so even though I didn't tell

parts. And so even though I didn't tell it what a candle was, then I certainly like the application doesn't know what a candle is. Uh and I'm not even sure that

candle is. Uh and I'm not even sure that cats can like blow, but it's correctly interpreted that and kind of

incorporated into the into the design.

So uh with incredible detail as well.

Still sad. Uh so the um this was really fun because this is like solving a lot of problems that might not be obvious like like vision models when it comes to

to structured data. Um number one there's much less vision training data than there is for text. Number two a lot of that training data like conflicts in ways that text does not. Other other

types of things don't. So for example the the y-axis on a on a cartisian graph as you go up that number goes up right?

So 0, one, two, three, four, five. Uh,

on the web, the y- axis goes up in this direction, right? The top left corner is

direction, right? The top left corner is zero. Your top left corner here, but as

zero. Your top left corner here, but as you go down, the y goes up. There's

left, right? Like there's your left, there's stage left, there's all sorts of uh uh things that that conflict within

language uh and within images. So, um,

training the model to kind of behave predictably and and produce things like this is, uh, was really, well, I used training, uh, prompt engineering the model to to do it was was really tricky,

but this was fun. Um, but we felt like it it didn't go really far enough because it was just oneshotting, right?

I wanted to do an agent. So, this is what cursor looked like back in uh, 2025 or something like that when I did this.

um draw a diagram uh of the life cycle of a butterfly.

So this put it into a kind of like agentic loop like you might have seen seen elsewhere um that I'm sure you you interact with dozens of times a day by by now for this crowd um where you have

it produce an output and then review the output and kind of iterate until it thinks it's done. Um and we really tried to hue to the conventions at the time of um you know coding agents that were

where these agents um this agent loop was seen most most often where yeah there's kind of like a lot of sub features like rejection you know seeing

its thinking seeing how it works. Um

great and now we have the uh the butterfly life cycle on the canvas.

Pretty cool. However, this was still not really enough because as as cool as this was, it still felt like I don't know, it felt like I was handing my keyboard to

some some other AI rather than someone collaborating with me. Um although this model has been used really well in uh a lot of design apps that use teal draw uh

things like love art or magic path. Um

and in in education especially where you have this kind of tutor of like help me with my homework and help me fill out my um you know uh oh gosh let's see if I

can do this on the fly. Steve Ruiz uh class you know age whatever. Um, and you can kind of ask it to like uh

complete my D&D character sheet. All

right. And it'll it'll kind of pick up what you're doing and fill out forms and do do fun stuff like this. Maybe I'll

come back to that as it as it kind of chooches along. The what I really wanted

chooches along. The what I really wanted is like to bring the agent out of the sidebar and into the the canvas itself.

Um, oh, I'm a fighter.

Nice. Nice. All right, I'll take that.

Uh, and so we did and we did it with fairies, which maybe maybe you saw, maybe not. Um, these are like little

maybe not. Um, these are like little little guys on on the canvas. You can

kind of throw them around. Uh, they

don't like to be held for very long.

They'll start freaking out. Yeah. Okay.

So, uh, and, uh, but you can do the same thing like, you know, draw a cat or something like that. Now putting the agents on the canvas have a whole bunch of interesting things. You can see the state of the agent, right? These are

multiple agents that I'm kind of running in in coding terms. These would be multiple terminal windows or something like that. Or this would be in compos

like that. Or this would be in compos composer, but you can kind of see what they're doing in a way that uh uh hang on. I'm zoomed way out.

on. I'm zoomed way out.

I did all the sprites myself.

Um and you know, not only can you kind of see its thinking, but you can see its action. And you can see where it where

action. And you can see where it where in the project it's sort of like acting relative to the other um other agents.

So, and you know these other agents can can see each other of what they're doing. So, if I ask this one to draw um

doing. So, if I ask this one to draw um a hat on the cat uh and I draw this one draw the cat's

neck.

We missed we missed the neck.

uh they'll they'll they'll get to work, right? And they're they're able to kind

right? And they're they're able to kind of work with each other's stuff at the same time. Um but we could also ask them

same time. Um but we could also ask them to work together. So if I grab all three of these uh these fairies, Ferris, Helen, and Joan, um draw some more

animals.

Uh one of them will be elected leader.

So, this one is is the leader, and it's going to go scout kind of what's going on on the canvas, and then it's going to create a to-do list, and it's going to delegate that to-do list to the other

agents, right? Um, this is all like

agents, right? Um, this is all like we were doing this in like December, October of last year. Uh, and

we're figuring this stuff out at the same time that that a lot of people were figuring out agent orchestration. This

idea of like, okay, how do we give them shared state? How do we uh, you know,

shared state? How do we uh, you know, have a leader follower? or like how do we manage the fact that these things are essentially blind while they're working and prevent them from kind of overlapping uh in terms of like what

they're doing. And so you can kind of

they're doing. And so you can kind of see the the the leader here isn't doing any of the work. Uh but it is going to kind of like observe. Oh no, that's that's the leader. That's the leader. Um

it's observing uh and uh and and judging and and establishing whether this is like done or not and and whether it's done correctly.

Um, still not enough, right? Fairies are

fun. If you want to play with this, by the way, this is at fairies.taljar.com.

Um, in the same way that make real was a really good introduction to um to just AI at all, right? Draw

something, click a button. Um, fairies

is a great way to talk about like multiple agents kind of working together. Uh, and they can do real work.

together. Uh, and they can do real work.

Let me try and grab a like this is a big description of like an ebook or something like that. And if I summon my fairies uh

uh make this make the make make the wireframes for this app. Cool. And I'll

I'll just kind of let them let them get to work while we keep talking. Um

the started 10 minutes late. I'm going to I'm going to take another five minutes before I jump. Uh the next step for this

one is to to kind of give more access to the canvas to the agents. Um

and there's really we started to kind of run into the barriers of of safety like what is actually safe to to do with our hackable thing for for users. Um because

we have a runtime API, you could just code against it, right? and AI is really good at coding. So maybe we could do some sandboxed, you know, stuff, but no, because we need the DOM, we need the kind of the browser as a way to see what's going on. We need to be able to

generate screenshots, all this stuff.

Um, so we decided to use our uh our desktop app instead. So I over the holidays, I threw together a um an app that Electron wrapper that wrapped Teal

Draw uh and I opened a port essentially.

I said, you know, okay, Claude, like make a little HTTP server and and open a put up an endpoint and anything that gets posted to that endpoint,

uh, treat it as JavaScript and run it, which is a terrible idea. Not a good idea.

Like, don't don't do that on on your app. However, for an an offline desktop

app. However, for an an offline desktop app that is file-based, like what's the worst that could happen? You could hurt yourself, I guess, but you're not going to hurt the rest of me. Uh, and look at them. Look at them going. Building my

them. Look at them going. Building my

little ebook reader. That's fantastic.

Thank you, fairies.

Uh, so so what does that what does that give us, though? Um, I'm going to I'm going to skip the demo where uh, as you can imagine, I could I could say, hey, visualize this code. Make a diagram.

Cool. All right. I'm going to change the the diagram, update the code to match the diagram.

Easy, right? you can have these kind of like let's pull up the the level of abstraction that we want. Um, but the the more surprising stuff was actually where I was like, you know, okay, like check this out. Uh, I'm going to draw a

little user interface or whatever, right? And I want this to be a leg

right? And I want this to be a leg length and I want this to be a t-shirt color.

Uh, and even though TL Draw doesn't really have the ability to, we don't have like primitives for on hover or on click or it's not like it's not a fully thing.

This thing can write code against the editor. So like uh make it interactive

editor. So like uh make it interactive and and we'll see we'll see where it gets to. Um, so far the results on this

gets to. Um, so far the results on this have been like really really cool in ways that are super strange and

disturbing. Uh, because like asking like

disturbing. Uh, because like asking like the AIS are like sure let's do some script script injection, right? Like

that's the way that it documents itself is like this is how you you should do this. Um, it has no qualms at all by the

this. Um, it has no qualms at all by the way changing stuff that's on your desktop on your computer. If you've ever wanted to like for example like we uh one of our team Max was like

you know what I don't like podcasts in my Spotify. I want to get rid of podcast

my Spotify. I want to get rid of podcast in my Spotify app. Claude can you just do that? And it's like sure let me go

do that? And it's like sure let me go through the minified code of the bundle of the thing and let me just rip and tear. Uh and it's happy to do it. It

tear. Uh and it's happy to do it. It

makes them happy. Uh they like it. Um oh

what the [ __ ] was that?

Um, I don't even know what you did. You made

a uh an HTML What the What? Like it it created a new HTML site

What? Like it it created a new HTML site out of this. And this is the pointer.

It's not even a slider. No, I want I wanted it in in the teal draw. All

right. Yeah, we we we love it. It's

blinking as well. I don't know if you caught that.

Come on, do it. Yeah, there we go.

Let's see if it can come on. Let's go.

Um, so yeah, like there's really like uh no limit to what what what it can do with the desktop app and it's happy to do it in a way that um I can almost tell that

it it it uh would love to do this to websites. Like it would love like just

websites. Like it would love like just let me just get my get my claws in there. Um all right, come on. Come on.

there. Um all right, come on. Come on.

Still not working. We're gonna we're gonna we're g we're gonna go set up the interactivity. Come on. Um this is gonna

interactivity. Come on. Um this is gonna be really fun. I think we're going to just release this. Uh we're the um I mean the it is released, but the uh the

notion that you can take like I love local first apps. I love

file over app. I love there's all these ideas that up to now have kind of been curiosities and uh almost like hang on.

Oh, come on. Oh, that's such a such a disappointment.

We're going to have to catch me later.

I'll make it work. Uh but now actually the like the idea of a local file-based thing that is is able to expose itself to to um to to cloud and agents like locally in order to to essentially

script inject kind of motivates a lot of that that stuff which previously was idealistic into like well that's the only way that you could do this. If you

really want to maximize the agency in order to maximize what it can do and take the risk uh and take on that risk, then you kind of just need to hand that to the user and say good luck. Um I

think OpenClaw does this pretty well, but like this these are sharp tools.

Have fun, you know. Anyway, uh that is my agents on the canvas talk. Uh work

continues. If you want to play with the fairies, I highly recommend it because it's super fun and you will find things that surprise you that have surprised

me. Uh they have IRC as well. Let me

me. Uh they have IRC as well. Let me

see. Yeah. Anyway, uh and if you want to follow along with Teal Draw, we are on Twitter X at Teal Draw and

then I'm at Steve and post a lot about this stuff. So, uh thank you for coming.

this stuff. So, uh thank you for coming.

Cheers.