Sentence Constructor - Anthropic Claude

all right I am back and now let's think about um how we're going to write up all these parts okay um so I'm going to just

pull this off screen I can obviously still see it uh you cannot and I'm just going to keep referencing it I'm going to put it up actually up here because it's a little bit hard to put it to the

uh left of the screen here I'm going to go back over to our document here and I just want to have a a a think of how

this is going so let's go over here and just start defining our components so um call this like States or

um agent flow the following agent has the following agent has the

following states we have here a setup we have I'm just looking here off screen

attempts attempt and clues okay um each state expects the

following kinds of inputs and outputs so for state for setup

State we expect uh student input for input we uh for input we expect um Target English

sentence we have uh for output vocabulary table sentence structure Clues considerations next steps

right and uh I mean we didn't really even call that we should really be more uh concise in terms of uh what those are um but yeah so that's for that state

and then we'll continue on to our next one again I'm just looking at this offscreen of exactly what it is right and then bringing those those words over

here so we'll go here and say um attempt so input here is going to be Japanese sentence

attempt and then our output is going to be and we could also be very particular and say like student input where we'll say user

input assistant output user input assistant

output um and I'm just going to copy paste these here my hands are getting really cold in the office here and uh it's really affecting my

typing and so we'll go ahead and copy this and then we have Clues and we have our user input just a second okay so I was just

warming up my hand hands there's a heater right beside me um but uh we're going to go here and um these are going to be indented

like this this oh that's the assistant up here there we go okay and so we have those there I'm pretty sure um Clues just have that and then we have yeah we

bring the assistant over here it's crazy how much your hands will slow down if your hands are cold I always forget that that's the biggest like I would say I don't mind the coal other than it just slows my

hands down um so now we have our clue section and so we've defined our uh the components that we expect to be there so each state expects the following kinds

of inputs and outputs um and expects the kinds of inputs and outputs and so we might even want to say like those are components of it so each

Spate expects uh uh kinds of inputs and outputs inputs and outputs uh contain

expected uh component components of text okay and so hopefully that is clear enough for for Claude um but now we can kind of Define those down below so we

have formatting instructions about these components and instead of saying formatting conru uh instructions we could just talk about those components and so now we can just go

here and just say components right and we have our vocabulary table we have our sentence structure which is somewhere here we have uh Clues

considerations the next steps which I've pretty much renamed okay and then we also have

um uh we have Japanese sentence attempt we we have the target English sentence we have um here the student

question we can tell that it is a question when the input it input uh sounds like a question about language

learning then we should assume that we are in this state oh actually well I mean like when the input

sounds like a question about language learning then we can assume uh we are uh the

user is prompting to enter the clues State uh except

if we are already in anyway we have that there right and so here uh we have the Japanese uh sentence

attempt so when the input is Japanese text then the student is making a Japanese an attempt

at the answer and then here we have when the input is English text um then it's possible the student

is setting up the lesson the transcri the transcribing the transcription to be around this text of

English the other thing is that we can also Define the states of flow so setup States agent flow the following agent States each States contains the f

following um and I would say states can has Thea have the following transitions setup which is the start setup um and we had a visual here so we

can kind of see it here right and so we have setup to attempt

or set up to question or we have um Clues to attempt technically we can go the other

way as well right we can go here and we can say attempt to Clues to [Music]

Clues and we have attempt back to set up okay the agent has the following states

the the starting state is is always set up okay and so now we are being very explicit about our state management for

our stuff um and I mean something that we could just do like we have like rules here but

the more examples we provide um I would say the more that we can um take out of here assuming like that's what I'm assuming here um

so another thing that we could do is we could also tell it to make more examples for us and then we could tweak them from there so yeah I'm just trying to uh

think about that because again we have sentence construct uh sentence structure examples which are good I mean yeah we can provide full examples or structured examples I don't

know but this is a good start for for this so I want to go ahead and just see if it can do State Management the other thing is that we need to tell it um you know

uh tell us at the start of each output what state we are in okay and it doesn't know about States until later on so it might be better to

for us to tell that later because the order of the stuff does kind of matter but I'm going to go over here to um uh here and I just want to see what ticket

number are we're on number three so just say um set up State Management okay I'm going to go ahead here and commit

this and I'm going to make my way over here and make a new project I'm going to say this uh JP sentence Constructor new it's better and these don't do

anything to the project but the nice thing about a project is you can upload files and if your files get too large then you kind of have to um uh start using projects but it we'll go over to

here and I'm just going to go grab this and I'm going to copy it here and we'll go paste and save the

instructions um and so now I have my examples file over actually have two example files and I'm going to go ahead and download this one and I'm going to download this

one here this where be a little bit more uh uh useful to be working locally because I don't have to rename the files constantly but I'm going to go over to here I'm going to upload these new files so we'll go ahead and bring these on

over and so now we are set up I notice we only have 1% of our knowledge capacity so we could provide a lot of examples here and so I'm going to

just say um I need example sentence it's like it's very cold in the office so it's hard for me to type on my computer

and we'll go ahead and see what we get back and told us what state we are in and we have uh an

example and it's fine like I'm not sure why it say difficult here but we never said that we couldn't um we couldn't do things like that and we'd have to think a bit more about it and so here we have considerations we

have two separate thoughts the first thought has two parts the first part asks about seeing something specific the second part thought to describe each verb will uh will need different conjugations based on their stuff try

conru in just the first question about seeing the Raven ask about the expression expression patterns in Japanese okay um so we have this and we could attempt each of those things as

it's telling us to do it's a very cold office so it's hard for me to type on my computer and we do have some examples

so it's very cold in the office um so there's no particular time we are in our why is it showing

Raven ah weird why why did you show Raven in the table of contents in the in

the vocabulary okay it was in there right Raven yeah looking at the state of English was not directly zerve I made the Assumption

there okay but but why but but why do you think I think you included it what was the teaching mistake so looking at examples. XML you provided

one of the core feedback points was not was about not providing unnecessary vocabulary I copy one of the example sentences did you see the Raven from the sentence structure unconsciously

included in the vocabulary so that's kind of interesting so I think what we'd have to do to fix that is we have to go back to our examples right here and we needed to vary the examples

so could you well actually we just say like can you fix the examples file uh so that it that doesn't happen again because I would think that what would

have to do is vary the file right and that's what it might try to do okay uh looking at the sence seure I can identify that having examples using

specific nouns like Raven well I mean is is that the problem or or was it because Raven because Raven was because Raven was repeated so many times all right and so it doesn't even

know itself and so it's going to copy and make us a new one okay can you can you format that as an XML

file this is a bit of a mess that it's providing it to us that way no it's not going to do a better job okay so um not really what I want but we'll go

back over to here and I have a feeling if I paste it in here oh no that one been fine okay it did right no no it didn't it has both of

them let's go ahead here try this again we'll paste it in so now we have one that is more varied so maybe this will give us less problems I'm going to go ahead and delete delete this here

and let's go ahead and download this but this is where you know you might have to tell it like hey do not use the exact examples and make sure you're being conscious about what you're putting in here so I'm going to go back over to

here and so I'm going to go delete this example out I'm going to bring in my new example right and I'm going to say uh the office uh is

cold therefore I cannot type fast on my computer

okay and now we have broken okay so why is the table broken why are you mentioning Raven and

garden and so I think the problem here now is we don't have enough examples right oh you know what um I think maybe the reason we're having another problem is

that when we C keep copying The Prompt it has this thing at the bottom okay so maybe that wasn't our problem here it was our prompt this entire time so let's go ahead and save this I'm going to go over to here this is a easy

fix got to be really careful with this stuff and we'll go all the way down the bottom and we'll save the instructions

we'll try this again so it is cold in my office therefore I therefore I need to turn on the heater

okay so we'll do that okay so now we're having less of an issue so I think that's where the confusion came from and that other examples might have not been an issue but the thing is is that we probably

want to vary things anyway because if we provide too many of the same examples it might start leaning that way um but notice here we now have our uh setup so we can see exactly what state it's

thinking in and that's really good because now we can expand a lot more examples and information for that um but yeah we have this there's a lot there's

just like too much text here um you're providing way too much text for consideration for

consideration for can you can you summarize this more okay so here's where we again we could provide another example and this is

where we could say like um considerations example considerations examples XML and I'm going to go here and just

say examples examples example example and then we'll go here and give it a

score and we will say like this is six and the other one's a 10 and I'll say reason like score reason so it knows score

reason we can also do it that way you'll see I'm like I'm varying I'm not being very consistent here we can also do score and then we could put value here but uh uh uh uh

concise uh this example scores 10 because the uh because the returned

information returned uh information is concise okay and then we'll go up to this one because there uh The

Returned information is to verose okay and then we'll go here and just say uh we have example example and

then actual text or output output scores this this output scores that and now what we can do is go ahead

and grab uh those two examples so I'm going to grab this one and this one is really good I like this one okay and and then we can go up to

here this one is has way too much stuff going on there I don't have time to read that and we're beginners we don't need that much information right um and yeah these are set up

as this there we go okay so now we have that example I'm not sure why there's a period in between that I'm going to go ahead and well I'm going to just try to be a bit consistent

here we'll see consider additions example and I'm going to have to update my uh uh my prompt document here and if we

go under to our um here we can copy this and we'll say whoops that's not what I wanted I just wanted this

line reference the file sentence structure examples so this one is considerations examples for good um terms of conciseness okay and so we have that um

I'm going to need to get my new prompts we'll go ahead and copy this one I'm going to save this here this will be three and we'll just say um

provide more uh isolate examples for consideration fix issues with um uh unne U

unnecessary vocabulary what interesting is like if we are successful at this level then we could possibly take this down to ha cou and then we might get okay

performance um but we'll go ahead and copy this here I wouldn't really call it knowledge transfer as it's not really the same thing but um we'll go back over to here I'm going to go ahead and update

this here and we'll paste this in as such and I will fix this here we'll save that it's still going to know what I mean um

send instruction examples is fine but we need to go ahead and download this one here which is our um considerations example yeah consider considerations

example and we'll bring in this text so now we have oh we actually have two in here that's not good which is the old one which is the new one oh this one just came in here so I don't need that

this one came in eight seconds ago I must have copied two by accident it's not a big deal and so I'm going to say

um uh it is cold in my office therefore I am going to uh stop working and go

back into my warm house can you tell there there's forced there's forced is not a word um but Samui that's definitely what it's

getting in here uh gimu show G show is office Shoto EA at atat atat nobody wants to hear me try to

speak Japanese but now we're getting a concise example so you can see how we can be uh keep breaking them up like that and get better and better results uh the question is could this

model perform well if we went over to Hau I'm sure it would let's go ahead and try this so um I am hungry

because I haven't eaten in five hours let's go ahead and give that a go it's not true I've I've eaten well maybe actually it's been five hours I'm not sure but um I mean this is performing

just as good the only thing is that it's now producing type and we never told it to do that and we're getting word and we never told it to do that um it is doing

cheat sheet which is kind of interesting so it's following it uh more strictly to the convention that we produced there it's saying reason verb reason so it's interesting that Haiku is not performing

as well but uh yeah maybe just takes a lot more effort to get Haiku to work um the way the way we want but we would really need to increase our examples there are ways that we could just create

more examples we could just keep feeding it in and saying like hey like at the smarter model and say give me 10 or 20 examples of more variants of these and maybe that would help ground it a lot

better um because it has more examples but that's an experiment that we would have to do um I'm pretty happy with the result that we're getting here but notice oh it's still giv us the state but yeah notice that we aren't getting

the best results with these um smaller models and by smaller models these models are still really really really large uh we could go back to meta Ai and try to feed in these documents and

Stitch them all into one and see how they perform um but yeah there's just a a challenge where you get these models and these models are just too small to really get the result and that's where

you might need find tuning or um some other kind of result but um or what you need to do is break the task down to be even smaller so we have it doing quite a bit I know it's just doing sentence

construction but um there are a lot of moving Parts there and so there could be a way that we could um slice up the learning task in different ways um but

uh like maybe just focus on particles so like you give it you get you produce a sentence and then you just have to figure what particle uh goes in the right slot that might be easier for a

smaller model whereas this one's uh getting a much broader or general task the other question is like how do we know that that this is well tested right like obviously we're just using it here

but what could we do to make this thing better so you know how can I make my like I'm going to go over just a regular CLA Let's go ask Claude claude's pretty good

so if I design a prompt um with examples how am I how can

I Mass test or like how can I well let's go back here okay um I want you to help me make the prompt

document better how can I what like what things can I do to test my um document in multiple

ways right it's like it's probably going to think I want to do a translation and notice it didn't actually follow along with it which is fine so here it says I help you analyze improve your prompt document through various so test different sentence

complexities so simple sentences compound sentences negative sentences test edge cases for vocabulary tables I mean test State transitions test

teaching scenarios an example of common student mistakes include scenarios where the students use correct particles so I almost kind of Wonder like could we generate all this stuff could like could

you generate all this stuff the only problem is like even if it generates it up for you you still have to as a person go through all of it and comb through it but I I was thinking

more just like ways where people might try to break break the document so here we're getting examples yeah I guess it's fine it's giving us all in this thing I actually this is I actually kind of like this

quite a bit um the structure but yeah I guess the thing is like could we copy paste it and then um what if we took these and then we brought it over to houp would it perform

a lot better um but that we go here and it's still producing a lot of the stuff so let's go ahead and yeah you know I'm just going to go ahead and download this

file and I'm going to go over back to here I'm going to bring it on in here because I'm really curious to see if what it might do is is if we use hi cou like would it make it work a bit

better um and I think this is actually an XML file I'm not sure why it's it's saying MD we'll go over to here oh I guess it is it's this and so um

I'm going to go put the top of the The Prompt file here oh maybe down here I'll DB it to the bottom it sometimes matters where you put it but I'll just just go here and say like it's

not what I want I want this um please read this file and um so you can see more examples

um to better provide p uh provide Prov better out provide to provide better output provide

better output and so this one is called um it's been a little bit silly here let me go over here oh this is being kind of a pain

okay we'll say file file and this one is called um Japanese

teaching test. MD so it's it's a

teaching test. MD so it's it's a markdown file we'll just fine and so I'm going to go ahead and copy this now and we'll go back over to here and I'm going to edit this

here and we'll save that there and now I just need to upload this new one so I'm going to upload that there and so now we're going to choose

Haiku I say um it's cold in my office I need to go inside but it is

dark out I hope I don't see any bears though since it's winter the Bears are sleeping and it's still

buros okay did you read the example files it's almost like it didn't do that okay but okay so notice I told it to

read it okay okay but the vocab table is not [Music] right vocab

table is still not right okay like how many columns are supposed to

be in the VOC uh in the vocab table there we go okay and so I don't know there's something that can be done here um but like if we know that it's always messing up on the same mistakes

uh we could just tell it like hey at the end like read this and make sure you don't make these mistakes because it's consumed all this knowledge and then when it gets to the end of it it's like confused and so that could be something

that we do where um uh we go all the way down the bottom here and just make sure you read all the um

example files tell me that you have make sure you check how many columns there

are in the vocab table so it's almost kind of like a checklist of like hey check these things over last minute which seems really silly but this might get us closer to what we want and sometimes you can front load that stuff

so it becomes more important um but we're going to go back over to here I'm going to try this one more time we're going to paste it in here I'm going to

hit save um I want to go to sleep because it is getting late we'll go ahead and give that a

go okay it's telling us that it read it and so now we're actually getting good results and now it's concise okay so we have tricked Haiku to work how we want it to work so we can definitely get

these smaller models to do what we want but this one's actually really simple it's just reason not exactly that but so we have um like here we might say like

hey go read um go read the did you would say like did you read the sentence structure

well okay so that one's getting better right and so I can go back here I'm it's like make

sure make sure you read the sentence structure examples file okay and I'm going to go ahead and try this

again but that's what I this is what I kind of remember the old model doing where I would have to tell it to do that it was very annoying um but I eventually got it to work and so we'll go here

whoops uh we want the whole file here this will be like the last change we make and then we'll just we'll call this call quits here so I am

tired because it is late so I want to go to bed okay and so I I need to make sure I switch this over to Haiku maybe the last

one did really well because it was um it wasn't Haiku um subject reason verb okay did

you read reason subject verb I mean kind of moved it around I guess it's fine but um maybe this is where we need to provide more bad examples of things we don't like and I think that's the thing is that if we

only have good examples we don't have bad examples I the smaller model can't do as well but we're going to call this done um and then after this we'll do probably have another video here of a postmortem I know that we're supposed to

have other guest instructors do other variants of this so it would be interesting to see what their outcomes are and I will be hanging out with them we'll go ahead here and just say um uh more example files okay for this thing

here I'm going to go ahead commit it push it that's all good and I will see you in the next one okay ciao