OpenAI and Ex-Shopify Product Lead Reveals How He Builds AI Products | Miqdad Jaffer

so when you build normal consumer products you have goals you have a path you have some requirements you have an estimate and then you kind of work throughout that and then you work on the edge cases in case there are any to be

able to to narrow in on but there is a deterministic path toward doing it in a non-determinate system it's very hard to predict what the error cases are going to be and it's very hard to plan for

every Edge case along the way to try to defend against it so the first thing that we had to do was assume that everything that we possibly thought would go wrong will go wrong we don't

want the AI to do something without the merchant knowing that thing is happening how do we push everything in front of the user in a way that they can review they can look over and they maintain full control so that this is

augmentation not Automation and that that's kind of where we started with a lot of our thinking on this and what made that unique is the way that we build is not through like one use case at a time in some cases it's like a data

set at a time of like this is what we need to be able to like understand this is the problem space that we need to understand and there are different ways to approach it you can find two and you can do something from retrieval augmented generation and then you can do

something just on General prompting but each approach has pros and cons to it so like being able to balance and evaluate on an ongoing basis while knowing that your evaluation is never going to be comprehensive and there's going to be

errors and so having that full Corpus of unknown means that we build from our principles and stick to our principles as we build versus having something linear that we know will work my guest

today is mikdad Jaffer X AI lead at Shopify and current product lead at open AI I've always admired shopify's craft Centric and media inverse culture so I

was eager to explore how they applied it to building AI products we spoke about how Shopify uses its get done process to accelerate its AI efforts how

Shopify built psychic its AI assistant for merchants and how Shopify prioritize quality and craft across all of his AI products keep listening if you want to

learn how generative AI products are actually built if you enjoy our conversation please like And subscribe to support this podcast all right mcdad welcome so let's start with a broad

question shopify's mission is to make Commerce better for everyone I'm curious how you got shopify's generative AI efforts started very so annual planning

wise we do where a situation where Toby writes his themes and he writes them in the form of a letter and the letter is from a merchant to Shopify and I love

Shopify because and one of the themes on there was Shopify keeps me on The Cutting Edge and he talks a lot about how AI is like a big part of that but

that letter happened towards the end of the year and what happened was November is when chadu kind of came out and did like Breakaway party of here's

generative Ai and the question that was asked was can we do something with this and that question comes from like this The Cutting Edge how do we get our

Merchant to get on this Cutting Edge and the simple answer was yes I there are a lot of things that we can do with this I think we can start with something that's pretty straightforward of product

description and why do we pick product descriptions there's th n billion products in Shopify and many of them didn't have a product description and we thought there's no better use for

generative AI then trying to fill in something that is such a regular task and happens so many times for merchant this is a good place to start so we put

together a team we put together a goal of let's get something out in front of our Merchant so that we can see how they'll use this whether they'll use this and what we can learn from it and

we launched something with our winter Edition in February at this point and there was a lot of popularity to it there was a lot of merchants using it

there was a lot of interest in it and our goal was can we keep this net neutral like that neutral to positive was our goal of how do we make sure that

at least this doesn't hurt anybody and it provide and at best just provides something valuable but like let's get the technology out there to test and feedback was great everyone started asking for more features more

capabilities but ultimately the thing they wanted was can you put this everywhere and so that's where the whole thing started oh that's great so you actually just found like a small

specific use case and just ship shipped it and got the reaction as opposed to right like a big strategy document or something yeah I I think it it's important to to get something in the hands of the users quickly and this was

like we could have wrote a strategy Doc and we could have said like this is our overall path for thing we didn't know whether this was going to be used or not so we fig we just find out awesome so

can you talk a little bit about shopify's git B system yeah and like how did you shepher the aiers through that

yeah so our GSD process is is of a few phases there's a proposal where you put together like a one pager to say that this is the thing that I want to do this is the problem that I hope to solve here are the principles I want to kind of

operate by and this is what it's going to look like then that goes through uh a review process and then we go into a prototype phase and in the Prototype phase we're testing so a lot of it is

sometimes we'll do ux things sometimes we'll do technology things but what we're trying to do is can the solution that we're proposing actually be a solve for what it is that we the problem that

we want to solve for and in this case was The Proposal was a symbol of like hey we want to put product descriptions in this is how many Merchants are affected by it this is how many blank product prescriptions there are which is what people write and the people that do

have product prescriptions do tend to perform better so we think this is a good place and we think we can do that with generative Ai and we think we can do that with our little bit of ux and so we tried a bunch of different ux pattern

there was like interruptive ones non-interruptive ones and we landed on a set of principles one principle being the user has ultimate decision-making power they are the ones that get to

decide what the message is going to be to their buyers and we don't want to interrupt that flow we want to be augmenting that flow and so all of our ux was around that and we went through a

proposal many different ideas we went through trying to prompt the AI and we realized okay we need more information so we started to build some embedding sources we started to build some like

rag based approaches of like what do we add in additional context wise to make this good and then we built our own like test methodology of like this is what a good product description looks like this

is what makes it that way and then ran it through a few thousand times to see whether it was going to do it or not and we felt good about it and then we moved into a phase called build where we present this is what we want to build

this is what we've learned along the way this is why we think this is the right approach and let's go ahead and build it and we did and then a matter and then the Final Phase is released and in the release phase it's like this is the

working version this is how it works across some multiple sets of shops and here are the caveats here are the gotas here's what we think is right and this is what we want to release too and our initial release was just the english-

speaking merchin and we thought that would get us to Market fastest and it did and we followed with multiple languages shortly thereafter I heard that sometimes you guys don't even have a meeting for some of these steps you

just like maybe you make a short video or something I this project was a very fast Cadence so what we tried to do was we had a Channel with all of our

stakeholders in slack and a lot of the conversation goes back and forth of here's the work here's what's happening and we let them be in on the public channel so like they could just watch the work as it goes and if they want to

interrupt they can interrupt and ask questions and then every couple days we would just make a demo video and just post it in the channel and just ask for feedback and a lot of like uh rep

especially Glenn my boss he makes a friction log and his friction log is like walking through the video or through the product and giving feedback on what he thinks is good or bad and we usually like turn it around in a couple

days but yeah it's a very like Fast cycle I don't think there was many if any synchronous reviews on this it was all async as we went oh okay and when were Merchants involved in this process like did you get feedback during the

Prototype stage or yeah in the Prototype phas we took mockups to them we had a couple conversations with them we said what what do you expect when this type of thing happens is this we had taken a couple of their products previously and

generated description and said what do you think about this like does this make sense gave them a few samples of it and we got really positive examples and we even asked the open-ended question of like what do you know about AI like what

do you think about it what are your considerations for it and so we had a good idea of the messaging that we wanted to go forward with we had a good idea of like what good would look like and we had good feedback for even future

things that we weren't thinking about from our Merchant said like hey can you also solve this problem can you solve this problem our Merchant started to enumerate a bunch of different ideas and so it it was a really good collaborative

process that's awesome so after this initial success with the descriptions feature like you should Shopify magic which is like a whole Suite of products right so like how did you go beyond this initial feature to that whole thing like

how do you prioritize the jobs to be done you know that kind of stuff yeah for us it's the culture of Shopify is like everyone is aligned around that mission of do whatever it takes to make

the merch successful and when a feature like this drops every kind of team comes forward and says hey are there other things that we can do for L resist technology as well because we see a lot

of different places where it could make that and for us we saw generative AI as this is a good place to be able to augment what the user already does how

do I solve for cold start how do I deal with some of the more repetitive tasks that are going on how do I save the merchant time so we started to look at where those places were and where the

best bit for this technology could be and what we found was there's a lot of effort on the marketing side of things and getting users in is a challenge and

then the merchandising part of it is helping them convert because they understand the product and can close at that point so a lot of the efforts that we made were around the marketing the

SEO Ty of things and all of the adjacent pieces there of like okay how do I get a merchant in or how do I get a buyer in or how do I make sure that they can walk through the funnel in a way that's going

to be meaningful for them so such that they'll convert so we went with email as a good place for subject line generation like if we can generate you a better subject line then maybe you'll get

better open rate or maybe you'll be more likely to send the email at all and then maybe we can help you write the body of the email and then maybe we can help you with some of the answers when the merchant is on the page of your online

storefront and they have questions maybe you don't know what questions they're going to ask but based on your conversations we can tell you and we can help you answer those questions as well in a way that's going to be impactful then how do we help you customize your

storefront so that whenever you have to write about a product or whenever you have to write about yourself you can get those things going and get started and a part of it is for a new Merchant some of

these things can be a challenge and Simply Having a button to push to give you an idea is all it takes to get you started as an entrepreneur and because people come with high intent of wanting

to do this and as much of the friction as we can remove along the way the better off entrepreneurship is going to be and the better off our Merchants are going to be got it so kind of thought through the merchant journey and like the biggest problems like around growth

and like yeah kind of PRI is that way right yeah got it what about um like did you have like a okr for this thing or are you guys not use olds what kind of goes did you said so we did a general like heart framework that we walked

through of this is what we want to be able to measure this is what we want to be able to do the one thing that we principled ourselves against was trying to be too growth hackery about this like

we didn't want people to have to use AI for the sake of using AI we wanted it to be part of of the process of what they were doing and it' be available so we don't have a you need to have we need to

have this much adoption or this much usage of this a capability or feature part of it was if we introduce this feature it cannot do any harm and when it works it has to at least be neutral

or positive in terms of its effect that's kind of the the initial goals that we set for ourselves and the rest of it was a matter of like how do we this in into a place where the users can

engage how do we stay strong to our principles and how do we make this an augmentation feature not a replacement feature okay so when you say neutral positive is that like from a experiment or just like for the merchant experience

is that theion experience and the experiment result when we're running AB test on it got okay all right so let's talk about another Fe feature like that

g news shy psychic right so can you just maybe start by describing what it can do so psychic can answer important questions and help you understand

Shopify you can make analytical query can navigate to the right places so if you don't know where something is how do I set up my Canadian tax settings where do I go for that how do I set up

inventory numbers or where do I process the return they can help you create different object so if you need to create a product if you need to create a discount if you need to create any kind

of draft order all of those things are possible with Sid and it also let you edit themes and dictate to the what the storefront needs to look like and what

changes you want to make from a semantic understanding perspective and that's a starting place but we we wanted to do everything how is like where are some challenges or learnings from building

psychic like how is it different from just building a regular customer product so when you build normal consumer products in my past experience and a lot of my stuff has been in data so it close

a little bit similarly but even in like a general product development cycle you have goal you have a path you have some requirements you have an estimate and then you kind of work throughout that

and then you work on the edge cases in case there are any to be able to to narrow in on but there is a deterministic path towards doing it in a non-determinate system it's very hard to predict what the error cases are going

to be and it's very hard to plan for every Edge case along the way to try to defend against it so the first thing that we had to do was assume that

everything that we possibly thought would go wrong will go wrong so we started from the error cases first and in this situation is like it's going to make mistakes and how do we plan for

those mistakes so if sidekick doesn't know how to do a thing which ultimately someone will ask it something that it doesn't know how to do how do we plan for that scenario there are going to be cases where the llm wants the answer

where it's not necessarily the correct answer what's the thing that we want to teach it not ex easier to be able to so all those things have to be built up front in addition to it you also have to

work with when the system goes down when there isn't enough GPU like these are normal error situations but when when an llm makes them they can be much more magnified especially when it becomes

part of a regular flow so there has to be ways to maintain the personality of sidekick while still making sure that it's still helpful and then you work around all the safety measures that you

need to build around the AI and so for us it was we don't want the AI to do something without the merchant knowing that thing is happening and some of that

has to happen from you can't have the AI right to the server cuz if that happens then some type of transaction took place and was written to death that the user

wouldn't know about so we couldn't do that so we switched it and said how do we push everything into the client State how do we push everything in front of the user in a way that they can review

they can look over and they maintain full control so that this is augmentation not Automation and that that's kind of where we started with a lot of our our thinking on this and what

made that unique is the way that we build is not through like one use case at a time in some cases it's like a data set at a time of like this is what we need to be able to like understand this

is the problem space that we need to understand and there are different ways to approach it you can find tune you can do something from retrieval augmented generation and then you can do something just on General prompting but each

approach has pros and cons through it so like being able to balance and evaluate on an ongoing basis while knowing that your evaluation is never going to be comprehensive and there's going to be

errors and so having that full Corpus of unknown means that we build from our principles and stick to our principles as we build versus having something linear that we know will work yeah I've

been working on AI product too and that's the struggle the entire product is like an edge case never was going to help happen yeah exactly yeah exactly can you talk about the different layers of a prodct psyching like you know the

data the model like can you brief describe it for read readers sure so let's go from the bottom to the top and we'll kind of work through that so at

the very bottom is the model itself then on top of that is the adapters that you might want to build on top of it so any kind of fine tuning would kind of apply an additional adapter then on top of

that is the API through which you will access and any kind of additional changes that you want to do in between so you can access the model directly but you can also build an API in between that handles a few different scenarios

for you some of that could be history management some of that could be parser some of that could be grammar enforcement some of that could be streaming some of that could be airor handling so some of that is an API layer that we've kind of built in additionally

and then in addition to that then you go above and there is the actual UI ey layer to it and in the UI layer some of it is the actual components by which it

will show up and then the components by which it will interact the components by which it will give optionality the components by which it will collect feedback and the components by which it

will allow for error management along the way in a safe way then there is the full integration with the rest of the admin so it's one thing for it to like interact in its own little box but it's a whole other to interact with the

associated system so it's what is the API between it and the system and again because we're not doing things at the server side and we're doing things on a client State we want to make sure that

the there is an API clean between sidekick and the admin that's going to be consistent and those are some of the main key part the data part of it is constant I think that the data part of

it is either in the form of embeddings or in the form of in the form of data for fine tuning and instruction but all of it is something that kind of comes from the entire machine and then the pipelines that build all of that as

initial parts of it got it so there's a lot of companies building these like chat Bots right and I think the way to why want to differentiate from like you know chat GPT is like it has more personal data about your business or

about what you want so like for psychic what kind of Shopify data is a train on and like how do you avoid privacy issues so we don't train sidekick on data that is personally identifiable some of it is

like our own created instructions of how we want to use the admin help documentation that we kind of collected and cured over time and some of it kind of comes in the form of rag which is

retrieval augmented generation which is at the time of the query rather than necessarily training on it so we try to differentiate accordingly so we try to keep the merchants context within the merchants context and then we keep the

general understanding of Shopify as things that we can potentially find to the model on got it okay so so so like when when the user submit a query maybe you add a bunch of information

additional information right like yeah we'll add like we call into the prompt into the context window specific things about what the merchant is doing things about the related products or whatever

it might be but it's all in the context of the session and never something that's like trained against the model that makes sense okay so let's talk about the prompt so you know there's all this like breakr about prompt engineering and stuff and there's also

just like eval process right that that like can you like just talk a little bit more about that for people to know what they are so the the prop side of it is obviously um something that kind of continually develops and I think there's

a lot of different methodologies and ways to do it I think what we're doing is that we start from the evaluation scenarios and work backwards and what we do there is like we'll set off a number

of integration tests of when this question gets asked this the answer we expect and do a few hundred of those for each different scenario and then you try to run it in an automated fashion and in

the automated fashion there are ways to evaluate and assertions just as you would any kind of unit test and some of those assertions are equivalency ones where it has to say something very specific because it has to match a

specific type or in other cases they're semantically correct and in the semantically correct version you want to do many examples of those because you're likely using an llm on top of that to do the evaluation between semantic

correctness between two responses and so that's an additional part of the evaluation side of it but what we do is we start with the integration test so I think for us it was was every new

scenario we built a few hundred of them and then evaluated against that but again because we knew that there were going to be cases where there were error we already had that built in from the start well the the initial first few

hundred cases do you do that manually or or like like entirely okay so you had like a you had like an eval team or did you do it yourself some of it we did ourselves some of it was like we have we

don't have a eval team so to speak like we have people that were subject matter experts in the area coming forward with here are the 100 different cases or 200 300 700 different cases that are

supposed to be passing and we'll hand write them like that has to be human done whether it has to be done by subject matter experts otherwise you won't really hit the mark of what the thing should be there are ways to be

able to like synthetically generate more of them from the initial tests from the initial hand curated one but what we wanted to do was create the scenario in which like we were handpicking the

things that were most important and that psychic had to do properly before we were okay with it yeah I went through something similar you know there just's a way to get the AI to evaluate ai's

answers right but like wait for the AI to generate more like potential and score it yeah but I think we went through something similar where we we went through the synthetic process and then we got a human to look at it and

and then turn out like a lot of EV was wrong you got to do the human part first yeah the other part that's hard about this is a lot of the things that we're used to in the world of unit test at

least is it's like single turn so it input output and I like black box for the the processing but as long as the input and the output are correct and I'm

okay but in the case of a chat it's multi-turn so you don't have a test that's just on one thing it's like it could go down a different path along the way of like let's say you ask the question of like what kind of discounts

should I set up so in that first scenario it should be like well these are the different types of discounts that are available and then the next question that the user is going to say is like okay well I forget that I want to ask about analytics like what's my

store actually doing before I go ahead and do this and it's not even on the same subject anymore but now we have to switch and it's like well your store did this much in the last year okay cool can you make me a BOGO discount I like okay

and so here's how you do it so you have to plan for these type of scenarios too in the manual curation and so that multi-turn is just as important as a single turn because this is not a bot

that is going to have one subject matter as it expertise has everything do you think it's worthwhile to try to train the user to submit better prompts because like a lot of times these like you know very short prompts aren't as

good as like you know few shots and like a bunch of like paragraphs of stuff right yeah I mean I think that people will learn how to do prompting over time and I think that like this is going to become one of those things like the web

where everyone didn't know it to begin with and the more you kind of learned the more you learned how to search in Google the better your results would be but at the end of the day you're like the job of the user is to be an

entrepreneur and not to have to worry about how the technology works or how to make the most of the technology our job is to do that and so we don't view teaching our Merchants how to use this

type of Technology as the the key for us for us it's about making it feel like magic and it that has to be simple that has to be like bring what you have and what you want and let's see how much we

can get you there got it how about when it comes to the underlying model right like do you guys have a lot I mean maybe this is private but do you guys have a lot of debate between GP 3 and four and like maybe some other model there's like

different costs and different capabilities so part of it is like that's kind of an always on question and I think it's about like running it through the evaluation and figuring out

what the best case scenario for the user is for us it's always the cost scenario is secondary the primary question is like what's going to solve the problem in the best way and how do we make sure

that our Merchant are successful and then secondary cost the secondary question is like how do we make sure that this isn't going to be something that is cost prib did it but if can make it work then let's make it work got it

and I guess one of the flaws of this thing is that you know it tends sometimes it tends to like make up right so like what is like acceptable hallucination and what is not acceptable hallucination for the stuff

yeah so acceptable Hallucination is like moving people along the path that's on the correct subject and unacceptable Hallucination is like issuing commands or things that are going to be harmful

and what we're trying to do is like add layer along the way to validate and verify that the that what gets generated is useful But ultimately we're accepting the fact that it's going to hallucinate

that's why every scenario is one in which the merchant will have full control of what goes out there and got it like that because that's like principle one we don't feel like a

hallucination will be harmful because the merchant will be the ultimate decider on whether this is good enough for them or not yeah there was a lot of like autog GPD and this kind of stuff before but I don't think this thing is good enough to just like run by itself I

either a human person to evaluate like to review and the thing is that even if it gets good enough like that's a decision the merchant gets to make like well for us it's like they're on Shopify because they get to represent who they

are and their brand and they talk to their customers and even with an AI and plate like it should still be them yeah exactly do you have any thoughts on like it costs money per token to use this

thing right to use the model so like and and I think you put material out there saying like want to rsy to all Merchants for free like how do you prove the business model of the thing it could just burn money my god do everything

possible to make our Merchant successful and then make money doing it so we can do more of it but we never want to reverse the order so it's always about like solving the problem first and if we believe this is one of those things that

can be transformative to our Merchants as we do then it's important that we invest in it it's important that we give it give them all the the power that we possibly can got it yeah because if the merchant makes more money than Shopify

helps the benefits too yeah so let's put it all together like you know what would like be a really AI power Merchant experience let say the merchant like I'm matching my store during holiday shopping season or something like and all all this AI stuff is already shipped

so what will my experience look like as a mer so I think that there's I think sidekick will help you a lot even before the the holiday sale begins I think it's about like that planning of that holiday

sale and being able to have an assistant with you along the ways of making decisions I think by the time the sale happens ideally what would have happened is the products that you wanted to create were created in a way that were

going to be comp in and we're going to be friendly for SEO the discounts or any kind of promotions that you wanted to create had all of the appropriate choices being made in terms of when and

why to to choose the discount and then you knew everything about your business that was going to give put you in the best position possible to be successful and after the sale and promotion happened I think you can ask for like

how did I do and to get be able to get that answer and to be able to walk through like what do we learn how do we do different what could we do differently I think a lot of that is in that hands of the merchant and one of the things that we realize is that

especially early on when you're by yourself entrepreneurship is hard and is lonely and having someone to be able to bounce ideas off of having just some stress testing on it and even being able

to get help on like how to do the best possible thing that's what's critical here and that's why we build sidekick and that I think is the game changer for people is before the sale even happen so that the sale can be successful makes

sense yeah I don't sell physical merchandise but like you know as an online Creator like I I just want to like create content right I don't want to do with like you know running ads managing my business like all this other

stuff so so there's there's like AI that can help me with all this sort of stuff that'll be ideal yeah yeah yeah so like before we move on to last section like I'm just curious you mentioned a few principles like one principle is like

the merchant is always in in charge do you mind just like sharing like all the main principles for our people building AI Pro I think you so I think the idea

we principles is that you put few of but the ones that you put are true principle no and like when I say principle I mean like this is the non-breaking one the one thing that like when you are forced

to make a decision you go back to these principles and this keeps you kind of grounded so for us it's always that first principle is key and that one of the big one I think the other ones that

we think about are around the experience and like how we want Merchants to interact but it all comes from that first one so they're all like the choice is in the Merchant's hand so it's not we when we think about like whether we

generate options or not what's going to be better for the merchant and what's better for them is to be able to see multiple options and to be able to choose and the I guess that's the main one that we've kind of built for the AI

side of thing others come for like sub products and things and ways that we're considering that the other one that we built in from a ux perspective is that the merchant needs to know that the

thing is working and so we've opted for streaming experiences at all times wherever possible we're showing that this work is is actually happening cuz it needs to feel like it's live and they're not waiting for it to think so a

lot of it is about like making that experience feel like magic so we have a ton of ux principles around that but fundamental to the whole thing is AI will not do harm yeah I think principles

are often like very underrated for prodct development like I I feel like a really good set of principles actually has a much bigger impact on a product than like some big strategy document or whatever you know because it actually

affects the decisions that you make yeah yeah and I think it makes it EAS easier to make decisions because you have a thing that you're lined up again so it's a different methodology but it's the thing that we probably spend the most

time on is making sure these principles are sound and then stress testing them on all directions awesome so this is wrap up like two questions right so like I think Toby came out and said he wants to make Shopify a crafter's paradise can

you give me an example of that from your product or like do you just have like a no meetings on your counter or like what give me give me an example of yeah craft I can I can give you multiple examples off of the team and I think it's not

it's not just about the meetings although it's a big part of it I I think it's about like where the decision making comes from and a lot of that is about being able to do things in the

open being able to bring forward the best ideas and you can see it from the way that he even built like product descriptions as a product we went through and it was the individual

Crafters that were bringing forward what the best solution was going to be it was the individual Crafters that made the majority of the decisions management is there to to course correct advise bring

forward anything that people might be missing but the CFS that are largely tating what the flow is and the idea of like working against a goal and setting your own timeline that comes from the

Crafters themselves like this is the thing that we want to do this is how we want to build and this is how we're going to make the biggest impact and I think empowering the team with principles and empowering the team with

a clear user that you're trying to hit and a clear mission that is resonates with everybody it makes it a lot easier as a crafter to get the noise out of the way and just focus on what's important

and if you get those things done early you don't have to waste time on the small things because you know what the big things are is there like is there pressure in you shy internally where like you know the Crafters are like oh you know like I got to get promoting

into management to to make it or like you know I got to get more Vis sure happy abely not I think there are two tracks there are entirely two tracks and they go in whatever Direction you want to and they go as high as you want to so

we have crafter tracks and have manager tribes and there are people that want to manage and there people that need to manage but for the most part as a crafter there are people that are equivalent to directors the VPS and Beyond and you can go as high as you

need to go that's awesome you guys should write more block post about that so more companies can copy those two2 track approach yeah last question like how are you guys using AI internally for employees at shy like is there some

internal tools or everywhere is the better answer so I think that like we've built our own kind of internal version of chaty BT which just goes through the

AI we have have a search spot for our internal Wiki we have slackbot that are AI driven we have ai tools from like image generation for people to play with

to access to just whatever API is out there for people to play with it highly encouraged we've had like hack days with streams of like AI work but like it's

everywhere the idea is for everyone to be able to use it to solve personal problem company problem but the idea is to get to use and to become more familiar with it so it's not like just it's not like there's some internal team

only work on internal tools it's like anybody can just build everybody does build that's great man yeah all right thank thanks so much big yeah I think that's lot the questions I had we got to go into a little bit more depth