AI+Data in the Enterprise: Lessons from Mosaic to Databricks

Enterprise customers are really evidence that you're going to be able to scale your business that you have some traction with companies that are going to be around for a while that have big budgets that when they invest in a

technology invest for the long run but on the flip side the best customers are often other startups because there's no year-long procurement process they're willing to Dive Right In and you can get

a lot more feedback much [Music] faster welcome to founded and funded I'm John turo partner at Madrona and today I have the privilege of Hosting Jonathan

Frankle Chief AI scientist at data bricks which he joined as part of that company's $1.3 billion acquisition of Mosaic ml a company that he

co-founded Jonathan is a central operator at the intersection of data and AI he leads the AI research team at data bricks where they deploy their work as commercial product and also publish

research open- Source repositories and open source models like dbrx and MPT on this episode we are diving into the evolving landscape of infrastructure for

data and Ai and how that democratizes access to these critical Technologies Jonathan shares his Insight on the initial Vision behind Mosaic ml the transition to data bricks and how

production ready AI is reshaping the industry we'll explore how Enterprises are moving beyond prototypes to large scale deployments the shifting skill

sets AI Founders need to to succeed and Jonathan's take on exciting developments like test time compute whether you're a Founder Builder or curious technologist

this episode is packed with actionable advice on thriving in the fast changing AI ecosystem Jonathan it's an honor to have you here welcome to the show thank you

so much for having me I can't wait to take our private conversations and show them to everybody we always learn so much from those conversations and and so let's dive in you know you've been

supporting Builders with AI infrastructure for Years first at mosaic and now as part of data breaks and I'd like to go back to the beginning let's

start there what was the core thesis of Mosaic ML and how did you serve customers then so the core thesis quite simply was making machine learning

efficient for everyone the idea that this is not a technology that should be defined by a small number of people that should be built to be one siiz fitall in general but that should be customized

for everybody by everybody for their own needs based on their data in the same way that you know we don't need to rely on a handful of companies if we want to build an app or write code we just go

and do it everybody has a website everybody can Define how they want to present themselves and what they want to do with that technology and we really firmly believed in the same thing for

machine learning and AI especially as things started to get exciting in deep learning and then you know of course llms became a big thing about you know I guess halfway through our Mosaic journey

and I think that mission matters even more today to be honest um we're in a world where we bounce back and forth between you know huge fear over the fact that only a very small number of

companies can participate in building these models and huge excitement whenever a new open source model comes out that can be customized really easily and all the incredible things people can

do with it and you know I firmly believe that this is technology that should be in everyone's hands to kind of Define as they like for for the purposes they see

fit on their data in their own way you know it's a really good point and you and I have spoken publicly and privately about the democratizing effect of all this

infrastructure and I would observe that the aperture of functionality that Mosaic offered which was especially about hyper efficient training of really

large models putting it in the hands of lots more companies that aperture is now wider now that you're at data bricks you

can democratize more pieces of the AI life cycle can you talk about how the how the mission is kind of expanded yeah I mean it it was really interesting mate

our CTO I was looking at his notes for a meeting that we had for a research team last week and he had just written in his notes kind of casually you know our mission has always been to I think it

was to democratize data and AI for everyone I was like wait a minute that sounds very familiar and you know I think we may chat at some point about kind of this acquisition and why you

know we chose to work together it's the same Mission we're on the same Journey um data Brick's obviously much further along than Mosaic was and wildly

successful but it's great to be along for the ride and so I think the aperture is widened for two reasons one is simply that you don't need to pre-train anymore there are awesome open source based

models that you can build off of and customize so even pre-training was the kind of thing that wasn't quite for everyone but that's not necessary anymore you can just get straight to the

fun part and customize these models through prompting or through rag or you know through fine tuning or through rhf these days and the aperture is also

widened to the fact that now we're at the world's best company for data and data analytics and the world's best data platform and and what is AI without data

and what is data without AI so we can now start to think much more broadly about a company's entire process from start to finish with a problem they're trying to solve what data do they have what is unique about that data and

unique about their company and then from there how can AI help them or how can they use AI to solve problems and back again this is a concept we call it data intelligence the idea that it's really

meant to be in contrast to general intelligence general intelligence is the idea that there's going to be one model model or one system that will generally be able to solve every problem or kind of make significant progress in every

problem with minimal customization and at data bricks we kind of espouse the idea of data intelligence that every company has unique data has unique processes unique view on the

world that is captured within their data and within how they work and you know within their people and AI should be shaped around that the AI should represent the identity of the business

and the identity of that business is captured in their and you know there's no obviously these are this is very poic to say data intelligence versus general intelligence the answer will be somewhere in between

but to me it's it honestly every day at work feels like I'm doing the same thing I've been doing since the day mosaic started just now at a much bigger place with a much bigger you know ability to

make an impact in the world there's something very special about the advantage that you have that you're seeing this

grade of customers who have been on a journey from prototype to production you know for years now and the most sophisticated among them are now in

production and so for that I I have two questions for you number one what do you think it was that has finally unblocked and made that possible and number two

what are those customers learning who are at the Leading Edge what are they finding out that the rest of the customers are about to discover so I'm going to I guess reveal how much less

I'm a scientist these days and how much more I become a business person I'm going to use the hype cycle as the way to describe this um and it it breaks my heart and makes me sound like an MBA to

do this but among Enterprises there are always you know the bleeding edge early adopter Tech first companies they're the companies that catch on pretty quickly and the companies that are more careful

and conservative and what I'm seeing is those companies are all in different places in the hype cycle right now for the companies that are like really early adopters in Tech forward the peak of

inflated expectations they hit that like two years ago around the time chat gbt first came out they hit the tro of disillusionment last year when it was really hard to get these systems to work

reliably and they are now getting productive and getting things shipped in production and they've learned a lot of things along the way I think they've learned they've learned to set their expectations properly to be honest and

which problems sense and don't make sense this technology is not perfect by any stretch and we're still I think the more important part is we're still learning how to harness it and how to use it in the same way that you know

having Punch Cards back in the 1950s or 60s is you know still turning complete and still a little bit slower but just as capable as our Computing systems today from a theoretical perspective but

50 years of software engineering later and it's much easier to build an architect A system that will be reliable and build it in a much other way and all these principles we've learned and I

think those companies are furthest Along on that Journey but it's going to be a very long journey to come and we know how big of a system we can build at this point without it Keeling over and where the AI is going to be unreliable and

where we need to kick up to a human which tasks make sense which tasks don't make sense a lot of them I've seen have kind of whittel it down into very bite-sized tasks the way that I typically frame it for people is you

know you should use AI either in cases where it's open-ended and there's no right answer or where a task is hard to perform but simple to check and you can have a human check I think GitHub

co-pilot is a great example of this where you could imagine a situation where you ask AI to just write a ton of code and now a human has to check all that code and understand it and honestly it may be just as difficult as writing

the code from the beginning or pretty close to it or you can have the AI suggest very small amounts of code that a human can almost mechanically accept or reject and you're getting huge

productivity improvements but this is a scenario where the AI is doing something that is somewhat more laborious for the human but the can check it very easily and I think finding those sorts of sweet spots is where kind

of the companies who have just been at this the longest they've also been willing to take the risk and invest in the technology they've been willing to try things they've been willing to fail to be honest they're willing to just

take that risk and be okay if the technology doesn't work the first or second time and keep whatever team they have doing this going and trying it again and then you have companies that are kind of I think a bunch of companies are in the tro of disillusionment right

now companies that are kind of a little less on the bleeding edge and then a bunch of companies are still at that peak of inflated expectations where they think that AI will just solve every problem for them and those companies are going to be very disappointed in a year

and very productive in two years you know naturally a lot of Founders who are going to be listening are asking how do they get in these conversations how do

they identify the customers that are about to exit the trough and and how do they focus for them what would you what would you say to those Founders I have

two contradictory lessons from my time at mosaic the first is that VCS love Enterprise customers because Enterprise customers are really evidence at least

if you're doing B2B that you're going to be able to scale your business that you have some traction with companies that are going to be around for a while that have big budgets that when they invest in a technology invest for the long run

but on the flip side the best customers are often startups because there's no year-long procurement process they're willing to Dive Right In they understand where you're coming from and understand the level of service you'll be able to

provide because they're used to it and you can get a lot more feedback much faster but that has taken as less valuable validation even when I'm evaluating companies Enterprise

customers are worth more to me but startup customers are more useful for building the product and moving quickly and so the answer is strive for Enterprise customers don't block on

Enterprise customers I think that's fair and I think optimizing for learning is is really smart but there's another thread that I would pull on and this is something that I think you and I have

both seen in the businesses that we've built which is the storytelling and I won't even say GTM the storytelling around our product

can be segmented even if the product is horizontal as so many infrastructure products are Mosaic was a horizontal product data bricks is a horizontal

family of products but there are stories that we tell that explain why data bricks and Mosaic are really useful in

financial services really useful in healthcare and there's going to be a mini adoption flywheel not so many in each of these

segments where you do want to find first the fast customers and then the big customers as you dial that story in and there may be product implications but there may be not I think that's a great

point and there are stories I think along multiple axes these days kind of in a social media world and in a world where just everybody's paying attention to AI there are horizontal stories you can tell that will get everyone's

attention and I think I mean one of the big lessons I took away from Mosaic was talk frequently about the work you're doing and have some big moments where you where you really buckle down and you

do something big don't disappear while you're doing it but you know releasing the m PT models for us which sound so quaint only a year and a half later and it really was only a year and a half ago

that we trained a 7 billion parameter model on one trillion tokens and it was the first kind of open- source commercially viable replication of the

Llama One models which sounds hilarious now that we have a 680 billion parameter mixture of expert model that just came out and the most recent meta model was a

45 billion parameter model trained on 15 trillion tokens it sounds quaint but that moment was completely gamechanging for Mosaic and it got the attention up and down the

stack and you know across all verticals across all sizes of companies and led to a ton of business and further moments like dbrx more recently kind of same

experience and so storytelling through these important moments especially in an area where people are paying close attention actually does kind of resonate universally but at the same time I totally hear you

on the fact that you know for each vertical for each size of company there is a different story to tell and I think my biggest lesson learned there is getting that first customer in any industry or in any company size or anything like that is incredibly hard

somebody has to really take a risk on you before you have much evidence that you're going to be successful in their domain having that one story you can tell leads to a ton more stories once you work with One Bank a bunch of other

banks will be willing to talk to you but getting that first bank to sign a deal with you and actually do something even for the phenomenal go to market team we had at mosaic was a real battle

they had to really fight and convince someone that they should even give us a shot that it was worth a conversation can you take me back to an early win at

mosaic where you didn't have a lot of credentials to fall back on yeah it was a collaboration we did with a company called repet before we had even released

the MPT models we were just chatting with repet about the idea that we could train an llm together that we'd be able to support their needs there and they trained MPT before we trained MPT they

were willing to take a risk in our infrastructure and we delayed MPT because we only had a small number of gpus and we let replit take the first go of it I basically didn't sleep that week because I was monitoring the cluster

constantly we didn't know whether the Run was going to converge we didn't know what was going to happen it was all still internal code at that point in time but repet was willing to take a

risk on us and it paid off in a huge way it gave us our first real customer that had trained in llm with us and been successful and deployed in production and that led to probably a dozen other

folks signing on right then and there and the MPT model actually came out after that how did you put yourself in a position for that lucky thing to happen we wrote a lot of blogs we shared what

we were working on we worked in the open source we talked about our science and we built a reputation as the people who really cared about efficiency and cost and the people who might actually be able to do this we talked very

frequently um about what we were up to and that was kind of a lesson we had learned early on where I don't think we talked frequently enough but we wrote lots of blogs when we were working on a project we would write part one of the blog as soon as we hit a milestone we wouldn't wait for the project to be done

and then do part two and part three and those MPT models were actually I think part four of like a nine-month Blog series on training llms from scratch and that got rep's attention

much earlier and started the conversation maybe one way of looking at if you want to be cynical is you know selling ahead of what your product is but I actually you know I look at it the other way which is to show people what you're doing and convince them that they

can believe you're going to take that next step and they want to be there right at the beginning when you first take that next step because they want to be on the bleeding edge you know I think that's what got the conversation started with repet and kind of put us in that

position but we were going to events all the time just talking to people trying to find anyone who might be interested in Enterprise that had a team that was thinking about this and you know there were a bunch of folks we were chatting

with but we had already started Contracting deals with folks but repet was able to basically move right then and there they were a startup they could just say we're going to do this and write the check and do it so being loud

about what it is that you stood for and what it is that you believed and being good at it like I think we worked really hard to be good at one thing and that was training efficiently you know you

can't fake it till you make it on that like we did the work and it was hard and we struggled a lot but we kept pushing at the strong encouragement of N and handland our co-founders they kicked my

butt to keep pushing even when it was really hard and really scary and we were burning a lot of money but we got really good at it um and I think people recognize that and you know it led to customers it led to the data bricks

acquisition and I'm now seeing this among other small startups that I'm talking to in the context of collaboration in the context of acquisition anything like that the startups I'm talking to are the ones

that are really good at something it's clear they really good at something it's been clear through their work I can check their work they've done their homework and they show their work you know those are the folks that are

getting the closest look because they're genuinely just really good at it and you you believe in them and you know the story they're telling is legitimate there's one more point on this which I

think compliments and extends what you just said that you folks believed in something and this is not about a story and it's not about results either that

you believe training could be and should be made more efficient and a lot of the work you were doing anticipated things like chinchilla that

Quantified how it could be done later oh oh we didn't anticipate we followed in the footsteps of chinchilla chinchilla was like early Visionary work I and I can say this you know Eric Elson who

worked on chinchilla is now one of my colleagues on the dat's research team but I mean there are a few moments if I really want to look for the pioneers of just truly Visionary work that was quite

early and when I look back is just kind of like tent pole work for LMS now chinchilla is one of those things the other is like a Luther AI putting together the pile data set um which was

done in like late 2020 like two years before anyone was really thinking about llms they put together what was still the best llm training data set into 2022 but we did genuinely believe in it I

think to your point like we we believed in it and we believed in science we believed that it was possible to do this and through really really rigorous

research we were very principled and had our scientific Frameworks that we've believed in or our way of working we had a philosophy on how to do science and how to make progress on these problems opening ey believes in scale and you

know now everybody believes in scale we just believed in rigor that doing our homework and measuring carefully would allow us to make consistent methodical progress and that remains true and

Remains the way we work it's sometimes not always the fastest way of working but at the end of the day it leads to consistent progress so here we are in

2025 and amazing Innovation is happening and there's even more opportunity than there than there has been seems to me even more excitement even more excited

people how do you think the profile of and the mix of skills in a new team should be the same and should be different as to when you formed Mosaic

so it depends on what you're trying to do we hire phenomenal researchers or rigorous scientists who care about this problem and are aligned with our goals

who share our values who are Relentless and honestly who are just great to work with I think culture cannot be understated and like conviction is the most important quality

if you don't believe that it is possible to solve the scientific problem you will lose your motivation and creativity to actually solve it because you're going to fail a lot and the first failure

you're going to give up but beyond that I think this is data science in its truest form like I never really understood what it meant to be a data scientist but this feels like data science you have to pose hypotheses

about which combinations of approaches will allow you to solve a problem and about measuring carefully and developing good benchmarks to understand whether you've solved that problem I don't think that's a skill that's confined to people

with phds far from it so the fact that data bricks was founded by a PhD super team now means that more than 10,000 Enterprises don't need a PhD super team

when it comes to their data and I look at you know our Mosaic story through to our data Brick story now in the same way we built a training platform and a bunch of Technologies around that and now

we're building a wide variety of products to make it possible for anyone to build great AI systems in the same way that when you get a computer and you want to build a company you don't have to write an operating system you don't

have to build the cloud you don't have to invent the virtual machine I mean abstraction is the most important Concept in computer science and data rcks has had a PhD super team to build

that low-level infrastructure that required it to build spark and every Delta and unity catalog and everything on top of that and now it's the same thing for AI the future of AI isn't in

the hands of people like me it's in the hands of people who have problems and can imagine a solution to those problems in the same way that I'm sure Tim burner's Lee who you know pioneered the

web did not exactly imagine I don't know Tik Tok that was not what he had in mind when he was building the worldwide web the kinds of startups I'm most thrilled about engaging with today are companies

that are using AI to make it easier to get more out of your health insurance making it easier for you to solve your everyday problems making it easier for for you to just get a doctor's

appointment or for a doctor to help you for us to Spot Medical challenges earlier that's the of people who are empowered because they don't have to go

and build an llm from scratch to do all that that layer has now been created so the future is in the hands of people who have problems and care about something for a PhD super team these days there's

still tons and tons of work to do in making AI reliable and usable building the tools that these folks need building a way for anyone to build an evaluation set in an afternoon so that they can measure their system really quickly and

get back to work on their problem there's a ton of really hard comp complex fuzzy like machine learning to do but I think the interesting part is in the of the people with problems how

was your role changing as you adopt these kinds of AI Technologies inside data bricks and you try to be I'm sure a sophisticated as you can be about it I

I'm still a scientist um but I haven't necessarily you know written a ton of code lately but I spend a lot of time these days connecting the dots between

research and product and research and customer and research and business and then come back to the research team and say I think we really need to do this how can RL help us do that and then go to the research team and say you've got

this great idea about this cool new thing we can do with RL let me go back to the product and try to blow their mind with this thing that they didn't even think about because they didn't know it was possible show up with something brand new and convince them we

should just build a product for that because we can and because we think people will need it and so in many ways I'm a bit of PM these days but I'm also a bit of a salesperson these days you know but I'm also a manager and I'm

trying to continue to grow the incredible skills of This research team both the people who have been with me for four years and the people who have just arrived out of their phds and make them into the next generation of successful data bricks talent that stays

here for a while and then maybe goes on to found more companies like a lot of my former colleagues at mosaic have so it's kind of a little bit of everything but I have had to make this choice about whether I'm just going to be really

really deep as a scientist write code all day get really really good at getting the gpus to do my bidding or get good at leadership and running a team and inspiring people and getting them

excited and growing them or get good at thinking about product and customers and what combination I wanted to have there um and that combination does naturally led me away from being the world's expert on one specific scientific topic

but towards something I think is you know more important for our customers which is understanding how to use science to actually solve problems there's an imaginative leap

that you have to make from the technology to the Persona of your customer and the empath the empathy

with that that I imagine involves being in a lot of customer conversations and it's but it's an inversion of your thinking it's it's not here's a hard problem that we've solved what can we do

with it it's a you know keeping an index of important problems in your head and spotting possible solutions to that maybe I actually think it's the same skill as any good research recher no

good researcher should just be saying I did a cool thing let me find a reason that I should have done it sometimes very occasionally this leads to Big scientific breakthroughs but for the

most part I think a good productive everyday researcher should be taking a problem and saying how can I make a dent in this or finding what the right questions are to ask and just asking

them and coming up with a very basic solution all of these sound like just product scenarios to me whether you're building an MVP like figuring out a question that hasn't been asked before that you think is important to be asking and building an MVP and then trying to

figure out whether there's product Market fit or the other way around finding a problem and then trying to build a solution to it I don't think much research should really involve just

saying I did this thing because I could that is very high risk and it's hard to make a career out of doing that all the time because you're generally not going to come up with anything I'm going out and trying to figure out what the

important questions are to be asking both asking new questions and then checking with my pm to see if that was the right question to ask and talking to my customers it's just now instead of my audience being the research community

and you know a bunch of PhD students who are reviewers and convincing them to accept my work my audience is now customers and I'm convincing them to pay us money for it and I think that is a much more rigorous much higher standard

than getting a paper in ifs I had dinner with a customer earlier this week and they're doing some really cool stuff they have some really interesting problems I'm going to get on a plane in

two weeks and just go down to their office for the day and meet with their team all day and just learn more about this problem because I want to understand it and bring it back to my team as a question worth asking you know

it's not 100% of my time but I think you should be willing to just jump on a plane and go chat with an insurance company and spend a day with their machine learning team learning from them and and what they've done and hearing their problems and seeing if we can do

something creative to help them that's good research and if you ever sent me back to Academia that's probably still exactly what do one of my favorite things that you and I spoke about at NE

some weeks ago was the existence of a high school track at the new Europe's academic conference about Ai and I wonder if you

could share a little bit about that and about what you saw and what that tells you about the next next next wave of thinking in AI so the high school track

at nerves was really cool and also really controversial for a number of reasons you know is this just another way for students who are incredibly well off and have access to knowledge and resources and a parent who works for a

tech company to get ahead further or is this an opportunity for some really Extraordinary People to show how extraordinary they are and for people to learn about research much earlier than

certainly I did and try out doing science but there are kind of generational changes in the way that people are interacting with Computing this is something that you know my colleague hanlin who was one of the co-founders of mosaic has observed and

I'm totally stealing from him so thank you Handlin seeing companies that are founded by people who clearly came of age in an era where your interface to a computer was just typing in natural

language whether it was to Siri or especially now to chat GPT and that is just the way they think about a user interface you want to build

a system well just tell the AI what you want and on the back end we'll pick it apart and figure out what the actual process is in an AI driven way build the system for you and hand it back to you

that's a very different way of interacting with Computing but that's the way that a lot of people who have grown up in Tech over the past several years a lot of people who have who are graduating from college now or have graduated in the past couple years who

are in high school now especially that is their iPhone that is their personal computer it's chat GPT you know it's not buttons and dropdowns and dashboards and

checkboxes and you know apps it's just tell the computer what you want and it doesn't work amazingly well right now someday it probably will and that day may not be very far away but that's

a very different approach and one that is worth bearing in mind I want to switch gears a little bit and get to a technical debate that we've had over the years as well which is about the mix of

techniques Enterprises and app developers are going to use to apply AI to their data and of course Rag and in context learning have been exciting

developments for years because it's it's just so easy and it's just so appealing to put data in the prompt and reason about that with the best model that you

can find but there has been a wave of excitement renewed wave of excitement I'd say around complimentary approaches

like fine-tuning and test time compute reinforcement tuning from open Ai and and lots more and I wonder if now is the moment for that from a customer perspect

perspective or if you think we're far ahead of our skis and what's the right time and mix of these techniques that enterprises and app developers are going to want to use and my my thinking has

really evolved on this and you know you've watched that happen but I think we've reached the point where the customer shouldn't even know or care I

just want an AI system that is good at my task and I want to Define my task crisply and I want to get an AI system out the other end

and whether you prompt whether you do F shot whether you do an RL based approach and fine tune whether you do Laura or whether you do full fine tuning or whether you use DSP and do some kind of

prompt optimization that doesn't even matter to me just give me a system get me something up and running and then improve that system surface some

examples that may not match what I told you my intention was and let me clarify how I want to handle those examples as a way of improving my specification for my system and making my intention clearer

to you and now do it again and improve my system let's have some users interact with the system and gather a lot of data and then let's use that data to make the system better and make the system a

better fit for this particular task who cares whether it's rack who cares whether it's fine tuning the only thing that matters is did you solve my problem and did you solve it at a cost I can live with and can you make it cheaper

and better at this over time and from a sci ific perspective that is my research agenda right now at data bricks but you shouldn't care how the system was built you care about what it does and how much

it costs and you should be able to specify this is what I want the system to do in all sorts of ways natural language examples critiques human feedback natural feedback explicit feedback everything and the system

should just improve and become better at your task the more feedback you collect and your goal should be to get a system out in production even if it's a prototype as quickly as possible so you start getting data and the system starts

getting better and the more it gets used the better it should get um and the rest whether it's long context or you know very short context whether it's rag with a custom embedding model and a ranker or

whether it's fine-tuning at that point you don't really care and so the answer should be a bit of all of the above and I think most of the successful systems I've seen have had a little bit of everything or

have evolved into having a little bit of everything after a few iterations in previous versions of this conversation you've said dude rag is rag is it that's

what people really want there's other things you can do to extend it but so much is possible with rag that we don't need to look past that horizon yet and I hear saying something very different now

I hear you saying customers don't care but you care and you're sounds like you're building a mix of things yeah I think what I'm seeing the more experience I get is there is no one-size

fitall solution that rag Works phenomenally well in some use cases and absolutely Keels over in other use cases and it's really hard for me to tell you where it's going to succeed and where it's not my best advice to customers

right now is try it and find out so there should be a product that can do that for you or help you go through that scientific process in a guided way so you don't have to make up your own

progression and so really for me it's now about like how can I meet our customers where they are whatever you bring to the table tell me what you want the system to do and right now we'll go

and build that for you and figure it out together with your team but I think we can automate a lot of this and make it really simple for people to Simply bring what they have declare what they want

and get pretty close to what a good solution or at least the best possible solution will look like it's also part of my recognition that this isn't a one-time deal where you just go and

solve your problem it's a repeated engagement where you should really just try to iterate quickly get something out there and get some interactions with the system learn whether it's behaving the

way you want it to learn from those examples and go back and build it again and again and again and again and do that repeatedly until you get what you want and a lot of that I think can be

automated too at least that's my research thesis that we can automate or you know at least have a very easy guided way of going through this

process to the point where anybody can get the AI system they want if they're willing to just come to the table and describe what they want it to do what's the

for this sphere of opportunity of new model paradigms such as test time compute now even open source with deep seek so I would I would consider those to be two separate

categories I I was playing this game with someone on my team actually earlier today where you know he was telling me like yeah deep seek has kind of changed everything I was like didn't you say

that about Falcon and llama 2 and mistol and mixol and dvrx and so on and so on and so on we're just living in an age

where the starting point we have keeps getting better and we we get to be more ambitious because we're starting further down the journey this is like when you know our friends at AWS or Azure come

out with a new instance type that's more efficient or cheaper I don't go and look at that and go like everything has changed I go and look at that and go those people are really good at what they do and they just made life better

for me and my customers and we get to work on cooler problems and a lot more problems have Roi because you know some new instance type came out that's faster and cheaper it's the same thing with

models for new approaches it could be something like a DPO or it could be something like test time compute and you know it's hard to those are probably not comparable with each other but just

these are more things to try these are more points in the trade-off space I think about everything in life as a PTO Frontier on the trade-off between cost and quality and test time compute gives

you this very interesting new tradeoff possibly between the cost of creating a system the cost of using that system and the overall quality that you can get every time another one of these ideas

comes out the design space gets a little bigger more points on this trade-off curve become available or the curve moves further up into the left or up into the right depending on how you define it and life gets a little better

and we get to have a little more fun and for this product and this system that we're all building at data bricks things get a little more interesting and we can do a little more for AUST customers so I don't think there's any one thing that

changes everything but it's just it's constantly getting easier and constantly getting faster and constantly getting more fun to build products and solve problems and I love that a couple years

ago I had to sit down and build the foundation model if I wanted to work with it now I already start way ahead I love that Jonathan I've got some rapid

fire questions that I'd like to use to bring us home Bring It On Let's let's do it so what's a hard lesson you've learned throughout your journey maybe

something you wish you did better or maybe the best advice you received that other Founders would like to hear today so I'll give you an answer for both I

mean the hardest lesson I've learned is honestly it's been the people aspects it's been how to interact productively with everyone how to be a good manager I don't think I was an amazing manager

four years ago fresh out of my PhD um and my team members who have been been with me that long or the team members who were with me then will surely tell you that I like to hope the team members who are still with me think I'm a much

better manager now um and the managers who have managed me that entire time who have trained me and coached me think I'm a much better manager now learning how to interact with colleagues and other disciplines or other parts of the company learning how to handle tension

or conflict in a productive way learning how to disagree in a productive way and focus on what's good for the company learning how to interact with customers in a productive way and a healthy way even when you know sometimes you're not

having the easiest time working with a customer and they're not having the easiest time working with you those have been incredibly hard one lessons that's been the hardest part of the entire Journey um the part where I've grown the

most but also the part that has been the most challenging the best advice I've received probably from my co-founders n and Handlin like one piece of advice from Handlin that sticks in my mind is

just he kept telling me over and over again that a startup is a series of hypotheses that you're testing that kept us very dis in the early days of Mosaic stating what our hypothesis was trying to test it systematically finding out if

we were right or wrong that hypothesis could have been scientific it could have been product it could have been about customers and what they'll want but it was turning that into a systematic scientific Endeavor for me made it a lot

easier for me to understand how to make progress when things were really hard and they were really hard for a long time I know that wasn't a rapid fire answer to a rapid fire question but it's a question I feel very strongly about

aside from your own what data and AI infrastructure are you most excited about and why I think there are two things I'm really excited about number

one products that help you create a valuation for your llms these are I think these are fundamental infrastructure at this point there are a million startups doing this and I think

all of them are actually pretty phenomenal I could probably give you a laundry list of at least a dozen off the top of my head right here and I bet you could give me a dozen more that I didn't name because we're all seeing great pitches for this I have a couple that I

really like a couple that I've personally invested in but I think this is a problem we have to crack it's a really hard problem and I think it's a great piece of infrastructure that is

critical the other thing that I'm really excited about personally is data annotation I think that just data annotation continues to be the critical infrastructure of the AI world no matter how good our models get and how good we

get at synthetic data there's always still a need for more data annotation of some kind and revenue just keeps going up for the companies that are doing it the problem changes what you need changes I don't know I think it's a

fascinating space in many ways it's a product in many ways like my customers these days the data scientists at whatever companies I'm working with are also doing data annotation or trying to

get data annotation out of their teams building an eval is data annotation and you know I mentioned two things these are both my second favorite because I think they're the same at the end of the day one is about going and just buying

the data you need togethers about tools to make it easy enough to build the data you need that you don't need to go and buy it and I I have a feeling both companies have made a lot of progress on AI augmentation or both kinds of

companies on AI augmentation of this process but when I do the math on the original llama 3.0 models this is the last time I really sat down I did the math my best guess was $50 million worth

of compute and $250 Million worth of data annotation that's the exciting secret of how we're building these amazing models today and I think that's only going to become more true with these sorts of

reasoning models where I don't know that reasoning itself is going to General wise but it does seem like you don't need that many examples of reasoning in your domain to get a model to start

doing decent reasoning in your domain and that's going to put even more weight on figuring out how to get the humans in your organization or to get humans somewhere to help you create some data

for your task that you can start to bootstrap models that reason on your task beyond your core technical Focus area what are the technical or

non-technical trends that you are most about so I think there are two as just you know one as a lay person and one as a specialist as a lay person I'm

watching robotics very closely I think you know for all of the interesting data tasks that we have in the world there are just a lot of physical tasks in the world that it would be amazing if a

robot could perform like thank goodness for my dishwasher thank goodness for my washing machine I can't imagine what my life would look like if I had to scrub every dish and scrub every piece of

clothing to keep it clean robotics is in many ways already in our lives these are just very specific single-p purpose robots but if we can really make a dent in that problem and I don't know if we

will this decade or in three decades like VR I feel like robotics is a problem that we keep feeling like we're on the cusp of and then we don't quite get there but we get some Innovation I

love my Robot vacuum that is the best investment I've ever made I got my girlfriend a robot litter box for her cats a few weeks ago I get texts every

day going oh my God this is the best thing ever and this is just scratching the surface of just the daily tasks we might not have to do I would love

something that could help people who for whatever reason can't get around very easily on their own to get around more easily even in environments where they're not necessarily built for that I have a colleague who I heard say this

recently so I'm not going to take credit for it but the idea of just things that make absolutely no logistical or physical sense in the world

that you could just do if you had robots in Bryant Park right now right below our data Brick's office in New York there's a wonderful ice getting rink all winter if you were willing to just have a bunch

of robots do a bunch of work you could literally take down the ice getting rink every night and set up a beer garden and then swap that every day if you really wanted to things that just make no logistical sense because they are so

labor intensive you could just do that and suddenly that makes a lot of sense you can just do things that are very labor intensive and resource intensive so that gets me really excited from data

intelligence to physical intelligence uh well somebody's already coined the fiscal intelligence term but yeah I don't see why not and honestly we're dealing with a lot of fisical intelligence situations at datab bricks right now so I think data intelligence

is already bringing us to physical intelligence but there's so much more one can do and we're just scratching the surface of that it cost Google what 30 billion to build extraordinary

autonomous vehicles and the whole narrative in the past year has completely shifted from autonomous vehicles are dead and that was wasted money to oh my gosh weo might take over the world so I'm excited about that

future I just wish I knew whether it was going to be next year or in 30 years the other Trend though I mean I spend a lot of time in the policy world and I think that's maybe even a good place to wrap

up before I was an AI technologist I was an AI policy practitioner um that's why I got into this field in the first place that's why I decided to go back and do my PhD I spend a lot of time these days

just chatting with people in the policy World chatting with various offices chatting with journalists just working with ngos trying to just make sense of this technology and how we as a society

should govern it um it's something I kind of do in my spare time I don't do it officially on behalf of data bricks or anything like that just because I think it's it's important that we as the people who know the most about the

technology try to be of service but I think coming as a technologist and asking how can I be of service and what questions can I answer and can I help you think this through and figure out whether this makes sense it's a very

fine line and you need to be careful about it but if you really come in with kind of your heart set on figuring out how to be of service to the people whose job it is to think about what to to

speak on behalf of society or to think on behalf of society you can make a real difference but you've got to build a reputation and build trust over many years but the flip side is you can do a

lot of good for the world that is definitely a good place to to leave it so Jonathan Franco Chief AI scientist of data bricks thank you so much for

joining this is a lot of fun thank you for having me [Music]