Economist Tyler Cowen on How ChatGPT Is Changing Your Job - Ep. 7 with Tyler Cowen

if chat gbt stopped existing today how would that affect your productivity I would feel much less smart AI is for learning perplexity is for references Google is for links I was in Tokyo I use

it as my Universal translator paraguayan food I was in Honduras mostly when I travel it adds flavor to the trip itself I actually created a clone of you I want

to do a segment with you called the Tyler test I ask you a question I'll have you answer it and then I will also ask the Clone that I made the same question question we'll see if it answers the question in the way that you

would what are the core lessons of Economics the first would be incentives matter the second would be there's always an opportunity

[Music] cost Tyler welcome to the show happy to be here thank you for having me on of course I'm really excited to do this for

people that don't know you you're an econom IST at George M University you're a prolific writer you've written I think 17 books and you've been writing your blog marginal Revolution for almost or

over 20 years and I find you to be a just incredibly deep thinker you think a lot about the impact of technology on life work and the economy and I think it's all incredibly relevant to to the

show and I'm just yeah super excited to have you happy to be here great cool so what we tend to do on the show is talk very practically about how smart people

use chbt and AI in their lives to get work done but I think before we get to that there are some sort of highlevel questions I want to ask you about your work and your view on AI and its impact

on technology and and jobs and the economy um so I think let's let's start there um and in particular I think there are a lot of people right now who are

trying to figure out what AI is and how it's going to affect their work lives and the economy more more generally and I think you've been thinking about that question about maybe AI in particular for for a little while but like

generally about how technology affects our work lives for a really long time um and in your book average is over which you wrote in 2013 you talked about this like stratification in the economy

that's driven by what you call what you called intelligent machines and this is before like real AI had had even sort of come out um well in chess we had real AI

in 201 true yes I guess uh uh you know before let's say just before this generation of consumer AI had come out or was even like on the horizon um and one of the things that you wrote is that

if you and your skills are a complement to the computer your wage and labor market prospects are likely to be cheery um and if your skills do not complement the computer you may want to address

that mismatch so you talked about this sort of stratification between um intelligent technical knowledge workers and the rest of the economy um and and

and you saw that Gap and you saw that widening in 20 in 2013 and I'm just kind of curious for you to talk about that prediction like why you wrote that what you saw then and what you think now

given the current generation of AI models well for a long time even well before 2013 I thought artificial intelligence was the most likely place

for the next major technological breakthrough to come and I'm not an AI expert in any technical sense but my intuition there was pretty simple I was

quite an avid chess player when I was very young and at the time people thought well AI can't really play chass chess is too conceptual too complex too

non-led and pretty quickly as I think you all know AI managed to overcome all those hurdles and it's now almost Godlike when it plays chess right so I

thought if AI can do that chess is really hard AI can do all these other things in time and I was convinced this what happen and it's happening essentially so that's the background of

the book and I wanted to be ahead of the trend and write about where I thought it was all going and um and I guess in and you're so I think you're I think it's clear

that you were you were ahead of the trend um and and I guess you you made some predictions about uh what how that would impact the economy like do you feel like that was that was spot on do

you are there any revisions that you want to make given what you you know I think it's probably at least somewhat surprising the the the the exact specifics of this current generation of

generative models um and I'm sort of curious how you reflect on that what's happening now is a work in progress I don't feel that we know yet which groups of people it will help the most and harm

the most like many people I didn't expect that AI the AI breakthrough would have so much facility with words and with emotions and with bedside manner I

don't mean that I ruled out the possibility uh but back then I would have been surprised if I was told well those are some of the things that will be best at so I thought it would be a

bit more of an autonomous reasoning tool in a way different than what it is maybe more along the visions of symbolic AI people uh but it is incredibly fasile

at taking some idea and writing a rap song around it or a poem or a shanty song or whatever you might want uh and that is different I think think uh what

are sometimes called the word cell classes may be under greater threat from AI than I would have predicted uh but again this hasn't all quite come to fruition yet we're waiting and I expect

many surprises got it so so it's it's sounds like you're not you're not really sure and you and you don't necessarily want to like Hazard a guess because it's too early to say no I'm very willing to

Hazard a guess but I want the the uncertainty brackets to be understood okay so my best guess is this in the short run it will be quite egalitarian as people who can't do

things well at all Now can do them pretty capably say like writing an online essay to get into college the smartest kids already could do that well GPT doesn't help them that much and they

were going to work hard on it anyway uh but I suspect the longer run effects will be less egalitarian that as people learn how to use these things and they

get better that more more people will use AI to sort of help them build out projects so the people who can start projects will be the major beneficiaries

they'll have better record Keepers and translators and mathematicians and coaches and colleagues and advice givers but large language models they're

not a source of actually doing the idea you can use them to get ideas but they don't make the decision to go out and do the idea yeah so some kind of hyped up executive function maybe is what will be

rewarded say over a 10-year Horizon yeah and that could be inegalitarian because it's talented people who have a lot of project ideas anyway who might benefit the most that's my best guess I think

that makes sense and and so to to sort of summarize what I what I heard from you is right now what it can do is it can give people skills to do tasks that they would ordinarily not have been able

to do at all so like maybe you're not really a programmer and it can help you build build a very very simple version of an app or maybe you couldn't really write a colle essay but it can it can it

can do that for you so it can bring the sort of the bottom cortile of of skilled of skills up um and then even bottom 60 70% right most people cannot write a

college essay most people can't finish college okay right so so it can it can it can augment those people's skills to let them do something that they couldn't do before and then for for other people

who are who are more skilled in a particular domain let's say a highly skilled programmer it makes them slightly better but not it's not going to necessarily make them do things that they could never possibly do before and

that's the that's the immediate effect is what you're saying yes right and then the SE and then the sort of secondary

effect um is something like uh it it enables people uh to who who want to make things in the world to like go to go and make those things with the with

the intelligence that it that it uh that it unlocks so like one of the think one of the ways I've been talking about it is is like the allocation economy it's like people who know how to allocate resources um so those are people who

have like managerial skills or investing skills um are going to be able to deploy those much more quickly and cheaply by using AI instead of using uh like

building a human organization that's right yeah I think there's another longer run effect that is not at all in place now but at some point will be and that is when we can use AI to evaluate

people's Talent MH uh and that will be highly meritocratic in a narrow way I'm not sure we'll all be happy with it to be

told just how good you are at something is perhaps unsettling yeah and on net diminishes human happiness uh but we will find potentially successful people

much more easily at some point that is not in place now I definitely find that it's it's really quite good for helping me understand who I am like giving me a

picture of who I am so like I'll I'll often do things like I'll uh take long journal entries and I'll throw them into chbt or into Claud and I'll say like hey what are the psychological patterns that

you're noticing here and it's really good at like taking them out and putting them into words for me and I think having that reflection if you ask for it

is really powerful you know I'm about to turn 62 so that's maybe of lower value to me but I agree with your point about large language models they're wonderful

therapists and analysts and they can be remarkably objective when that's what you ask for yeah yeah I want to just this is this is a sort of this is a sort of slight tangent but I want to just dig

in on the sort of allocation economy idea that we've been talking about because I've I'm writing an article about this for Friday actually and um one of the one of the I think interesting implications of that is

right now the skills of of being a manager um are not that widely distributed or talked about because there's only a small group of people who

are managers um but I think those skills may need to be um more widely distributed in the near future as everybody is moving up a layer of

abstraction so they're even Junior employees for example are thinking about things like for example a really common management problem is like how far into

the details should I go um um like I ask someone to do something do I micromanage them or do I let them go do it but then it comes back wrong like that's a that's a specific skill that managers have to

learn that Junior employees don't have to learn that I think might be valuable in an economy like this have have you thought um more about like what are the specific if you're uh if we're thinking

about allocation or allocating resources as like a as a core skill of the Next Generation have you thought about the like specific ways or skills that are that are bundled under there that are going to be important

that's a very good point also if you look at even the top management consulting firms the kind of advice they give to many people it seems rather anod

and boring and repetitive and cliched and that's a sign GPT models can do it really well yeah and I think they can so that might just end up automated and you

might still need the consulting firm to make it stick or to be the focal voice putting the message forward yeah but the actual amount of Labor you need doing that so it's like everyone

will have free McKenzie level advice or nearly free but another thing I worry about as you know there's a lot of talk about Ai and biorisk what about developing dangerous pathogens I think

it's a big constraint on terror organizations that they're very poorly run that's a much bigger problem than oh they can't find the optimal pathogen to do us all in right they just don't

succeed or even try very often so if they have access to AI is giving them better management advice I think that's actually one of the biggest risks of AI technology that is fasina not the

pathogen just the here's how you run a successful terror group and maybe current models are sufficiently protective that they won't tell you though even there you can

probably break through the dam but there'll be very good AIS that will tell you that pretty soon right yeah I think that that's such an interesting underappreciated risk because yeah I

think anyone that's thinking about AI risk is really interested in the hard problem of like here's how you assemble chemicals in a specific way to like make

a bomb um but the the human coordination problem is actually like a significant deterrent for people doing bad things um and human coordination

deterr yeah yeah and human coordination is like one of the things that I think AI going to be really good at um and and we see that people talking about that already in the in terms of business like

you even yourself wrote um you wrote an article recently about chat gbt and and and your career trajectory and what you what you said is that um small

integrated teams will produce the next influential big thing um which I think you meant um you can do a lot more with AI with a small team and I think that's

a really positive thing but but the the the the negative is that you can do a lot more harm that's right a lot of companies might become much smaller but

still rather potent yeah I think mid Journey when it first had its breakthrough had what seven or eight people working there I read uh it may be more now but it's not going to be so

many more yeah it's very small um I'm kind of curious like on on that on that point um one of the things that you've written a lot about um and

you you wrote about the in in a previous book um called uh the age of the inore is um there are certain skills that are

super valuable in the knowledge economy um specifically like ordering knowledge um and also like being very attuned to being able to remember and use small

bits of information within a specific subject area that you associate with people who are autistic um and and and that book in particular talks a lot

about um why those traits are actually quite valuable uh in this current ERA and one of the things that strikes me is like that's also a lot of the things

that like chbt or or other AI tools are are fairly good at and I'm curious how that changes or what that does to your perspective to like to look at it that

way I think in the short to medium term AI accelerates the value of those skills ordering information grasping how things fit together knowing a lot because you

now have this extra way of learning but there could be some point much further out where the AI simply does all the work and the value of that skill becomes quite low and skills like Charisma and

executive function are what rise in importance yeah but I think to get to that point we would need AIS really incorporated into major workflows and

systems in a way that's fairly distant so I don't think it's oh just a few more years and then this is going to flip I think for the foreseeable future uh it

will help the infavor GPT models and so I I I assume you you you include yourself in that like sure of course yeah are you thinking for yourself like

about upping your your charisma level or you feel confident enough that it's it's far enough out that you don't need to really worry about that or maybe you feel charismatic enough you don't need it well that's for the world to judge I

will say I've decided to do more personal appearances both as a way of projecting but also as a way of learning things that I can't learn from you know say a large language model so it's

definitely influenced my behavior already okay cool uh I'd love to start to get into some of the Practical aspects of this um which is yeah more

specifically how how you use Chachi BT and before we like you're you're already in in a screen share um but before we we dive into it I I'm kind of curious like how would you summarize that a high

level the way that it fits into your life and your work there are two quite distinct ways I use chat GPT one is on iPhone and the other is on my laptop and they're

totally different so I don't know if you want to cover both or take them in sequence but the the iPhone is pretty simple do you want to start with that let's do it so if I'm in a foreign

country I was in Tokyo uh don't speak Japanese not many people there speak English I use it as my Universal translator M it will get better than

that but it's fine already yeah it's not Star Trek great but it's amazing and the other thing I do is I use it to read menus or even just tell

me what should I order yeah so I was in buos Aris I was at a paraguayan restaurant Paraguay is one of the countries I've never been to so I'm an

idiot when it comes to paraguayan food I took a photo of the menu asking well GPT what should I order here what which are

the class dishes and it tells me now that's amazing I was in Honduras a few days ago you see a plant you see a bird you don't know if you can take a photo

of it you take the photo you ask what's this and it tells you right so I use it for that mostly when I travel again that's distinct from my main uses yeah uh but I'm quite sure it will be

enduring what do you think that that opens up for you like what is that what is that like to be able to take a photo of a bird or take a photo of a menu and just like know what the menu says or

know what the bird is well you just learned something right but a big part of the value is the interactive experience with the other person you're with that you in some way

are discovering this together makes it more memorable and I think you're more likely to do it with another person or people than if you're alone yeah so it

adds flavor to the trip itself like you have these now multiple modes of how you discover things that's really interesting it reminds me um I don't know if you know Andrew Mason he's the

former um he's the former CEO of Groupon and he runs descript the the podcasting software company I might have met him once but I don't know him yeah he used

to run this company that was it did like audio tours of cities and so what you would do is and they were all I think created by the company and then eventually created by like third party

tour guides where where wherever you were like I did this in Rome you could um walk around and it it turned the whole city into audio tour sort of similar to like the one that you go on

in a museum but it was the entire city um and you could pick one that's like history or food or architecture or whatever and you would walk to a specific spot and then it would say like look to your left and this is a building

and here's everything about the building and it that company didn't work but it was a it was a pretty magical product experience and it sort of strikes me

that in some ways Chachi BT is that um without needing it to be a a separate company yes and if If you're sort of on the road out there and there's some fact you want to know like oh what's the

population of the city uh it's easier and quicker than Google or Wikipedia maybe it's a small Edge but for for something as important as that I'll take

a small Edge yeah I just ask it right what if you like this this actually reminds me of um I before we recorded I I I asked Twitter

for uh questions for that people wanted to ask you and um Patrick McKenzie who you interviewed on your podcast recently uh asked me to ask you how do you check the output for correctness

workflow wise and where do you feel comfortable not doing this and I think that's that's kind of related to this example because like particularly in a in a country like Paraguay or Japan

where you don't speak the language um it's you're it's like hard to actually check the output so like do you have any specific rules or heuristics for like

when you when you trust the output and and um when you don't I worry about hallucinations a lot less than most llm users I think so a lot of my questions

oh what bird is this let's say it's wrong and it's just rolling my leg okay like I didn't know what bird it was and now I think it's the wrong bird like who cares I'll just ask it again and move on

uh but pretty often I'll just ask it are you sure that's the right answer or please correct any hallucination and most of the time there's a problem it does correct it not

all of the time I'd guesstimate 80 to 90% if you just ask it again it'll correct an error yeah so that's one layer of check uh I think also the areas

where I tend to use it this is now laptop GPT it just makes way fewer errors than the areas where a lot of other people tend to use it so uh that

also limits the degree of hallucination how do you characterize the areas where it makes fewer errors I couldn't give you a general answer yeah but I can tell you I'm typically using it to learn

obscure history and obscure history maybe by the nature of the questions you're sort of pointing it in the direction of an intelligent part of the information

space that's not too crazy not too controversial yeah if you ask you know about the old Byzantine Empire it does pretty well I did a whole podcast or I

made believe I was talking with Jonathan Swift you know the the Irish writer 17th and 18th century I asked did a series of detailed probing questions for an hour

this has been put online uh it didn't make any mistakes not one and I there were no redos or no trial runs I just did it and what I did on the first try

was the final output so presumably it's read a lot of Swift read a lot about Swift it's obscure and detailed enough maybe it just ends up boxed into the

corners of Truth a lot yeah so for me uh you know I know how you can get it to hallucinate if you just ask it what are the three best books to read on Jonathan

Swift my best guess is one of the three answers won't exist I understand that but I don't run into that so often because I'm not asking it that question right that's really interesting um I

definitely have found that too I haven't used it for obscure history but I use it a lot actually to read old books so um one example I was reading Moby Dick

recently and it's really helpful to just take a picture of a passage ask it to explain what's going on or even to like visualize the passage where you're just like hey can you can you show me what

this looks like with Dolly um and I think the reason why it's really good for that is uh it has access to all the text it's read all the text and there's a lot of supporting there's many decades

of supporting commentary so it knows what to say whereas for more recent stuff it's more likely to be copyrighted and and less likely for it to know have

read it another thing I use it for a lot where hallucinations are not a risk so I have my own podcast conversations with Tyler I have guests on I want to learn

the background context to what they do yeah so I had one interview with Lazarus Lake he runs ultramarathons he hasn't really written anything there's not that much written

on ultramarathons that's readily available but if you keep on asking the questions about ultramarathons you acquire all this background

context maybe some of it's wrong it's not really going to matter I'm the one asking questions not answering them and I just feel I know my way around the topic pretty well yeah I'm learning now

about the insurance industry we can talk about that more which actually the published literature on economics of insurance and its history it's very bad

and sparse for whatever reason if I keep on asking GPT again I'm not convinced the answers are perfect but I know I'm doing better than I would do any other way right and I'm not the

one required to know the exact fact what I want is context that makes sense yeah I guess uh when you're when you're using it for example in in the insurance area and you

say you're not the one that is required to know the exact fact like don't you get sort of nervous that you're going to have something in your head that's like kind of slightly wrong well I already do

right that's the problem I could show you one of the things I asked it if you want to look at something concrete yeah I want to sort of set the stage here really quick so we're going into your

use of laptop GPT yes and one thing that I think is really important to note is you're not in Chachi BT you're in the playground um and so before we we head

into a specific chat why do you use why do you use the playground tell us I I don't have any good answer I mean I'm I'm stupid basically I like the fact

that it doesn't look like a consumer product and it works for me and the maximum length I can slide across make it longer shorter maybe it doesn't even

matter yeah but I just enjoy that and I'm used to the look yeah so I use it do you think it's a mistake for me to use it I actually don't think it's a mistake I mean you're you're sacrificing certain

things right so like uh you don't have access to code interpreter you don't have access to Dolly uh there's no unified experience between mobile and your desktop um so there's a bunch of

things that are not as good about the playground but one of the things that is going to be better about the playground is you have a lot more control over the output so like you can see right right

on your screen you can see you can set the system message so the system message for people who don't know is going to set the um kind of like the personality of how gbt interacts with you and and

and setting that can be like really helpful on the right you can also see you can set the the model you can set the temperature so temperature is sort of like creativity for the model um How

likely it is to be factual How likely it is to sort of like make stuff up um you can set the max length all that kind of stuff so you have a lot more control and then another thing that I think is probably true of the playground that's

not true of chachy PT is the content restrictions are going to be looser in the playground because it's built for developers and not for consumers and so they're going to be less careful about what the model says because it's only

being used by a certain group of people who are using it to make make products instead of like hundreds of millions of people who are using it every day for random stuff so I'm not stupid that's nice to hear I mean I have Dolly and all

the rest I just don't use them much uh the images are not my main interest I think it's amazing uh but I'm not sitting there playing around with Dolly I don't think

it's a waste of my time yeah um okay so so tell us so let's let's go into into the first chat you you have a chat right here I think you're I think it's about

insurance tell us about like um tell us about H how you did this chat what what was the motivation behind it and then and how you started it well Alex tabarok

and I agreed that we would be recording a podcast about the history of insurance and insurance economics and I'm really not well informed about that area in part because

the literature is not that good I do read a lot so one question I had was about this difference between whole-term and regular life

insurance uh the whole product is like a savings investment bundled with the insurance itself and the other is just the plan old life insurance so that

distinction between the two kinds of insurance when did that arise I don't know probably I still don't know but I thought I would ask GPT and you can see my question on the

screen I can scroll SC down and show you the answer so to read the question you said when did the distinction between the whole term between whole term and plain life insurance arise historically

right and it looks like uh GPT said the distinction between whole Hall life and term life insurance involved over time with the development of the life insurance industry term life is designed to provide coverage for a specific

period with no savings so we're getting some definitions whole life insurance uh offering a death benefit along with the savings component allowing the to accumulate a cash value became more

common in the 19th century as the industry matured um so yeah what did you think of that answer like did it did it give you what you wanted well I don't know if it's true uh

so much of the insurance industry matured in the United States after the Civil War so if that distinction came at that time it's at least plausibly the

case uh and then it says well the distinction became clearer in the 20th century as life insurance markets became more diverse yeah uh I thought it was a good answer I

hope it's a good answer I'm not going to flat out assert that in a dialogue but if it had said oh that was there from the beginning in the 17th century or it came only in

1964 yeah then I would definitely look more into it and try to learn something quite specific but right now it's just telling me it's following some general

patterns I might have expected anyway M that's useful information m i probably don't use it directly M uh but then I go on to my next question can you see that

now on the screen I see it yeah so you said why would anyone want life insurance bundled with the savings return doesn't it mean inferior returns tell me tell me about that why'd you ask

that question well it's a common piece of advice for Americans investing that you should not invest through your life insurance policy that you can do better

with the direct purchase of equities through a lowcost diversified mutual fund right that's pretty standard advice so I'm asking GPT why would anyone want

to save through their life insurance which is a known question so it gives eight reasons uh a few I hadn't thought of at all one or two may not be good

reasons but they're probably actual reasons for particular human beings yeah and it's a good answer what can I say I learned something I don't think it

screwed up that was my answer right yeah how do you think um how you summarize the kinds of answers it seems like there's there's some things you would have thought of some things you wouldn't have thought of and some things you

think are not that great like what is the what is the value of that for you it Spurs my thoughts I get additional bits of

information uh I one thing I didn't do but what I would then continue to do is ask it a question like well if someone were writing a criticism of whole-term life insurance yeah what points against

it would they make yeah and my guess is that answer will be quite strong do you have any I don't feel I need to ask it partly I know but uh that would be the sword of

followup if I had just done this you think we could try it sure let's try it we can talk about what's in between also yeah we can go back up in a sec but I'm sort of curious about

that if you were to write a critique of why individuals should not by whole term

life insurance which points would you stress please answer like a sophisticated investment advisor those

add-ons you don't need as much as it used to I find but they're not going to hurt interesting tell me about tell me about that tell me about asking it to sort of simulate a sophisticated investment

advisor again you're pointing it to a more intelligent part of the information space so if I ask it a question say from economics what is inflation the answer is not wrong but it's not really better than

Wikipedia because the question's too General if I ask it a question what is inflation answer ASW Milton fredman you get a better answer usually it's not nothing to do with agreeing with M and

Friedman or not though I mean you might I guess I do uh you're just again pointing it toward smarter bits you know in the matrices somehow right and

what this the stuff it knows connected with milon Freedman is smarter than the stuff not connected to fredman right so as you get more specific you're going to get more specific detailed responses which is what you want that's right and

it loves compare and contrast even if you don't want to compare and contrast you often get the best data by asking it to compare and contrast because it's like Crossing two swords and maybe

something about the auto regressions they just get better interesting uh so here's the critique there's some General blah blah blah at the

top uh cost efficiency whole term life insurance has higher Premia uh investment flexibility it's not as

flexible less transparency less control that's true you're less liquid it mentions that's number four uh five it's comparing the tax

considerations to a Roth IRA excellent point I hadn't explicitly thought of that mhm the whole term can have more financial complexity higher opportunity cost which

is the key main point could be invested elsewhere I'd like to see it mention equities there but okay it didn't risk diversification the insurance company may not be optimally

Diversified that's also true you know on any exam I'd give this answer an a maybe not A+ but a solid a right it's all correct nothing wrong

with it it did great one thing that we might want to try which could be could be interesting is if you delete this specific answer um yeah which I think you can you can do pretty easily um

you'll have to tell me because like I said before I'm stupid yeah yeah scroll over it click into the assistant answer there should be a minus button there learn something new every day see the

button there you go okay so I got rid of it yeah so what I want you to do now so I think it'd be fun to play with the temperature so right now the temperature is one um but if you move it up it'll

get more creative yeah um it might be interesting to see if you go all the way all the way out what does it do oh yeah this this is something they're

gonna ban on Twitter let's try it'll [Laughter] become see there you go it's too creative right too creative um but I

think that's that's an interesting reflection you can you can try it again with a slightly less yeah slightly less temperature like maybe Midway um and if

you just uh click the minus button again yep ask again uh press uh press cancel see where it says cancel yeah yeah and then submit

submit opportunity cost comes first now which is I would say better interesting so you think it's maybe slightly better so far slightly better it's the same

point yeah but in a better prioritized order and then it went off the tot nonsense now it's off the rails but that's fine yeah I I think that that's I think that's a really interesting thing

about using the playground as you get to do this and you get to see these are some of the rough edges that like they don't really want you to see in a in a consumer app um but it does allow you to

sort of explore the space of possibilities yes and the the upside the downside of exploring the pace of the space of possibilities is that you

sometimes get junk but the the upside is you'll might you might find stuff that no one has found before um and in this case it like before it went off the rails it actually gave you a slightly

better answer and I think that's an interesting tradeoff for people to consider when they're thinking about when to use chachu BT or when to use something like the playground if I used it for power tree I would use the higher

temperatures more right I've experimented only a bit but I'm actually pretty fine with it one right yeah that seems that seems like it makes sense for you um I'm curious like yeah obliterate

this we can talk about some of the other queries in here let's do it yeah okay so uh here's one where I couldn't quite pull the teeth

out so I was reading a book about the the Byzantine Empire it's a area I don't know much about and it mentioned inflation under Emperor aelan which is the early part is

still Roman Empire times before before it splits and I was like okay well how high was that inflation I wanted to know I really wanted to know I didn't just do

it for this show and I asked it and it understood the the query very well and it told me all about the limited data it may just be the best answer

maybe we don't know we just know if tax for people complain inflation was high it mentions his story in Richard Duncan Jones M I have an imperfect recollection

that his is the right name to site so that's helpful I recognize it but wouldn't have thought of it so I could put that into scholar.

gooogle but then at the end it says quantifying it with any exact figure D D D D du you know it's like falling asleep on the job right not willing to commit

so what do I ask if you had to guess a specific number maybe I should now be demanding my money back right sometimes that works now it just goes into full evasive

mode as an a I developed right don't even need to talk through that everyone listening knows the deal right then more blah blah blah basically it should have just said I'm not going to tell

you uh but here finally some scholarly estimates say it might have been over a th% over the course of half a

century now I don't believe that I think that has a decent chance of being a hallucination MH uh but it told me something and it's like some estimates

we're not sure like I know we're not sure so the hallucination may not be worse than the hallucinations of you know my fellow economic historians

that's okay I'll live with that yeah if it had just said oh we think it was 30% uh maybe I wouldn't trust that yeah but it said something

okay this this actually reminds me of of of a really kind of important Point going back to to you know talking about the skills of managers being more

important in this in in this new world um so like one of the skills of a manager that I think you're you are picking up on and you already have

because you do this I think probably with research assistance is you have a sense for when you get a fact like that a th% it's probably not right there are some things in I answer that are

probably right and some things that are not right and I think there are a lot of people who are evaluating gbt chbt that see that and they're immediately like well I can't trust that

I'm not going to use it it's not useful right but what managers of people know how to do generally is to take an answer from someone and know which details are like maybe they need to follow up on and

maybe the ones that they they don't um and that is a skill that you are deploying here to hold lightly to the things that that uh that don't that sort of trigger your sense of I should

probably check that and then maybe I don't know if you did this in this chat maybe like go deeper on it or maybe switch to another tool to like follow up and that's an example of I think a skill

that managers have to know that Junior employees don't generally have to as much that's right now I might try perplexity AI at this point which is out

to be great do sure yeah under aelan what was the rate of price inflation and I'm clicking and I'll tell you the

answer we go here we go into perplexity there you go some sources say 3% a year others say 5% a year the basement of the

currency now one of the great things about perplexity is that it gives you citations and I would track those

down and uh one is Reddit the other's J store I I would look at J store most of all I know who it is I should ask and if

I needed to know I would do that that's how my process would proceed I would next try perplexity rather than torturing GPT anymore because I know

sometimes when you torture it uh it's like no the dog just doesn't want to go for a walk right right so let's playground so like let me let me just let's unpack that let's stick with that

for like a sec so it sounds like the stack for information gathering is something

like Chachi BT then perplexity then maybe the sources from perplexity then people does that sound

about right often now if I need to know something quite particular to say finish a piece of work I might start with the person of course MH and there's an

advantage to starting with the person the person may be delayed in responding but you know the other things won't be so that which will delay you ask first

to cut down on the length of the delay right there's something doesn't logically make sense right there there's something really important missing from that list which is Google when do you

use Google I use Google when I need links which is quite often so I still use Google a lot but I use Google less and less for information but when I write a column for Bloomberg or do a

blog post I need a link getting the clickable link Google is still clearly best by far got it but not for learning things not for learning things now the link may lead you to what you want to

learn I get that uh but Google is for links Google is for links AI is for learning AI is for learning perplexity

is for references and sometimes has context that GPT doesn't because it's looking to tie it to references and why do you think you

start with chbt versus starting with perplexity it depends on the query if it's something recent and I want to know the citation above all else I might

start with perplexity but I've only been using perplexity a few weeks uh and just my patterns with it will still evolve right so it's a little less familiar it's pretty new it's

really having a moment right now like it's they raised they raised a big round and and a lot of the guests on the show have been using it I've started using it more my dad's using it which is actually

like usually a big sign for me um he was using Zoom like years before anyone else um and so he he just he's sort of like the canary and the coal mine for me of

new new interesting software um so uh going going mainstream so uh yeah I'm I've i' I'm really interested in following their trajectory yeah someone

should try to buy them if they can yeah um now here's a question where I got hallucinations do you want to do one of these please let's do that so on Twitter Jordan Schneider asked me

and here you can see on the screen what is the best book on the Civil Service reforms of the progressive error as related to the Pendleton Act some detail up further on the screen but don't worry

about that yeah and I I asked GPT but I knew that wasn't the way to go I just thought well I'll do that as an experiment in part for this program and the first book it mentions

Pendleton Act of 1883 by Jonathan grotzinger it's not an Amazon uh I'm not sure it exists probably it doesn't exist uhuh it's sort of the book I would want

but it doesn't exist uh this other book reforming the Civil Service I couldn't find so it's

hallucinating more than usual uh then it somewhere maybe it was the the query one up it recommended Robert V weeby yeah the search for order

which is quite a good book and it's a good book for me to recommend not mainly on civil service but that was something it where it jogged my memory Al so mostly it failed on that question

but again I would know to begin with not to ask it that it's asking for trouble right I've personally found it to be stunningly good for book recommendations

um and in particular what I what I like to do with it is have it help have it helped me to identify my taste so I'll throw in a bunch of books and then be like what are the commonalities between

these books and it'll be like here's here's the kind of stuff you like and then I'll push it further and be like okay tell me more stuff that I haven't heard of and I've found a number of books that I that I absolutely love from

it I use people for that basically and that works very well for me that's also good um yeah if you like if you like a person liking their book recommendations you're pretty you're pretty likely to

like their book recommendations um well let's move on do you have other chats you want to show me well let me see what I have in here so I

need to bring you back yeah to screen the playground oh we did aelan yes it is again I was reading the

book on the Byzantine Empire and chry of stor is mentioned and I know I've known about him but it's Rusty knowledge to me so I

just thought I would ask about him and get more context right who was he and why was he important so you're kind of yeah go for it the answer seemed quite

good to me again it's obscure history yeah and it's working so you're kind of using it as almost like a second screen for

reading where exactly you're reading a primary source and you want more details or insight into it and and you're just you're turning to chat te for that

that's right yeah now another thing we'll use it for a bit this is typically with my wife we'll use it for the dog so we have a dog it was given to us

by our daughter who now has kids so we we haven't had the dog for long we're not dog experts so there's different questions about the dog like will the

dog get fleas or if the dog is limping should we worry or how much you know should the dog be eating or well everything about the dog yeah and that's

useful I I think it it must know a lot about dogs yeah I'm I'm sure it does um I I do that too I do that um not not NE I don't have a dog but for like basic health questions and stuff like that

sometimes it won't answer but a lot of times it gives me like a general sense which is nice um so so far what we've done is we've covered I think we've covered a decent

amount about how you read with it so reading with it as a second screen I think we've covered a little bit about how you write with it so doing a little bit of extra research on topics that you might not be as familiar with or sort of

getting the lay of the land on topics um I'm curious if there's anything else writing wise that you've started to use it for um or if that that's really the

extent of it that's most of it I think just as a pure writer anthropic is better but I don't use it to write anything I do I'm really very very careful not to do

that uh I just think it's a bad habit and it's in many cases unethical or even illegal uh but I don't do it don't want to do it but if I were doing it for

whatever reason I would use anthropic got it it sounds more natural and more human that makes sense yeah I I I find the same thing I think I think um Claude

is a bit better at capturing a specific tone um that you're going for if you if you need to use it for that um and then I'm curious if there's any way in which

you use it or don't use it in the classroom well I'm teaching a class right now we started last night and uh gp4 is on the reading list everyone in

the class is required to pay the $20 a month to subscribe to it the class is history of economic thought so it's mostly a class in obscure

history and I think it will work very well for the students but the class just started now I taught a class the year before to law students where I made them

all write one of their three papers using GPT not solely but you and GPT together figure out how to write the paper uh and that was very successful the papers

generally were good people likeed The Experience they learned like limits and scope of GPT yeah and uh that's how I've used it in the classroom also that's

interesting so it seems like you're you're leaning into it versus Banning it um oh of course and I told the students last night I was like look if you want

to quote unquote plagiarize with this thing your paper still won't be great so like don't do it and probably I'll know you did it I know enough about this

stuff but just figure out some way to use it wisely and in a smart Manner and you're going to learn things about it that I don't know and yeah just figure

it out is what I told them I said I'm not going to bust you for being on one side of some vague line or another just don't do something stupid you could ever get in trouble for and it should be fine

and I think that's going to work out I'm not worried that's really interesting have you found any any specific or interesting ways that students are using it that you wouldn't have predicted that are helpful

for them well foreign students especially from China yeah use it to improve their English and it has a big big effect yeah not just a marginal

effect uh and I don't think they even always know that maybe they're not supposed to or maybe it's allowed or maybe it's a gray area personally I

think it's fine universities deal with this in different ways but I've seen people go from like bottom 10% of writing skill to top 10%

overnight there's a risk right that becomes a crutch and you never learn a lot about English yeah I I've I've seen

the same thing um I think uh there are a lot a lot of business obviously is is conducted in written form and there are people that I work with or work for me

who uh speak great English but uh writing it is a little bit challenging and you can sort of tell but the minute chbt came out their writing turned perfect and it was so helpful you know

and um it made me think about because I I think that that's actually true in a lot of other areas where a lot of a lot of people on the people I'm talking about like they could write in English but it

just you could tell it wasn't totally right but I think that and and they just needed this like little subtle translation from their level of English

to like natural fluent English but I think that there's actually a lot of other of those kinds of translations that have to go on that gbt facilitates

so like a an example is uh my dad who who I brought up earlier he's not he's not a particularly like technical guy um but he's has an

idea for an app and usually when he does that when he has an idea he'll go to me and say hey I need you to write this up for me so I can hire someone to like do it

um but with GPT he can actually just like say his idea and it will write a product spec that looks like it's written by a product manager and that kind of like translation it's all in

English it's an English to English translation but it's it's a translation from you know his level of technical fluency to a product manager level

technical fluency is like super helpful and important because previously he wouldn't have necessarily been able to do that um and I think people sort of Miss that when they think about gbt's

translation capabilities context is that which is scarce as I like to say yeah um

so okay so we we've gone through I I think the I think the classroom stuff is is really fascinating before this

episode we um we I tweeted about the fact that you're coming on the show and a bunch of people ask questions so I want to do a little bit of a rapid fire

question asking of uh Twitter questions if that's okay sure cool so Henrik asks in the least possible number of Dimensions if chat gbt stopped existing

today and other llms how would that affect your productivity would it be less posts on marginal Revolution worse interview questions something

else worse interview questions I would just no less I don't even know how to measure my productivity right maybe it wouldn't go down I would feel much less

smart however maybe the stupider version of me is more productive but M much less smart I think is a really important interesting thing

to say but the returns to being smart in labor markets they're not as high as you might think that's why there's a lot of uncertainty in in my tone here so it may

just be private consumption being smart I don't think say it would get me more people listening to the podcast May get me fewer right

interesting what do you think of as the most underrated or overrated use cases for chubbt uh and James asked that question I don't have a good sense of

that I'm not even sure like what is rated how I think the problems of hallucination are somewhat overrated uh that I I do have a sense of

things like programming I don't do uh it just seems to me almost everyone under uses it for most things yeah that makes

sense um uh Daniel asked which I think is relevant how do you manage to switch between reading books and using chbt to interact with the content while still maintaining your reading

flow well it slows you down quite a bit yeah but you're still reading with GP you're just reading on a different screen MH uh but you should do it in areas where you don't know a lot and

really do need to learn something not just do it all the time yeah yeah that makes sense um at this is gray asks

would you use AIS to speak for you online oh I'm not sure I understand the question um let's say like would you use

an AI at some point there will be text to speech and vision models that are good enough that you can have a little clone of yourself that might go on a podcast I think is is the is the thrust of the question would you would you

allow a clone of yourself to go on a podcast I'm not sure the choice will be up to me I'd like to see what it's like I will say this you know I published a

book online this last year called goat who is the greatest Economist of all time and the audiobook version will be read uh by an AI and it remarkably like

me and I'm happy about that not just fine with it happy about it the other stuff I'm not sure yeah why are you happy about it reading your entire book out loud

isn't a very costly and taxing exercise and this means I don't have to do it that makes sense that makes sense um

okay so we are rounding toward the final part of the interview uh which is uh it's appropo of this of the previous question which is about cloning

yourself because I actually created a little bit of a clone of you um and uh uh in this I want to do a segment with

you um called the Tyler test which sort of play on the Turing test where I ask you a question um and I'll have you answer it and then I will also ask the

Clone that I made um the same question we'll see if it answers the question in the way that you would and we'll see how good it is um and just for a little background the way that I made this is

you can make a custom version of chat gbt you can give it a personality I think one of the things that makes it likely to work for you is you've written so consistently for so many years that

GPT has probably ingested a lot of your writing and we'll see how it does on on some of some of the questions that I have for you and uh and one important thing is like I I there's no web

browsing and I didn't upload any books or anything like that so it's only what's actually just in GPT what it knows about you itself one thing I would know you know my online book Jeff Holmes

and I created you know a a rag based GPT bot to answer questions about the book yeah it's not claiming to be Me overall I think it's pretty good it's not perfect it'll get better we want to

upgrade it but I'm happy with it and glad I did it that's cool I would love to link to that um so definitely send that to me afterwards um that's e on go.

a. perfect so we are going to get started with this [Music]

segment it's called Tyler bot welcome my first question for you what are the core lessons of Economics the first would be incentives matter the second would be there's always an opportunity cost those

would be the two most important principles of economic science and trying to apply them consistently would be the first Mark of a good Economist so we we're asking Tyler bot

we said what are the core lessons of Economics incentives matter there's no such thing as a free lunch which I I have heard you say before yeah that's the same as

opportunity cost you know I would give it that answer is correct okay interesting so it's going Beyond two yeah the answer is very good very

good how would you rate it out of 10 I'm not sure what the scale is but I'd give it an a say Okay um so that's so that's number

one that's our first question um second question and I got this from uh from your most too much now like I get that it feels it ought to give more

but there's something about economics where often less is more I'm not going to take back the a uh but a good answer to that question is actually pretty

short let's see if you had to make it shorter how would you do it make it short well I would just pick the first few yeah well let's see if it knows to do that because that's that's what you would

do okay so the first two are now the same now we're uh and the third one is correct so it's doing this well and now it's just now it's just summarizing it in bullet points instead of uh so it's g

off the rails it's not quite it's not quite you but it's on track yeah I'd give it good Marks here okay okay here's here's my next question which I I took

from your book um Talent what are 10 words your spouse would use to describe you curious works

hard loves to travel I'm not sure I should mention them all let's see maybe we can do five so you're you're you're you're got

four good father nice and should walk the dog more it's not gonna get that last one

okay what are 10 words your spouse oh I say what are five words your spouse would use to describe you

intellectual curious curious we got curious analytical Ecentric those are good answers they're not worse than my answers you know you can't expect it to

know about the dog right that's why we have to ask Chad GPT about the dog a year from now we'll know everything about the dog well you know I

like that it got it got curious that's that's a that's that's the main overlap um but it it missed the travel and good father and it missed the dog but maybe

gbt 5 maybe gbt 5 um maybe Tyler five right what what you say it's Error or my error don't assume the errors is in the gbt that's true we can't assume

that okay last question um in your view what is the most underappreciated way in which AI particularly tools like chat BT will

fundamentally alter our understanding of EP economics and human behavior in the next decade I don't think in the next decade it will fundamentally alter our

understanding of Economics I think at some more distant point it will be possible to simulate small economies using GPT like methods and we might learn a lot from

those simulations uh those seem distant with with data collection as the constraint not not the constraint on the AI side per se so I

don't think it will matter for quite a while okay well let's see uh let's see if uh GPT Tyler GP Tyler uh shares your uh shares your

view huh do you agree with this it's saying I think it will change radically transform our conception of information

costs and decision-making processes those might be true claims but I took you to be asking how will it change economic science and research and

it's answering how will it change economies yeah I think I I I more meant it as how it will change the economy um but I think the I think the answer of

how it will change economics is a good one it doesn't sound like me this answer I don't think they're bad answers but it doesn't seem like a Tyler bod it seems

like it couldn't find what Tyler might think and it came up with a a pretty good generic answer okay all right so so that's what what letter grade would you give that

one well for accuracy it's doing fine for Tyler I'd give it a c minus Okay C minus so we got an A and the middle one was yeah we had two excellent ones and

then one that a perfectly fine answer but has no Tyler in it there you go well that's that's the story um uh Before I Let You Go anything that you wanted to cover or talk about that uh that we

didn't get to well I would just uh you know recommend to people that they read my book eong goat. a they can

just put it on their Kindle print it out read it on screen but it's also embedded in an AI in the form of a chatbot and I made a deliberate decision to publish a

whole book quote unquote in gp4 and we're working on a claw and also Gemini versions of it wow and I think this is the future for at least some

book publishing like why not interact with the thing it's an option you don't have to do it uh and I'm very excited to have been able to do that what have you learned or what have

you found so far having having done that that you didn't know before well many people people email me or on Twitter they say what their questions are for the thing and you learn what your

readers really care about and it's not what you want them to care about and that's part of the experiment so maybe books of the future will in fact use this data and give readers what they

want to what they really want to care about I saw a question on Twitter this morning someone said they asked the chatbot what would Tyler think that Milton fredman would say about IR

Rand which is not really a question about economics now the answer was good it was both a good answer and a good Tyler answer but you just learn people want

something really quite subjective out of even economics books and I think that will shape how we write and think over the next few decades that's fascinating I love it um well I think that's a

that's a great place to leave it thank you so much for doing this it was super fun to have you my pleasure thank you Dan