On the dangers of stochastic parrots: Can language models be too big? 🦜

hello everyone welcome uh we're really excited today to have a wonderful uh set of people to come and talk about a very important uh set of topics um i'll just briefly

describe the format um to make sure everyone knows what's going on we're gonna open up with emily bender uh who has graciously uh joined us to talk about her paper uh

that co-authored with that with several other people and i think she'll she'll talk about how it's actually written with with more than the people who are officially recognized on the paper and so that that topic she'll

cover briefly and then she'll she'll tell us a bit about the content of the paper which is really interesting about the nature of language models and uh even though they're exciting why we need to be

concerned about some of the limitations and potential risks that are involved in them so emily's going to speak for around 30 minutes or so and then we have some very distinguished panelists who are going to respond just

for a few minutes each uh we have zach kenton from deepmind and cope snake from cambridge university and we very much hope that angelina zamda will be able to join us

she's the ai and human rights team lead at the turing institute so they'll respond then emily will have a chance to respond then we'll get into a kind of

panel discussion around related topics and there will be time at the end for people to ask some questions um to enable that everyone has graciously said that they can continue until quarter past five

uk time so we're going for about an hour and a quarter hope that's good for everyone everyone's also very kindly agreed to that we can record this so if you need to run off that you will be able to find a recording of this fairly soon

and just so you know if you want to ask a question the way this this will work is is you should see q a at the bottom of zoom if you press that you can type a question in there or you can also look through the

questions that are there already and you can upvote questions so we're likely to go with respectful questions please that are near the top of that of that

table uh wonderful so uh let's let's uh let's get ready to start um so again delighted to have uh professor emily bender with us today

she's professor uh of um language and natural language processing from university of washington and she's going to tell us about her her paper which has attracted some attention

on the dangers of stochastic parrots thanks very much emily thank you so much for having me and i hope you don't mind that i have assistance here for my cat oiler um hopefully she will not be too

disruptive welcome everyone i'm very glad to see the interest um in the content of this paper um and i want to let you know that this link for the slides will be on this slide

the second slide and the final slide and the reason i put that up is so that you have the slides but also because there's some bibliographic information in the pdf there so as noted i am presenting on the

dangers of stochastic parrots can language models be too big and that parrot emoji is officially part of the title this paper is joint work with many people so the

listed authors on the final paper are timny gabriel angelina mcmillan major and margaret mitchell because um well it's a long story that i'm sure you've seen recovered in the press

but the main thing that i want to say is that we also had the valuable assistance of actually margaret mitchell and vinay kumar prabrakaran mark diaz and ben hutchinson who are all um

well the final three of them are still researchers in the ethical ai team at google and it is only because we had this group large group with diverse scholarly backgrounds that we were able to pull

together this paper that drew on so many literatures and i think it's really unfortunate um that some of these researchers were required by their employer to remove their names from this paper um and therefore not get the recognition

for their work and so i just wanted to take a moment and to promote a few papers um by these colleagues of mine and these are all listed now that is distracting oiler

these are all listed in the bibliography and the slides that i mentioned so if you would like to see sort of more written from that same place of expertise as they've contributed to this paper i encourage you to go check out

these other papers listed here um so in writing this paper we started from the position that uh we were observing the field both in academia but especially in industry

in this like rush to ever larger language models um and um in discussions so the the initial discussion was between me and and to meet um

sort of asked this is exciting there's interesting things happening here but what should we pause and consider um before we dive into this or continue to dive into this

um and so the questions that we wanted to ask are is this track of ever larger language models inevitable or even necessary what are the costs associated with this research direction and what should we

consider before pursuing it um in fact does the field of natural language processing or the public that it serves actually need larger language models if so how can we pursue this research direction while mitigating its

associated risks and if not what do we need instead so these are the the guiding questions for this paper and we would like you to consider them as well what i'm going to do here is give a very

brief history of language models and then talk some about the risks that we identified and some risk mitigation strategies that we also cover in the paper and the risks in particular fall into four categories

and we talk about the environmental and financial costs of this path and we talk about what happens when you're dealing with unmanageably large trading data a brief comment on the impact on research trajectories

and then some thoughts about what happens when you have synthetic language that is um strings put together by artificial agents that are recognized by humans as human

language so the brief history um i and my co-authors define language model as a system trained to do string prediction um and this can be

linear so what word comes and then that blank the system will be trained to fill in that blank um or in the masked language model setup what word mask here how do you fill in that masked token in the middle

and this is not in the very broadest sense new technology it was actually proposed originally by claude shannon in 1949 and then implemented very importantly for speech recognition and machine

translation in the early 1980s um so that's the earliest ones were engram language models then we start seeing different neural architectures up

through the transformers and the big takeaways um that i would like you to take from this history is that uh there's this pattern of getting

better scores on various benchmarks through larger data collections and bigger models and then you sort of max out what can be done with that particular approach and then move on to a new architecture and so on

that's the recurring pattern we are now in a situation where not only are language models monolingual but in fact there's multilingual training setups where data from many languages are combined

together people are working on how to reduce model size for example by distilling out smaller language models from the large ones

and very importantly the growth of the models in terms of both the number of parameters um and the size of the training data has been proportionate to an extension

in the range of the application of those models so originally it was just things like speech recognition and machine translation to do with re-ranking possible outputs from another component say well what looks like a plausible

sentence in this language and now we see language models as components of just about any language processing task um so how big is big um this is a graph

that is probably a little bit hard to see but i just want you to get the overall sense of the trajectory from um if you notice the time scale on the bottom this is

late 2018 through early 2021 looking at the increase in size in orange of the data set measured

in gigabytes and then the number of parameters those are the blue circles um and those are which is measured in parameters um and uh one thing that i want to mention about this is that this graph is

certainly already out of date when we were producing the camera ready version of this paper in january of this year um it's angie who was working on this section and i said hey did you get that latest model from

google she said yeah and i went and looked at the table and she had g shard i'm like no no the latest latest model from google which is switch c up here and so even in the space between when we submitted the

paper in october and uh when it was uh finalized for camera ready in january there was already this enormous increase in size so this is

this is a continuing trajectory all right so we are you know on this roller coaster um is it really scary well what are the risks the first

um series of issues that we looked at are environmental costs and financial and accessibility and here we drew on a growing literature looking at and the environmental and

financial costs of machine learning in general and language processing in particular including work like that by emma strubel where um they found so in comparison the

average humans emissions um per year are five tons of uh carbon dioxide and that is averaged across the entire globe so those of us who live in industrialized societies are

responsible for much more than that but five being the average for a human and in that context drew but i looked at um what the cost is of

training particular different models so they said based on some estimates doing some of the training steps for a transformer model and then projecting that out to if

you've done the full grid search using gpus that would be 284 tons they also looked back at some work that they had done on english to german machine translation

where they gained a whole tenth of a blue score um and they found that that uh required about 150 000 worth of compute um and

so this is a lot it's not you know catastrophically any one of these a lot but it's big enough that's worth paying attention and saying hey do we need this before flipping the switch and it's really valuable i think that folks like

strubot all have sat down and done this work because when we work on computers it often feels like it only exists in the abstract world of math and doesn't have any physicality to it

and that physicality is real and it's important and it's big enough to have an effect on things like the climate in aggregate um so strew but i'll say we can as a field

do better about this by getting in the habit of doing things like reporting training time and sensitivity to hyper parameters um and making sure that these are that doing things efficiently is also something

that is valued alongside just getting higher scores or lower scores depending on your metric um esther but also suggest more equitable access to compute clouds through government investment

because alongside the environmental cost there's this question of who actually can have access and who can play in the space and who is left out and some of that is which researchers but also which

languages um so current mitigation efforts um a lot of folks are laudably looking to renewable energy sources um to make sure that if they're going to be using a lot of electricity

it is done um off of a renewable source that's great but even renewable sources do have environmental costs um here in washington state we are very smug to use a lot of

hydroelectric power um which is great from a carbon footprint point of view um not so great for the salmon that need those rivers too and also up until the point where we actually have

enough clean electricity to do everything we want to do anytime we're using electricity um even renewable electricity for something that is not a survival necessity we might stop and think hey is there

somewhere else that this could be used better um another direction that i think is very promising um is there are efforts now towards prioritizing computationally efficient

hardware and computationally efficient approaches and i'd like to point out the sustained nlp workshop which is happening again this year at emnop um

and work like shorts at all 2020 that are talking about actually promoting efficiency as an evaluation metric and then finally there are existing emerging tools for documenting energy

and carbon usage alongside other aspects that we report of our research so these are all worthwhile things to be looking into another angle that we think is really

important to take when we are doing these risk benefit analyses is to think about who's actually getting the benefits and who's paying the costs and what we notice about large language models is that they are mostly

for high resource languages because they require these enormous amounts of data and therefore they are going to benefit those who are speakers of those high resource languages and that is disproportionately people who already

have the most in society on the flip side if we think about climate change and who's going to be impacted the most and the quickest that would be marginalized communities and so we picked a couple of examples here i'm sure there are more

and the maldives are threatened by rising sea levels um recently 800 000 residents of sudan were affected by flooding um and to the extent that the work we're doing to build large language models

more and more and more larger and larger language models for english these folks are not reaping the benefits and we aren't building large language models for devehi or sudanese arabic all right so that's the environmental

and financial costs the next set of risks has to do with unmanageable training data now the internet is a large and diverse virtual space and so it's easy to imagine that very

large data sets such as common crawl must therefore be broadly representative of the ways in which different people view the world however on closer examination we find that there are actually several factors

which narrow that internet participation which narrow the discussions which will be included via the crawling methodology and finally which narrow the text likely to be contained after the crawl data are filtered and

critically in all of these cases we find that it's the voices of people who are most likely to hue to a hegemonic viewpoint that are the ones that are most likely to be retained so in a bit more detail

if you look first if you think okay well we're going to crawl the web that's the whole web that's a lot well who's actually on the web who has access to the internet and is contributing and that tends to be younger people and those from developed countries

who's being subject to moderation so who's being pushed out even if they're on the web who's having a harder time contributing their voice and fighting for a place in the conversation there's some really interesting research

by dr leslie k jones looking at how on twitter um accounts belonging to people who received death threats are actually more likely to be suspended in many cases

than those issuing the threats and so we get marginalized voices disproportionately pushed out even if they've tried to participate in the first place and then we say okay well what parts of the internet are being scraped

for a lot of these projects people are starting with reddit not the text of reddit but the web pages that are pointed to from reddit and uh in the us anyway um reddit users

are 67 men and 64 of them are in the ages of 18 to 29. um similarly wikipedia is a strong um search but a survey of wikipedians found

that only a small minority of them are women or girls and where people have gone to lengths from you know various less represented communities to create online spaces for themselves

those sites are less well connected to these starting points for the scraping lazar at all do some interesting documentation of a community of bloggers of older folks looking at issues of

ageism and aging and those kinds of blogs are not going to be as well connected and not as well represented in the crawls and then finally there's this filtering step

which is done surprisingly starting with basically a list of keywords that initially came from someone's open source project that had to do with making sure you didn't get alarming

things popping up as um search completion suggestions in like a music site or something um so it's this list of 400 or so words um that primarily have to do with words

referencing sex but also included a bunch of things around lgbtq identity and so this filter then filters out a bunch of lgbtq online spaces and with thanks to willy agner for

pointing this out you're therefore going to lose the text of people positively describing their own lgbtq identities which is only going to therefore magnify

anti-lgbtq discourse that is still contained so why is this a problem the problem is that if we over

represent hegemonic viewpoints um then we are over representing the language of people who consciously and not deliberately or not use their language in ways that are consistent with systems of oppression

and express those systems of oppression and by that i mean things like racism misogyny ableism ages and transphobia and others another risk has to do with the fact

that as we are working to push back against systems of oppression our language changes right so language models that are trained on old data run the risk of what we call value lock reifying older less

inclusive understandings and an example for this um comes from tremendous work on the black lives matter movement where they found that in the wake of the blm movement there was an increased number of

articles about shootings of black people on wikipedia and furthermore past events were documented and updated and all of that was done with less of a

white gaze and so if you had your language model trained on stuff prior to the black lives matter movement you would not therefore have this updated more inclusive view of the world

included in the language model it's worth noting in this case that um there's plenty of things that happen um that would be valuable to include in a better sort of viewpoint of the world

but don't get covered in the media because they are less dramatic and so there's an additional bias that creeps in that way um so given that the data that language models are

trained on over represents hegemonic views the language models are therefore going to encode those views um one possibility is to think about retraining or fine-tuning to try to mitigate that and there's a

really interesting initial proof of concept that's come out from open ai by saliman and denison where they find that they can reduce although not eliminate

um by doing some fine-tuning um with some very carefully carefully curated data um representing uh sort of the viewpoints they would like to see given a particular social

context um i say it's a partial proof of concept um because they were able to show a reduction in toxicity and also some human evaluation found that the text was better in that

sense but that evaluation wasn't grounded in what kind of harm would this be doing in the world and what we need to do to make it safe but definitely interesting proof of

concept um and it absolutely turns on careful curation of that fine-tuning data and then finally this point about value lock and how language models reify older less inclusive understanding um is echoed in a recent wonderful paper

by birhani at all looking at the values of machine learning research um where they note that machine learning applied as prediction is inherently conservative in the sense of reproducing existing prior patterns

all right another set of risks has to do with bias so if you've got that overly hegemonic training data that is going to encode systems of oppression and biases

um and sure enough we find that if you go looking for it in the lms it's there um so there's a wealth of examples now um i recommend blog.2020 for a critical overview

of the work doing this probing i think they looked at like 149 papers examining um bias in language models and documentation of the problem is a really important first step but not in

itself a solution and we note in our paper that the automated processing steps that are used as part of that biased documentation can themselves sometimes be unreliable and there's work by martin

sapp and others who find that identity statements tend to come across as toxic if someone just says for example i am a black woman because there are so many uses of the

phrase black woman in toxic context that simple identity statement itself gets rated by automatic processing systems as toxic and so this work has to be done very carefully furthermore if we're going to

be probing for bias as a prerequisite for doing something about it um we have to know what biases we're looking for and what social categories um are the marginalized ones and that

absolutely requires local input for the context of deployment of the technology and one of the interesting things about high resource languages is english in particular is that it is used across many many

different societies in many social contexts and so one person trying to fine-tune a system um can't fine-tune it or one team in one location can't fine-tune it for all of the possible deployment contexts in

english and salim and denison are very clear about that and i appreciate that um so one question that we frequently get asked given the title of this paper the subtitle is okay how big is too big why where should we stop

um and the answer is not a specific number um but rather a question of documentation so our recommendation is at the beginning of a project where you are collecting a data set to budget for documentation

and only collect as much data as can be documented because that documentation so this is along the lines of data sheets or model cards or data statements um can allow people to understand

sources of bias and develop potential mitigating strategies and if there is no such documentation if we don't know what's in the data then we have the potential for harm without recourse and we refer to this as documentation debt

which is the case where data sets are both undocumented and in fact too big to document post hoc you have this debt that you're carrying and there's risks embedded in there and you can't see very well what they are because you don't know what's in the

data all right the next category of risks has to do with research trajectories um the last few years have been very exciting in some ways in nlp as we've seen you know the state-of-the-art

scores on various leaderboards just routinely rapidly broken particularly in tasks that are meant to test for natural language understanding but there's also been work showing that

when language models are doing really well in these tasks they're actually just picking up on spurious data set artifacts and furthermore i've argued elsewhere with alexander kohler that if you're going to train a system only on linguistic form

the meaning is actually not in that training data and so if the language model can apparently win at some natural language understanding task it's not

because it doesn't have access to meaning and so if we've done that if we've managed to basically bulldoze all these various nlu leaderboard tasks using language models are we actually learning something about machine

language understanding or are we just going down a rabbit hole um so i will have more to say about research time as a valuable resource on the later

slide that's what i was looking for um finally i want to talk about the potential harms of synthetic language in the sense that i introduced at the beginning and at this point i get to

tell you what the phrase stochastic parrots actually means from linguistics and psychology we learn about how human human interaction is co-constructed

so when i am speaking to you um i am doing my best to model your understanding of what's going on and craft what i'm saying to match that obviously quite difficult um in this context i can only see the size of the

audience and make guesses as to who you are um but conversely you have your guesses as to who i am and what i'm trying to do here and so forth so therefore in that together we are

starting to construct a shared model of the world obviously in face-to-face one-on-one conversation this happens in a much more tightly integrated seamless fashion in contrast a language model is a system for

haphazardly stitching together linguistic forms from vast training data without any reference to meaning without any reference to communicative intent or co-constructing a model of the world

and so we call it a stochastic parrot nonetheless as humans whenever we encounter speech or text or sign in a language that we are proficient in

we can't help but make sense of it so when we encounter synthetic text stitched together by one of these stochastic parrots we still make sense of it in other words coherence is in the eye of the beholder

so what follows from that well in the first place when you come across as a speaker let's say of english some hate speech or other more subtle denigration

or stereotyping language in english you can't help but interpret it and if that denigration is integration of an identity category that you hold then you are going to have the emotional

response to that perfectly naturally similarly if it is not hate speech about you um but it might reinforce ideas that you hold that raises the risk that you're going to enact harm on bystanders who hold that

identity category so that's one potential harm to do with synthetic text another is the possibility that people can deliberately use these systems to create synthetic

text to do harm in the world and then mcguffie and newhouse produced a study about how i think it was actually gpt2 could be used um to boost extremist recruiting maybe gpt3 anyway somewhere in there

not the latest and greatest language models by now um but good enough to create text to populate message boards where people would feel less alone in starting to adopt extremist ideology

another thing language models do is they make the output of something like machine translation sound more fluent and when it sounds more fluent um we are more likely to think that it's actually accurate

and therefore we can attribute the wrong meaning to the initial human author in the other language so all of those harms have to do with humans encountering text

generated by computer without commitment to its actual veracity or appropriateness in context and interpreting it as we do because we're humans and it's our language and there are other harms um so language

models can be probed to replicate training data for personally identifiable information this was shown by carlini at all last year and language models can also function as hidden components

influencing things like query expansion and results and so all of those biases that have been picked up by the language model can then be used um to be replicated for example under the hood in a search

engine where this seemingly objective computer gives you back denigrating answers such as noble 2018 documented for google searches um in the last

decade around identity categories so many different potential harms that are all worth thinking about so what do we do about this well some risk management strategies

i mean here's where i said i would come back to this notion of um research time being valuable because our research time is valuable we should allocate it carefully and that means doing things like incorporating energy and compute

efficiency and planning and model evaluation so that as we're spending our time choosing how to do our research we are doing it in ways that are also minimizing environment environmental impact and maybe actually

maximizing our ability to do this work in an environmentally friendly way and we should select our data sets intentionally rather than just grabbing everything that's accessible

and here we're very inspired by this quote from berhany and prabhu who are in turn inspired by ruha benjamin feeding ai systems on the world's beauty ugliness and cruelty but expecting them

to reflect only the beauty is a fantasy and so what do we do instead of just grabbing everything we think about what we need for our use case and we build our data sets intentionally as we're doing that we should be

documenting the process that we followed the data that we collected our motivations for doing so and thinking about potential users and other stakeholders so that we can then do informed analyses um

along a variety of well-established practices so one is called pre-mortem analyses and that's where you consider what are some possible worst cases and how would that come about and then

what can i do to head that off and there's a bunch of other techniques from the school of thought called value sensitive design that involves identifying stakeholders and thinking about

what are their values and how does this technology interact with them hopefully to support their values in that light we um challenge ourselves to say okay

we are asking people to make a change we're asking people to slow down and possibly back off from this large language model pathway are there risks of that change what about the benefits of large language models like for example improved auto

captioning um which is certainly a useful thing in many contexts um so to that we ask are large language models in fact the only way to get these benefits and what about cases where you're working with a lower resource language

or a time and processing constrained application and what i found looking briefly into the speech processing literature is that actually a lot of the work is looking at lower resource

approaches precisely because you don't always have the time to do all the processing that the large setups would require um another question to ask is are there

other ways that if we are going to use large language models that the risks have been medicated so that we could still use them and use them more safely and so for example things like watermarking synthetic text so that we are in a position of people

to know when they are encountering something generated by a human rather than a generated by a human or generated by a machine and then finally i think it's worth being part of a policy

discussion about how to do effective regulation in the development and use of large language models and i say being part of that discussion because effective policy requires people who understand the technology

people who understand the ways in which communities are affected by that technology and people who understand how to write effective policy um and i certainly um am not a policy maker

myself um and i think many people on this call might also not be if you are wonderful thank you welcome let's be in conversation so i'm going to end with the same set of questions i started with

are ever larger language models inevitable or necessary what costs are associated with this research direction and what should we consider before pursuing it do the field of natural language processing or the public that it serves

in fact need larger language models if so how can we pursue this research direction while mitigating his associated risks and if not what do we need instead and there is the link for the slides once again if

you didn't get at the beginning and wish you had and um that is the end of this presentation thank you so much for listening fantastic thanks so much emily there was a great talk

highlighting a lot of really important things for us to think about with respect to language models um so we're going to come back to emily soon but first we're going to give each of our distinguished panelists just

uh two to three minutes to to give some quick thoughts on what we've heard and first up we have zach kenton who's a research scientist in ai safety at deepmind mainly interested in specification and

alignment zach yeah thanks uh and and thanks about emily for the brilliant presentation um on your very important work um i really love the appearances of the parrots uh throughout

and um i think the you had a very broad scope in your investigation which i really liked um yeah i wanted to echo your point um about selecting data sets intentionally um i think

that's a really important point um and also you know something that appears in my work is um trying to have humans in the loop so that uh they're there to try and uh

kind of sense check or be there as a safety guard throughout training um yeah i wanted to pick up on a couple of points um

and try and like relate it to some of the work that i've been doing recently so um i think um i have a lot of kind of complimentary um views

to to to to to what you had in this paper and um i think in my most recent work i've been looking more at kind of the first order or like direct arms

um rather than these kind of uh secondary effects you know such as environmental and financial costs and i do think those are really important but it hasn't like been the main focus of my work but um

i think areas of overlap and things i'd love to discuss with you are things around kind of what we can do about trying to avoid harmful outputs such as bias and discrimination

stereotypes but also some other bits like um how do we stop um these models from lying to us or deceiving us or manipulating us um and and so yeah

i'm kind of interested in in those kind of threats and uh ways to detect them and to mitigate them um and you know i'm especially trying to

come from this framing of you know it it's not like the designers of these systems are bad people or anything it's more that they kind of make mistakes or accidentally misspecify what they want from the system and

then you know the resulting uh threats kind of happen um after that um yeah and uh i just wanted to pick up on

on one point in terms of the title and and kind of thinking about is is bigger or always better and for me this this isn't necessarily the crux it's more about the competencies

of the of the system rather than you know how many parameters or how large was the data set that it was trained on and with those competencies come different kinds of threats and we need to understand

how competent our systems are and and whether those competencies are in line with what we want our systems to do and so yeah i'll i'll finish there and would love to discuss with you further on on lots of

those points and any others that come up so thanks fantastic point zach i'm sure we'll get into important questions of how you can try to figure out competence and you know

the issue of lying and manipulation is i think a super interesting one um so next we have anjali mazander who is the theme lead on ai and justice and human rights at the turing angelique we'd love to hear your

thoughts thank you adrian um and thank you emily for a great presentation um and and zach for your um comments uh

i guess i really wanted to echo and sort of um empathize on the um risk strategies from my own perspective and an approach that i have

tried to take in my own work and in particular thinking about things from a pre-mortem perspective and how we can really sort of

interrogate and ask questions beforehand to think about sort of downstream um consequences and also just understanding who's interacting

with the system so what the data is based on and then who's interacting with the system which may not be the same group of people and

i actually have more questions and probably answers and one of them is um i hadn't really thought about things from the

uh financial or environmental perspective either um i've really thought about questions from an equality and human rights

perspective and inclusion uh as part of that uh but one of the things that i think we face sometimes as researchers is that there's a push on

sort of an out this output whatever however you want to define that um how would you sort of suggest we incentivize

to consider um all of these questions and issues right at the forefront um and the other question probably comes at the

more policy regulate regulatory perspective but the the other um area of work for me

has been when um scientific evidence goes to court and the results of algorithms which we you know we've seen them in some instances they're increasingly probably

going to happen so who's responsible for that um there's researchers who develop methods and models but they're then tools that are based on

that research they're not necessarily the same people the data set in terms of who's curated it and these are all questions that i'm

thinking about and just wondering um what your thoughts are on that thanks so much anjali really great questions and our final panelist is professor anne cope stake from

university of cambridge and is the head of the computer science department and an expert on computational models of human language with i believe a particular interest in semantic competence

and um so thank you so much for the well for the paper and um also for giving this talk of course there's so many things in the paper and

there's so many angles that one could could go into i want to perhaps ask one question which is whether we should really

be careful about distinguishing the uses that language models are being put to because i was looking at the closed captioning that the transcription that was going on

and the way that that has improved over the years is absolutely fantastic um i mean the one place where i noticed switching it on and off but the one place i noticed it got it wrong

was when you said hue to a hegemonic viewpoint um which it clearly hasn't heard very much of but i mean but this is a this is really a massive advance and massively helpful

but that type of thing is very different in i think from claiming that the language models are capturing meaning and i think

there's different ways we could draw some of these things out one is that actually it's really quite easy potentially to evaluate on transcription

we can tell whether it said something that's right or not whereas with many of these so-called meaning-based tasks as people are discovering it's not at all easy the second thing is that i think

that there's a set of tasks which are to do with genuine language understanding where actually much smaller language models

will work very well and that's partly based on research that we're doing in my group which is what the um paper with well not my group actually

um people that i'm associated with are doing some people in cambridge other people like orally who are not in cambridge anymore or at least in toronto and she's the main author of that paper that i put the link

to and there's lots of things that we you know in that paper but one thing that i think is really important is that a much smaller language model

trained on potentially very highly curated data could with more structure in it can actually be on some of these more

these tasks facing more natural language understanding could actually do very well so i don't want us to end up as a field in the sort of situation where we go down the um

stochastic parrot route for language understanding tasks partly because i don't think it will work very well um so i want to sort of

you know raise the question it also relates to something that zach said about competency about whether we should be very careful about talking about the different tasks that language models are doing and distinguishing say from transcription

which we can evaluate and which crucially people do not think that there's a human behind the transcription i assume i mean if you see this thick stuff coming up and it's almost perfect but

the way it changes is very cool you don't think of that you don't model that as a human whereas when you've got something like alexa or siri or something then i think you are forced

to model that as somewhat human-like and i think that that is very dangerous potentially for all sorts of reasons including all the ones you're bringing in with the language models

so i wonder whether we should actually be talking about that and the differences between those types of tasks a bit more um yeah and there's huge numbers of other things i

could survey but i do want to recommend that people read your um paper with alexander collar as well as the stochastic parrot paper because i found that very um enlightening and very amusing as well

thanks brilliant lots of food for thought and emily you've been listening patiently to all that um are you happy to to respond to pick bits that you'd like to respond to yeah i mean i i would i would love to

have hours to chat with this group of people because um i really appreciate these perspectives and there's a lot to say um but one sort of theme that i've noticed or that i that i want to pull out of this

is is i'm building on what am just saying um specificity right so um zach talks about competencies um and anjali talks about um who's going to be interacting with the system and

who's building it and ann talks about well let's think about particular tasks and i think that that is one of the problems with scale that we have seen with the large language models right so it's

more parameters more processing time more energy more data but also this sort of push to generality that we're supposed to use it for everything and it's one size fits all and that makes it much harder to answer these questions

right so how do we test what it's actually doing we need some specificity we need particular tasks that are understandable and where we can say you know if the system has done well on

this we know why um or we at least you know we have reason to believe it it isn't just manipulating forms to get something that looks like the right answer um and um if we're talking about

i mean if you're creating policy policy has to be general right but i think grounding these discussions of um who's responsible for what parts of the system

in terms of um specific types of tasks and systems will help get us to a point where we can build general policy um and i um and then around this is i think anjali

again talking about how do we incentivize research that engages with these issues rather than you know the current system which is you know if if you've

um hit the state of the art on some leaderboard then there's a sort of sense that you deserve a publication out of that um which really does not make for very very many durable

contributions right um something that i picked up somewhere and i wish i could find the credit for was um the pithy statement that um if you're worried about getting scooped you're probably not asking a

very interesting question right if it's something that lots of other people could be doing at the same time why are you putting your work there right and so i encourage people that i'm working with to think about okay what perspective do i have what can i

individually really contribute here um based on you know what expertise i bring based on the milieu that i find myself in um that is going to contribute to the conversation

rather than just being part of that race to the to the top only to get you know passed over to the next time in the next lap around that race um so how do we incentivize that um i

think a lot of that has to do academically in terms of our reviewing processes and sort of what we value as reviewers

and i see a lot of people who speak as though they would like to see this change um and i see a lot of review forms that start asking these questions but there's still this really um

difficult bass line to get around which is that well if soda then publish like then it deserves publication so i think we need to really raise the bar on what counts as publishable but that is not going to solve everything because this isn't just

happening in an academic context and so we have you know all of the stuff where there is um corporate incentives um and to this i want to come around to

zach's point about accidental misspecification um which i think comes from a place of saying it's not that researchers are trying to build evil things

right um but framing it in terms of accidental misspecification i think takes some of the um the blame in effect away from

the corporate profit incentives that are really driving a lot of this um and so when we're looking at things that end up misspecified first of all i think it really helps to get specific about the task you know what is it that we're

that we are specifying in particular um and then look at what are the um what are the incentives that the person doing that specification had to answer to right and it isn't always just sort of this pure well i'm doing research and so

i want to make an interesting thing it's oftentimes in the context of um you know what is valued by the corporation where the person works or the funder you know if the person's in academia they might

still be influenced by their funding source and so looking at those systems levels issues i think is important too if we want to get to something that is overall safer and i feel like i have not begun to do justice to the interesting

points that you all raised but i also think that it might be more interesting to switch over to a discussion mode where it's not just me talking so i think i'll end there for now i mean there was a there was a brilliant

uh way to try to pull together a whole load of things that were brought up i just wonder if i can press on on the topic related to that last point so um as angely brought up and you were just discussing

there's a there's a challenge in academia to try to motivate and incentivize people to produce certain kinds of work and not just try to be a top of the leaderboard and it's it's difficult enough to try to address that but let's suppose you even

could address that even so you've got big tech firms that are deploying these models and um they're subject to different kinds of incentives where

it's it's very tempting for them to build a big model they they feel that it's easier perhaps for them to to do interesting things with in some ways i wonder if anyone wants to respond to

the the institutional factors that might lead a big company towards nevertheless going in this be well being pulled in this direction without any

notion of ill will and how might we how might regulators or they genuinely try to monitor how dangerous that is and what

could be done about it does anyone want to want to take that anjali anjali then zach um i'll start with um i don't have something

coherent necessarily but i wonder whether or not it's drawing on um sort of the social corporate responsibility and the environmental sort of the esg's and if there's a way

from that perspective um to pull on that thread of you know the investor um

element to it uh just as as corporations are looking at you know how they how they do in terms of their esgs

could be one way it's a good thought let's suppose would that were to happen who's going to measure and who's going to measure what good question zach did you want to suggest something

uh yeah i just i think it's um it's a very interesting question about like where are these incentives coming from to to use these large language models i mean

i think you know you might draw the line between say academia and industry but i think even within industry there's different kinds of companies that will have different uh use cases and motivations for

building these systems so you might have uh one kind of company that's more interested in it kind of as a scientific artifact and trying to probe it and understand it and um you know certainly from

from my work that's that's where i'm coming from with it to you know to have these things and to try and understand and make it better and make it safer on the other hand you might have some

some companies that you know have a specific application of it in mind there'll be uh profit based incentives um for doing that one thing on on that side is that then

there's a bit of a tension between uh the kind of generality aspect um which uh which emily um was speaking about in terms of you know these language models are supposed to be the catch all they're supposed to be

able to do everything um but that that might be like a double-edged sword and it may not actually be um the best uh technology for the specific use case that a company

might have so it might actually be that these you know for for some uh kind of profit-based uh incentives this may not be the way to go um and so i think it's going to be interesting to see how this plays out

um uh and that that tension in between the the generality and and the use case and a lot of the times that we see um kind of troubling outputs from these

large language models is when uh we give it some kind of prompt which kind of hints to it that it it it should be kind of completing in a way that's uh that's that's not right so you know

if you if you prompt one of these systems with something that sounds like something you might find on like a forum for like far-right um extremism or something then it will

often respond in a similar way and so um the generality of these systems is that they they could potentially um respond in a wide variety of styles it's

just that those that's a double-edged sword that's that may not be appropriate for what you want to do um so yeah i just wanted to pick up on

that point um and i wonder how different um academia and research-based companies really are

because um at least in the sort of mainstream of our field we've got into this situation of being so driven by these

tasks um in a way you can actually sort of analyze that as though this is you know the profit motive in in academia

getting your paper into acl or whatever and a lot of companies aren't actually even trying to make money they're trying to get bought out by somebody else and the way to do that is to get

publicity so it's not so different perhaps in in in some cases and i think that sort of analysis is in a way if a company were really trying

to do something profitable with some of these language models they might not be behaving in the in the way they are um i'm not sure that you know it's really sensible to be doing some of this stuff

for some of the tasks uh as i say i think that you know much smaller data might actually turn out to in with a more detailed structure might turn out to be much better for natural

language understanding but once you get caught in this loop which academia is caught in just this way same way as companies it's very hard to get out of here that's really interesting i wonder if we

can think about that perhaps with regard to a particular very recent example here your thoughts about that which um i don't know if you've seen the paper that i think maybe come out today from open ai on codex

did you did you see this i've seen a few nodes and i don't know if you saw it there's maybe emily and zach you did see it couldn't comment but i'll i'll give a very uh short high level summary which is

that open ai took their big language model and then they um they fine-tuned it on the task of trying to turn what they call doc strings which i

believe is meant to be a way to describe what you would like a program to do so they take a doctrine and they turn that into code and actually they also go back the other way which is an interesting thing

but what they showed is that just taking the regular their regular pre-trained language model a little bit of fine-tuning and then with a few other little tweaks they come up with they did spectacularly well on this task

and i'm just giving that potentially as an example to show that that it seems like maybe there indeed are real commercial pressures towards using these big language models because they seem to be able to do remarkable things

with just little tweaks um emily zach what do you think about that so i want to jump in i actually have not read the paper i saw the paper came out today but you know it's only 9 a.m here so it hasn't been a lot of

9 a.m here so it hasn't been a lot of today yet um but there was a bunch of twitter chatter about copilot right which is the um sort of implementation of this

in um i mean it's github right and apparently behind copilot they took all of the public data on github as training data and this raises really interesting

questions about um the use of data in that way so this this practice of just grabbing data because it is accessible and not thinking about um norms and licenses and things like that and people

are starting to raise a stink about the fact that a lot of that data sort a lot of that code was released under gpl um and so the question is what does that mean for further code that was written with the

assistance of copilot if some of the things that get inserted come from gpl sources um and it's not that we it's new to be grabbing things sort of without

thinking much about usage rights and licensing but it's a new set of impacted stakeholders who might be motivated to raise the stink in a way that hasn't happened in the past so i'm

sort of curious to see how that unrolls in a policy perspective right in some ways arguably maybe to an intellectual property lawyer this is a bit like musicians who grab parts of people's

songs and use them in different ways but in some ways it's quite different exactly do you want to want to comment yeah um similarly i haven't read the whole paper just had a quick skin through

earlier today i wanted to highlight here that what i really like to see is there's a long extended section on the broader impacts and hazard analysis which i think is really great to see and it i think it sets a really good norm

for um publication style and i would really like to to see others adopting that as well um and in particular you know so that you know they have this in the main paper

and then they also have appendices um i haven't read them in full yet but um at least one of them is um around this this idea of misalignment um that i i mentioned a little bit earlier

about like you know you kind of you have to tease out the the kind of continuations that you want to see from the model you have to kind of prompt it in the right way to

get that and one of the things in this use case for for generating code um if they give it a prompt which maybe contains some buggy code then they'll get bugs

out from the code that's generated i think that's really interesting to see um and actually this use case is actually quite a good one for trying to measure the misalignment because when

it's just language it's hard to say oh well you know like this this bit of language is wrong language isn't usually wrong it may just be you know the perspective on it um we don't agree with it or something but

actually for code when there's uh when there's a task that's specified in the docs string which is kind of uh how it's presented in this paper then you can you can say with some objectivity the the code

here that's generated is wrong and i think that's really interesting the other thing that i think is cool here is um going back to this kind of uh notion of

deception and how your model might lie to you um they do some experiments in their appendix here where um they show that the model does know the

correct answer to the uh solution and yet it doesn't generate it unless you know they do some some extra things so

this is maybe one of our first hints um that uh of a more general phenomenon that we might see where um a system has a competency and yet it doesn't choose to use it because it's

it's it's uh motivation is not in line with ours um yeah so that that's kind of my my hot takes on it but i need to read the paper and some of what i said might be wrong so early days

thanks zach and did you want to compare to that well since i've read the paper i don't want to talk about that but i just want to indicate worry about the use of the word know and choose here from zach i'm sorry

i mean i shouldn't pick up on things like this i've done that i do this sort of stuff myself all the time but um i think it's problematic i mean we don't

have that we don't have the vocabulary to describe behavior by non-human agents properly i think or

non-um sapient agents probably but you know these models are not even by the standards of what we're doing in other parts of ai these models aren't

aren't agentive um or do you think they are i mean i guess that's a that's a question but i mean i would say they're not agents can i push on that a little bit um i

think that probably everyone in this panel is probably going to agree with that position and but but i think it's an interesting question for for many people the there's a

emily explicitly i think said earlier on on this meeting that we're talking about models that are just looking at the form of language maybe andy you also said that and i guess that also

contributes to this notion of that just being stochastic paris they're just taking strings of symbols and putting them together without really understanding what they mean but someone might say well maybe that's all humans are doing um on

the other hand you could say well humans are also connecting it with other kinds of sensory input which these machines don't have but someone might say that it's possible that there's enough information in the

huge corpus of language out there that maybe you can learn about real things from the language um and i wonder if anyone wants to take a position on that emily yes i have a definite position on

that so first to the argument of well maybe that's just what humans are doing i find that argument actually very dehumanizing because no one's own internal lived experience is that that's all we're doing

but if we assume other people are doing that we're basically saying well you're not as human as i am so i sort of have decided that i'm not going to enter in discussions with people who don't recognize my humanity which i think that that position

does not which you didn't take but you said someone might take um and then in terms of well what if we had all the language ever um and this is what we get into in the paper alexander kohler and i the um

it's called climbing towards nlu i mean that one is the octopus paper where we have something called the octopus test um and um did not put an emoji in the pedal um but in russia's facts should have um

and the thought there is it's scale doesn't matter like you can you can do some things with skill that you can't do without it but you can't get to information that's not in the training data just because you

have more and more and more of the training to do the information's not there it doesn't matter how much of it is there what you can get out of massive scale is really fine-grained modeling of co-occurrence patterns

in language and then this harkens back to the notion that um is attributed to wittenstein that so you know a word by the company it keeps or meaning is in um use um but the point is that

use isn't just textual distribution use is actually use in situated contextualized communicative intent and so as we use words and as they come to change meaning and i think ann's point about being

careful about things like know and understand and intend is one of those i think that um there's a general sense of these words that have to do with modeling very closely what humans do and

then sometimes people working on let's say language modeling for nlu or other tasks within ai um use them in a specific way to refer to states of machines

that are not directly analogous to what's going on with people and when you have conversations where not all parties have agreed on what that other meaning is it can go off the rails pretty quickly and i think we have to be coming back

around to this question of policy and incentives and things like that because our field is now so much in the limelight because the media likes to talk to us because people are interested we have to be careful to use language in a way

that is legible outside of our technical context even when we think we might only be in a technical context thanks emily and

i think i want to come back to um this idea that emily um mentioned and perhaps you know could have could have argued about more with this

co-constructed meaning we um [Music] we really are in a way forced to think of these models as humanlike

because they are generating language and i don't think people have fully understood the implications of this it's it's really very deep and it's us

it's not the models so um i mean this this you know it i i really think this has has huge implications but

coming back to um coming back to sort of some of these things that we're talking about i mean um if we

i'm sorry i'm slightly i'm slightly i'm slightly losing my train of my ability to express this i apologize i mean i think i think it is really hard to talk about these things i mean turing

started it by sort of saying mom i'm creating a brain you know i mean the the field has been doing this for many decades ever since the first computers were

um were really being developed but i think i i honestly don't know what to do about it all right i mean i think that talking about

these words like no and and so on it's incredibly difficult to avoid them but i honestly do not think i don't think that language models are in the least bit agenda because they're not wired up to be agentive

right i think that some other types of ai model could be agentive i don't think that there's any possibility whatsoever

that a language model can be learning everything it needs to learn about the world i mean they're not grounded even if you start relating them to vision and so on that

still doesn't give you grounding there's a lot of discussion we could have about that the example i like to give with my students is to say okay you know i've got a pen on the desk

i'm going to do with that what happens to the pen i mean that's what grounding is about now a dog understands that in some sense a dog is going to understand that if you go

with its toy it's going to fall over there um there's there's something sort of there's a very very basic understanding of how the world is and so on there's absolutely nothing about that

in language models i do want to say though that i think there may be some types of vocabulary where all there is really is

the form um you know abstractions um if you're going to talk about some of these things including many of the things we're talking about right now i guess i think a language model might be able

to learn them in the same way that a human does but those very very basic things there's very the fundamental underlying things which make us make us human they can't do that there's no there's absolutely no possibility

of doing that and it's certainly not not what humans do i mean these sorry just to go say one more thing these language models are being trained on a billion times more data than we could ever possibly hear

it's just not a comparable thing we are not training something which is human-like well a billion times more natural language data because we obviously take a lot of data in

in other ways but um well you've raised lots of really great questions and we could talk about those for a long time which would be fascinating but i think we should try to uh ask some of the the great

questions that the audience has asked in in the last 10 minutes or so so uh if we could look at some of those they think that the top the top question here is a question about the environmental cost of language models or even

machine learning more generally should is it fair that we talk about this resource allocation issue with respect to language models um but not really about about other areas of machine learning

um what do people think about that so just very briefly i think that um we need to think about environmental costs in everything we do and um to say you know if talking about it happens first in language models and it's slower in other areas of machine

learning i actually don't think that's true i think that the green ai folks are actually looking across machine learning um doesn't mean that we shouldn't also be thinking about it and similarly if you look you know from machine learning compared

to other areas of science and industry um you know should we you know should we not stop because you know other people aren't stopping well no that doesn't you know

we everyone loses that way agreed and there really it has been quite a bit of talk about large models in general not just language models yes anjali did you want to say something just echoing

emily but i think if we're talking about risks and harms then we should be it shouldn't just be restricted to the bias and discrimination as we

you know sort of defined um more clearly um and looking at all these other um downstream consequences whether they're director or indirect

great good point so we have a question from uh from carolyn ascherst um thanks for the excellent talk and paper data set curation of language models uh seems extremely challenging is there

a fundamental tension between the inclusion of sensitive topics for which we might not trust a stochastic parrot to deal with properly um and the harms of erasing those

perspectives from the model and how should we think about dealing with that problem i guess i'll i'll give a first answer but i hope that my co-panelists have thoughts here um yes this is difficult it is not easy

um and i think it's really important to think in terms of better not best um so it's not possible to create a completely bias-free model or data set um unless i guess if you shrink it down to

my data set is about arithmetic problems described in english and i'm not going to use any word problems i think you might be able to construct very small toy worlds where you can say okay this is bias free um but at any any level of

sort of interesting application you can't hope for bias free but you can work for less bias and you can say okay i want to include sensitive topics ideas and terms but i'm going to look for people who talk about them in a sensitive way

and i'm going to work with a data scale where i can actually examine the data and pull out cases where i find that someone that i thought was going to be sensitive about this in fact isn't um and so there's there's sort of room to to build something

um that is better thank you zach uh yeah i just wanted to uh i think echo that and i guess there's a parallel between how do we educate people to talk about sensitive

topics um it's not like we exclude um all mention of that stuff from them so in a parallel kind of sense maybe uh

you know uh we do want discussion of uh sensitive topics to be in our data set we just either need to have them appearing in a way that we would be happy with it reproducing if we're just going to be doing this kind of language modeling predict the next

word or we need to come up with other training objectives or training methods that don't just you know predict the next

word or something very simplistic like that but actually uh capture more what we actually want the the system to do so you know there's an example of this um where uh there was this kind of

summarization paper where they they um it's from open ai as well they had um humans kind of uh rate different summarizations of text

which was generated by a language model and then they kind of fine-tuned it using reinforcement learning so that it was not just predicting the next word given a prompt but it was actually

doing something else which was to try and summarize that piece of text and i i don't mean to say that summarization is like the be all and end all or anything but that that kind of general technique um seems like it could be used for many

different applications and one of them could be uh to try and uh reduce um the amount of like uh harmful material that's that's coming out of our models and that that doesn't mean that we have

to eliminate that from the training data set although that would be ideal maybe instead there are these kind of uh post-hoc things we can do it they don't come with guarantees and and i think that that's important to

recognize um but doing some empirical investigations of those from a scientific perspective seems like it would be valuable to me um even if that's not something that goes into production because it's you know

still too unsafe um thanks eric anjali um so i just thought i'd um go back to a previous point but bring a context of one of the projects that i

have which is looking at sexual exploitation online and so the the language that's used um on um online platforms

um and the the the question or the issue there is is actually trying to determine whether or not um potentially language there is suggesting that

um there is something coercive happening versus not uh and and so i should say my postdoc rosa lavelle hill is working on that

and what we um find is that you know that this is a real challenging issue but drawing on the the idea of bringing in stakeholders

um and community to help on that um understanding what features and and how they should be considered is part of that process but i think going back to this

point around filtering and um potentially value targeted data sets that um

open ai um spoke about in their blog too i guess one of the questions that i sort of sit with is well who is determining that value base and how do you

you know is it a person is it group how is that brought together how do you know that you sort of brought a sufficient um diverse sort of group of stakeholders to inform on that

yeah that's a wonderful question that i think we're going to have we're going to get we're getting close to be to be leaving on that note but let's give the last um the last minute or so to emily to

address that as she likes maybe you can talk a bit more about value sensitive design that you touched on before if you think that links to this sure yeah so i thank you i am i wanted to actually respond to that because i think that

it is a really good question of of how do we as we design these things get the input from people and give people a seat at the table or make sure that the tables you know has seats for people who

represent values of the communities they're going to be affected and it's an interesting point because i hear it in two contexts i hear it in the context of how do we do this right and i hear it in the context of

see this is impossible and i and i've heard you using the first version of it um and i think that one part of it there and this again comes back to value sensitive design where this idea of better not best and progress not perfection sort of is

throughout there um and i think what you can do is you can say um as the open eye folks do this is who we are and this is the context in which we identify these values and so if you say this was the

process by which we brought people together this is how we identified them this is how they identify themselves and that is the context for these values then by providing that perspective and that documentation

you allow somebody else to come and choose to pick up the system or not and choose to deploy it or not or you know if they have it deployed on them they have something to push back against and say hey look this doesn't match

our values here and so there's a lot to be gained from documentation um and then i guess sort of the brief thing to say about value sensitive design is that it's not one idea i mean there's this core idea of identifying stakeholders

and identifying their values but then there's this just huge array of methodologies for well how do you interact with people to identify the values how do you use them

to do some predictive work not just about what the good things are going to be um that comes out of your technology but also what the possible pitfalls are and on and on it's it's not a um it's not a plug and play like

okay i've now i've now import value sensitive design okay on right but it's a series of practices um that can be picked up and are all very much grounded in interacting with people

and i think this comes back around to a lot of what we've been talking about throughout is that we do better work when it is grounded in particulars and when we um if we're building something that

people will interact with especially if it's language where as ann keeps pointing out we can't help as humans but imagine a sentient being on the other side of language because that's

what language is for for us how um if we ground it in those specific contexts then we are better positioned to build something um that is understood and safe and deployable in a good way

thank you that's i think it'll have to be a good point on which to end uh let's thank all of our speakers as thank thank emily fantastic talk and for raising these important issues along with uh timothy gebrew and meg mitchell and

other authors thank you so much and uh thank you to anjali and anne and zach for a great discussion hope everyone enjoyed it thanks a lot for joining us uh i'll just say finally that

for those who are interested next wednesday at 3 p.m uk time we're gonna have a talk from nicole turner lee who's going to come and talk about uh algorithmic bias and ai system so continuing on the same theme so

please do join us then if you're interested and again thanks very much to everyone