Design of new protein functions using deep learning–David Baker (University of Washington)

SHYAM GOLLAKOTA: Welcome to the distinguished lecture from David Baker.

David is a professor in the Department of biochemistry and is also the director of the Institute of Protein Design.

Fun fact, he's also adjunct professor in the Paul G. Allen

School of Computer Science and Engineering.

I don't need to introduce his work.

Everyone knows about his work.

He has made some really transformative contributions to the foundations and applications.

He's really the leader of computational protein design, with significant impact on applications and foundations of both biotechnology as well as medicine.

He has won numerous awards, including the Nobel Prize.

I won't bore you much, but he's going to talk to you about some of the work he has done on using deep learning for protein design.

But please welcome me-- please join me in welcoming him.

And the floor is yours.

DAVID BAKER: Thank you.

I'm excited to be here.

And thanks for coming.

So today, I'll talk mainly about how we're using generative AI methods to design new proteins.

And I'll give you a survey of the different types of functions we can now design.

But I also wanted to touch on a couple other topics, which I thought might be interesting to people here.

We're starting to think more and more about active learning approaches where-- and I'll show you how we're thinking about that.

And then I think there's been a lot talk about general models of biology.

And I'll show you how we're starting to think about going from the structure prediction models we and others have developed to more general models of how cells work.

So basic introduction, proteins basically do all the-- carry out all the important functions in all living things.

They're really, really small.

This is an artist's rendition of a cell, where each blobby thing is a different protein that's carrying out a different function.

The proteins in biology evolved over millions or billions of years to be-- carried all the functions that were important during evolution.

But there are a lot of problems that we face today that there aren't really proteins that solve them.

And so the goal-- one of the goals of protein design is now to design proteins to solve modern day problems. Maybe just to set the overall context for people not so used to thinking about biology, so in biology, we have-- every one of us has a genome.

The genome has made out of genes.

They're made out of DNA.

Each gene encodes an amino acid sequence, and that amino acid sequence folds up to a protein with structure and function.

So the problem of biology is pretty much, how do you go from genes to proteins and their functions?

So I'm only going to be talking about that problem a little bit today.

So that's going from genes-- from DNA sequence to protein function.

I'm mainly talking about the inverse problem, where we're going to take proteins that don't exist, design sequences that-- so we're going to design-- we're going to start with a function that we want a problem that we want to solve.

And then we design a protein we hypothesize that will solve it.

Then, once we have that protein, it has a new amino acid sequence.

And what we can do then is we can basically print a DNA sequence which encodes that protein sequence.

It's a synthetic gene.

And then we can actually make that synthetic.

So we have that synthetic gene now.

And we can put that synthetic gene into a bacterium or some other type of living thing.

It will make that protein.

And then we can get that protein out, purify it, and see whether it actually has the function that we designed.

So in protein structure prediction and in biology, you go from DNA sequence to amino acid sequence to protein structure and function.

In protein design, you start with the function and you work backwards to an amino acid sequence, and then a gene which encodes it.

So that's like, the number one distinction that that's important to understand this.

Sometimes after my talks people say, well do you actually test any of your proteins?

And I've just given the whole talk showing them all this experimental data on designed proteins.

So everything I'm going to show-- well, for the first part of my talk, which is the majority of it, I'm going to-- basically I'm going to be saying, well, here's a function that would be great to have that doesn't exist.

And we're going to design a protein which has that function.

Then I'm going to be showing you experimental data showing that protein, what that protein can do.

So how do we actually do this?

We are big consumers of recent developments in computer vision and generative models generally.

And so probably all of you are familiar with diffusion models, where in the case of images, you have to take a very, very large set of images.

You noise them to different extents.

You train a network to remove the noise.

And if you've done this properly, you can start with completely random sets of pixels, successively denoise, and generate a new image which looks like a member of your training set but actually is completely new.

So we did exactly the same thing with proteins.

We took all the protein structures, and there's about 200,000 of them that have been solved and deposited in a database called the Protein Structure Database.

We took these 200,000 structures.

We noised them to different extents.

We trained a network.

We carved diffusion to remove the noise.

And then we can start with completely random placements of amino acids, successively denoise.

And what we get is something that looks like a perfectly respectable protein structure.

But it's not in the database.

It just looks like it could be in the database.

So this is unconditional generation, what I've described so far.

And of course, it's not very useful because you just get-- you get whatever the denoising process gets you.

So what you really want to do is to condition the generative process.

So in the case of images, you can give a text prompt.

You can say, generate an image of a cat sitting on a table.

In the case of proteins you can say, generate a protein with a particular function.

And so what I'm going to be showing you in the first part of my talk is the different ways we can condition this generative AI process to create proteins that do different things.

So that's basically the intro.

Does anyone have-- just to make sure everyone's on the same page.

Any questions at this point?

All right.

So how do you do the conditioning?

Well, the first way-- I'm going to show you different ways you can do conditioning.

One way you can do this is by fixing the coordinates, some subset of the atoms. So if you want to make a protein that binds to another protein-- in this case, it's the insulin receptor-- you keep that fixed.

And now you basically-- you start off with a random distribution of residues.

You progressively denoise.

And what comes out now is a protein that looks like a protein that should bind the insulin receptor.

But, of course, if it binds the insulin receptor.

So we make a synthetic gene.

We make the protein in the lab.

And now this is a very misleading talk in a way because I'm going to tell you about the designs that did what they were supposed to do.

But there are a large number of designs which actually don't do-- don't have the desired activity.

And that's because our models are not perfect.

But we did get binders out that bind the insulin receptor.

And we've been able to use these to make some very interesting insulin mimics that they bind to the receptor in a slightly different place than insulin does.

And so there are mutations which cause diabetes, which block interaction with insulin.

They don't block the interaction with these designs.

And so they look promising in some disease-resistant, insulin-resistant disease models.

So here's another example.

This is the TNF receptor in gray.

This is the central player in inflammation and inflammatory disease in our bodies.

So there's a large number of drugs, antibody drugs-- names like Humira and Enbrel, which some of you may have taken-- which target this pathway.

So here you see a diffusion denoising trajectory starting with a random distribution of residues around this-- around the TNF receptor.

And we make the proteins.

They turn out to bind extraordinarily tightly to the receptor.

And they have some very exciting features in animal models of inflammation.

So here, this is comparing to Enbrel.

So we have a number of-- we're exploring a number of ways of-- a number of disease applications with this compound.

And it's actually kind of interesting.

The pharma companies are interested in this as sort of an improvement over the antibodies for suppressing inflammation.

The Gates Foundation is interested in this as a way of improving response to vaccines.

So there's really a wide range of different things that you can do with proteins that block central parts of response.

So we have designed proteins which enhance the immune response to tumors.

And we have collaborators at the Dana-Farber Institute who are getting very promising results with pancreatic cancer models.

Which is, of course, a very, very bad form of cancer.

We can condition the process further by not only providing the protein target that we want to make binders to, but we can also specify what the overall shape of those binders should be.

And as probably many of you know, the pharmaceutical industry is really focused on antibodies.

I mentioned some of those before.

And so we can tell the generative method, generate-- we can condition it on the presence of the target.

But we can also say it should be an antibody-like molecule.

And in that case, what gets generated our antibody-like molecules.

And when we make them in a lab-- this is against the influenza virus-- they bind to the influenza virus quite well.

And we can solve, using electron microscopy, we can determine how the antibody actually binds.

And this is comparing the computational model to the actual experimental structure.

So some proteins in cells, you might know are not folded.

They're actually what's called intrinsically disordered.

And so those have been a real problem for drug design because they don't really have a defined shape to target.

But in this diffusion procedure, what we can do is fix the sequence of the target.

That's this black thing here.

But otherwise, denoise it at the same time we're denoising the designed protein.

And what happens is they end up sort of folding up or becoming ordered together.

And this turns out to be a very, very effective way of making proteins that bind to disordered proteins.

And so there are a number of applications here for things like chronic pain, diabetes, and other diseases.

I wanted to show one case, which is neurodegenerative disease is-- many neurodegenerative diseases like Alzheimer's are linked to a protein called Tau.

Tau is a disordered protein.

And so we can use this procedure to make proteins that bind to Tau.

And we can use these proteins to target Tau for destruction.

So these are cells that make a lot of the Tau protein.

And that aggregates and forms fibers in the cells.

And then if we make this designed protein that has this destruction signal on it, we can completely eliminate it.

And we're working with colleagues now here at the UW to see what happens in animal models of neurodegenerative disease when you knock out the Tau.

OK.

So those are some applications in medicine, but the applications aren't really limited to medicine.

We can design proteins for applications like sensing.

So some proteins, as probably many of you know are not just floating around in solution.

They actually insert into membranes.

And so we've been designing proteins that form pores in membranes.

So here are-- and we can design these.

These are pictures.

We can design these proteins to have pores of increasing sizes.

And we can measure the conductance across these membranes when these pores are inserted in.

And as you can see, the conductances increase as the size of the pore increases.

So there are a number of different applications we can-- for these types of pores.

One of the ones that we're quite excited about is for sensing.

And so what we can do is insert on top of the design-- say we want to sense a particular compound.

We designed a protein which binds that compound.

And we put it on top of the pore.

So the picture is kind of here.

In the absence of this molecule we want to detect, this protein has actually got three chains.

It doesn't come together.

There's nothing driving it together.

So the pore is open and we get conductance, as you see here.

When we add the compound, it drives-- these protein parts come together to bind the small molecule.

And that closes off the pore here.

So we're quite excited about this.

The way that the olfactory system works isn't really quite like this.

So here we've designed something that's specific for a single molecule.

But the way that our olfactory systems work is they're each-- we have a finite number of olfactory receptors, I think in the 500 or so.

Actually, I'm not sure exactly on the number.

But each one reacts to a number of different compounds.

And so your olfactory systems deconvolute afterwards from the pattern of response to all the different receptors.

So it's now eminently possible to do this with these designs.

There's nothing that limits the number of design proteins we can make, so we can make hundreds of different pores or thousands of different pores.

And we can design them to have binding sites of different classes that should interact with different classes of small molecules.

And I think this is eminently doable.

The next part.

So if we could take each-- take these many different pores, embed them into membranes, and make recordings, then you can imagine having an arbitrary solution of small molecules.

Or you could have something on your phone where it's just sensing things that are in circulation, and be able to basically have an electronic nose.

The hard part for this application is actually not the protein design part.

It's the interface.

It's basically the electronics and device aspect.

So I just wanted to bring this up because a lot of the problems we're stuck on now are problems where there are probably people in the audience who are much better suited to-- or may have ideas how to solve them.

There are other problems that are like this, which I'll get to in a minute.

Oh, and then one thing I promised, we were talking about just before.

We've also gotten good at taking proteins and getting them to sit down in an oriented fashion on electrodes.

And we can do-- we've done this mainly with gold, but we're looking at graphene as well.

And so we can put-- this will become more relevant in just a moment.

But we can make proteins that switch between two states, kind of like what you see here.

But now this is the electrode.

They kind of do this.

So if we put something like a spin label on the end here, then it changes the properties, the local properties of the metal.

So you can read that out.

And so I think this is another way in which you can couple proteins to electronic devices.

So we're very excited about this area.

And again it'd be great, I'd be very happy if a few collaborations come out of this talk.

OK, but along those lines, lipid membranes are not really the currency of the realm when it comes to electronics.

So we wondered, could we actually insert proteins into silicon nitride?

And this is collaboration with Jens Gundlach at the UW.

So the idea is you take silicon nitride chips, you drill holes in them.

There's been a lot of interest in these solid state nanopores.

But the problem is that they're variable.

There's a limit in the precision with which you can drill these.

But the idea is if we design proteins that-- and this is what we've done-- we can design proteins which have an outer radius, outer shape which is somewhat funnel-like, as you can see here.

So they insert in.

But then the conductance is determined by the interior diameter, which is uniform.

And so the idea is you can take these somewhat heterogeneous, solid-state pores, insert the proteins into them, and make them now into homogeneous pores.

And so that's the picture here.

And we have a number of designs now which give very, very stable and reproducible conductances after insertion.

So the applications here are generalized sensing, as I described earlier, but also things like DNA sequencing and protein sequencing, which now use naturally occurring proteins-- not design proteins-- that insert into lipid membranes.

There aren't naturally occurring proteins that insert into silicon, because that was never something that biology tried to solve.

So this is sort of an example of being able to design proteins to do something, which is really quite different from what any naturally occurring protein evolved to do.

OK.

Now, switches, I already talked about the interest in switches for sensing.

I think there's some very interesting computing applications of switches as well, which I'll talk about in a minute.

So we've been designing proteins which have two states.

So this is a schematic.

We have the straight state.

There's an effector that comes in and there's a state-switching, as you see here.

So this is a toy example.

But if we have a bent state that's bent so you get this triangular structure, and we change the degree of bending-- sorry, it wasn't bent.

Sorry.

This is the straight thing.

We have an interface here.

So this thing forms a trimer.

Now if there's a bend here, we get this square, as you can see.

So this actually works.

So these are now designed proteins.

Naturally occurring proteins are way more complicated than this.

But when we design things, we can design them to be really simple.

So here's the electron microscopy view of this protein before we add that effector.

Just a small molecule.

When we add the effector, it switches to the square-like state.

And here's a case where we've designed it to go from a square to a pentagon.

So this is kind of abstract.

Do you really care about going from one shape to another?

I'll show you some applications in a minute.

But one application I'm quite excited about is for super low power computing, because proteins form crystals.

Probably many of you know that.

If the crystals are made out of proteins that have this switch-like property, then you can imagine having a 3D crystal, a 3D array of the switches, where you put information in on one-- say on one face, and it propagates through.

Because depending on how you set the state on one face, that will determine what the-- since it's in a crystal, the neighbors will interact with each other.

So it's kind of like this sort of 3D Ising model, where information is proceeding through and there's really no energy input.

The hard part there is, again, the interfaces.

Because to get information in, you need to have-- the proteins are small.

So you need to have pixels on the 10 nanometer size range.

And there's a lot of them.

In a protein crystal, you might have 100,000 molecules on each along an edge.

So you need a lot of-- if you're going to control it with light or electricity, you need a really-- you need high resolution things.

So you basically put your information in, it would propagate through the calculation of be being done, and you take it out on the other side.

So there, you would need arrays of sensors.

So here's one biological application of this, where we have-- so in medicine, you want to activate the system, like for cancer immunotherapy, but you don't want to overdo it.

And so here we've designed a switch-like effector where it comes in, brings in the two chains, and then the effector comes in and shuts it off.

Because it causes the purple thing, which is the designed-- the thing that it's the design medicine.

It comes in.

When it brings those two things together, it turns on the-- it hypes up the immune system.

And then when the yellow thing is added, it turns it off.

And you can see on the right that this really works.

So this is the amount of signaling that measured on cells.

And if you just add the blue thing, then you get the blue curve where it goes up like that.

When you add the effector, that's the lower line where you see the signaling is lost right away.

So we can make on-off switches for medicines as well.

So we've been really interested in making machines and motors.

OK.

So a machine or motor is something where you put energy in and you drive repeated cycles.

So here's an example, coming back to this.

This is work from Cullen and Kathryn.

So you have this, again, this very simple switch-like protein.

Except now you can toggle between these states with light.

And so here, this is just measuring the distance.

In this experiment, it's fluorescence energy transfer.

So there's different-- sort of measuring the difference between these two dyes shown here.

And as we cycle between different wavelengths of light, we drive the system back and forth between these different states.

And so we can actually do this for many, many iterations.

So now we can put energy in, we can drive this conformational cycle.

And now we're trying to couple to things-- this to things walking along a track.

And so there's been this kind of dream in nanotechnology for a long time that you have these little miniature motors or things that go out and do things.

And I think we're getting close to doing that.

And so one, we're also working towards making machines like that, that use a chemical fuel.

And you can imagine machines zipping around in circulation, cleaning up everything that's messed up.

And there's no drugs that do that because you need a way of putting energy in.

OK.

So applications and sustainability.

And so we've been doing a lot of work on making catalysts to make and break chemical bonds.

And so for in the case of making chemical bonds, that's sort of interesting opportunities for Green Chemistry, for coming up with cheaper, more efficient, and less environmentally harmful ways of making chemical bonds.

And for breaking chemical bonds, things like breaking down polymers like plastics that are accumulating in the environment.

And we're making some progress now in designing proteins which break down plastic.

So I wanted to show a sort of a recent result.

So one thing that proteins do is they break down other proteins.

The reason this is interesting is because most drugs that we take block the interactions between proteins.

But you need a lot of the drug because you have to block-- there are many, many-- there are many, many-- that interaction happens many, many times in the body.

And you need at least as much drug as there are things that you're trying to block, but you have saying that works catalytically, that goes and just chews up the target, breaks it.

Then you just need-- you could use a very small-- have a very small amount of drug.

So I won't go through the chemistry here.

This is a protein which has been designed to hold a metal ion, a zinc ion, so it activates a water molecule.

But we can make proteins-- which again, I'm not going to go through the biochemistry-- but we can now make proteins that break chemical bonds.

And they break peptide bonds in the bonds that hold proteins together, which are normally very, very stable.

This design actually increases the rate of this reaction by over eight orders of magnitude.

So now we're designing proteins that break down targets, things like A-beta and Tau that, as I mentioned earlier, are involved in neurodegenerative disease.

And so we can, again, solve structures afterwards at these catalysts and see if they work the way that we intended them to do.

And what we find is that the residues that we positioned to carry out the catalysis are very precisely placed.

And then final two examples, and I'll be done with design.

So I've been kind of fascinated with materials like bone and tooth and shells, which are made out of proteins that template the deposition of inorganic minerals.

And so we've been very interested in doing this, both with naturally occurring minerals like calcium phosphate, as well as things that don't occur in nature, like zinc oxide.

And the reason to be interested in things like zinc oxide is you could-- it would be a new way of patterning a semiconductor rather than top down, like as in lithography, from the bottom up.

And so we design protein assemblies which then have regions on their surface, which template the deposition of mineral.

And I'll just illustrate this with calcium phosphate.

So here's a crystal of calcium phosphate here.

And we design a protein which places protein amino acids in a way that's complementary to the crystal lattice.

And then what we find-- and then we can build these into tubes like this.

And so if we look at the structures of these, before we add calcium phosphate, we see rings and tubes.

And the tubes are empty.

Once we add calcium phosphate, these rings fill in and the tubes fill in.

And if we look inside, then we have crystals, which have the spacing we expect for calcium phosphate.

So this is kind of exciting.

It sort of opens the door to design of a whole host of new hybrid materials.

Again, we can use Earth-abundant things like calcium phosphate that nature used, or we can use more exotic materials that could have applications for all the things that semiconductors get used for.

But the important thing is we're patterning these with proteins, which again, are very, very small.

These things are just angstroms apart here.

So we have really atomic-level control.

And then the final design example is photosynthesis.

Of course, nature is very good at capturing sunlight and turning it into the synthesis of molecules.

And so we've been able to design proteins that mimic the central step of photosynthesis, which is the capture of light by the photosystems. And we're now trying to couple that, catch that absorbed energy, use that absorbed energy for catalysis, both making and breaking bonds.

Primarily making bonds because that's, of course, what happens with photosynthesis.

If you look at what happens in nature, the coupling between absorption of light and the making of all the things in plants, it's very complicated.

So we're trying to make-- considerably slim down systems that can be autonomous, light-harvesting systems that carry out chemistry.

And we can assemble these light-harvesting systems into assemblies shown here where each of these blue things has two chlorophyll molecules.

I'm sorry.

These are chlorophyll molecules, which the designed protein positions.

OK.

So that's all for design.

So I just wanted to have five more minutes, tell you about some of the methods development stuff.

If you guys want to come in a little bit, there's plenty of room in here.

All right.

So I'll tell you about different things.

This is actually-- this was done by first year CS students last year.

And here the idea is-- so what I showed you before was-- the generative model was at the level of amino acid residues.

Jasper and Yan Jing and the other people listed here found that it was actually even-- we could get considerably more control by carrying out the generation at the level of atoms. So each atom is an independent unit here.

And that's nice because a lot of the problems that we're working on now, for example, this catalysis problem, we really want to be able to position individual atoms in space.

So it's very easy to implement constraints where you have a particular 3D configuration of atoms you want to reproduce.

It's very easy to implement those, that kind of conditioning, if you're actually carrying out the diffusion of atom coordinates.

Before we had to-- it was a little bit more tricky to implement atom level constraints when you're diffusing amino acid residues.

And so for example, for the proteases that I described, we're using this method, which is called RFD3.

Jasper and team just released it a couple of weeks ago.

So I wanted to talk a little bit about the active learning problem.

And I already alluded to the fact that not all of our designs work.

And for really hard problems, where you want something that-- like, for catalysis, where let's say you want to break down something like polypropylene or something that's really not very active, it's unlikely it's going to work straight out of the computer.

And so what we want to do there is we can use the procedures I've described to make an initial set of designs.

We can measure the activities.

And then we want to use that information on which designs had activity to go back for a second round.

And so we can-- that's the active learning part because you're suggesting a next step or you're trying a next step.

You're collecting data and then you're updating based on that.

So the problem is unlike a robot arm moving, the time to collect your next-- it takes more than a few seconds to get your feedback on what the-- the feedback from the environment that you're going to learn on.

So this effort has sort of involved both computation as well as trying to really speed up the experimental cycle time.

So we've been doing this with plastic degrading proteins.

So the idea here is we can take a design like this one here, and we can chunk it up into pieces.

The expensive part of everything I've shown you is the manufacturing of the DNA.

And so what we're going to do is divide this protein up into some number of fragments.

And then we're going to use computation to come up with hundreds of alternatives for each of these different colored fragments.

And so basically, you have this space, then, where for each of these fragments, we have a large number of possibilities.

Typically, because the way these things are done, 384.

And so the space of the total number of combinations of possibilities is we have 384 for each of the different colors.

So it would 384 to the fifth.

And we've set this up so we can very rapidly manufacture any of those 384 to the fifth proteins.

But we can't do it for very many of them.

We can only do it for 1,000 of them at a time.

So then you can see what the problem is.

We've got this massive space.

We can get feedback pretty quickly, but we have to-- and we've got computational methods that will-- that let us make predictions about how active any one variant will be, but we want to augment that with the experimental feedback we get.

And so this is basically how it looks.

We have the reward is the actual-- well, actually, here we're-- this is using an in silico metric.

But anyway, so we can go around this loop to optimize.

We can do this both first using in silico reward functions, but we can do this second where we actually are experimentally measuring activities.

And so we've basically put together a pipeline where we can do all this pretty quickly with about a two-day iteration cycle.

And so now, we're at this kind of interesting stage where we have this massive space.

We can test 1,000 variants about every other day.

And the question is, how do you most efficiently choose your choices to get to maximize your information so you make-- so you learn as fast as you can?

So that's kind of a fun problem.

I'd love to talk to people about that if there's interest.

And then, finally, there's been a lot talk about general models of biology.

And the way these are typically framed is you collect data sets which are called single cell RNA seq data sets, where you collect data sets on all the RNAs, mRNAs which are being made in the cell.

And we've been thinking about general models of cell function, which instead-- which work more from the ground up, from the genome to proteins then to the protein functions and their interactions.

So the first step in that is going from sequences to structures.

That's the structure prediction problem which I described earlier.

And we've had some really interesting results there, which I just wanted to show here.

So these methods and the design methods I showed before are all trained on the protein structure database, which is a collection of all the proteins whose structures have been solved by crystallography or electron microscopy.

The problem is it's a very great database, but it's finite.

And so we're data-limited in trying to improve these models, both for prediction and design.

So what I'm showing here is-- so you're all familiar with the concept of distillation, where you have self-distillation.

You take a model and you make predictions with the model.

You take the confident predictions and you use those for to actually train.

So here, what we did is we took sequences.

So normally, what your training structure prediction methods on is sequences where you know the structure.

So here we took sequences from what are called metagenomes, where you have just these soups of organisms. People determine all the sequences of all the proteins in them.

And so all we-- the only experimental fact we used was that these sequences exist in some organism.

Therefore, they likely fold to a three-dimensional structure.

So we took our model, predicted the structures of those sequences, and then use those-- and then train the model on those.

And what was really quite remarkable was, I mean-- so if you do this, it's not too surprising the model would get better at just predicting protein structures.

But this model got better across the board.

So it got better.

I should say this is a general model.

It's called RosettaFold-3.

It got better at predicting protein DNA complexes, predicting RNA complexes, protein ligand complexes.

So basically, what we're finding is that if you have a general model that can predict quite a few different things, and you have a rich source of data in one domain, you can distill on that and it will improve your performance across the board.

Now this is really exciting in biology, where we have these troves of data that are very rich in different areas.

And the concept we have for moving up the biological complexity ladder from just protein structures, these interactions to assemblies, and then to some more complex things that are in the cells.

As the model gets better can distill more and more effectively on these more complex data sets.

And then train on that data and the model, if this example is representative, we'll keep getting better.

And so we've been using this concept.

And it actually is really working if we just even look at-- this is coming back to the structure prediction problem.

So the state of the art for quite a while has been AlphaFold-3 from DeepMind.

And we're now getting models that are systematically better than AlphaFold-3.

And it's kind of interesting, this is how accurate the models are with the model I'm described compared to-- sorry, this is AlphaFold-3 and RosettaFold-3.

And you can see that the models are systematically more accurate.

And it's kind of a neat thing for this distillation idea, because you can now take a more accurate model, go back to the cases where you're distilling, and build more still more accurate models.

So we're hoping there's a virtuous cycle, which lets us move up the complexity ladder pretty systematically.

So what we've been doing is trying to collect from the literature all of the examples of-- so again, the protein structure data bank is crystal structures.

But there are a lot of data sets where people have simply reported that two things bind, like two proteins bind or protein binds DNA.

There's a lot of examples.

So we're collecting these examples, starting to use this to train, to distill on, because we can build models of the complexes.

And now we're in this very interesting situation.

We're partnering with the Allen Institute.

And we have the opportunities now and the resources to generate a large amount of experimental data ourselves.

And so the question is, what data sets do we need to generate to maximally improve the model to, again, be able to make more complex predict the properties of more complex systems so we can then access new data sets to distill on and so, keep moving, improving the model.

Having all this data opens the door to using reinforcement learning for training.

So you can be your models can be guided by the experimental data.

And OK, so I just want to tell you a little about some of the computational problems we're working on.

And it's been really fun to work with CS students in my group, and it'd be great to have more interactions with the department.

So there's applications I've shown you for protein design in medicine, technology and sustainability.

And it's been a lot of fun working with everybody, so I'd be happy to take any questions.

[APPLAUSE] SHYAM GOLLAKOTA: All right I think we have microphones across the room so you can just speak up.

Just raise your hand first before you speak up.

AUDIENCE: Fair enough.

High level.

How long do you think it takes for some of the discoveries here to actually find their way in?

DAVID BAKER: Yeah.

In medicine, it really has to do with the whole clinical trial, FDA approval thing is really the bottleneck.

It's actually pretty straightforward now to generate new proteins that have properties that make them look like they could be good therapeutics.

But the process takes a while.

We've had a couple of proteins in human clinical trials.

My colleague, Neil King, actually had a vaccine approved that was a De Novo designed protein.

And so there are proteins that are now appearing in the clinic.

And there's also been a whole host of companies, some of which we've founded and some of which have been founded by others, which are-- they say they're joining the protein design revolution.

And they're making all kinds of new drugs.

So there's a lot of claims right now.

But I think there's a lag period there for getting them out.

In the technology and sustainability things, I think the challenge there is-- so in medicine, you have a very active VC community, investor community that will put the money in to develop things.

I think we have a little bit-- there's still, I think, a mental gap between, say, using proteins and devices.

So there, it's a matter of-- I think to get them out in the world, probably a matter-- probably maybe teaming with people here, it's just it's a matter of convincing investors that this is really-- this is the way of the future.

AUDIENCE: I just had a question about, you're talking about in medicine in binding to the insulin receptor and things like that.

And when you do that, how specific is that binding?

I mean, are there off-target effects and things like that?

DAVID BAKER: Yeah, so we've actually been measuring it.

There are people who can do experiments where you take a protein and you mix it with every protein that's in a human cell.

And you basically pull things down and you see what sticks.

So for about half of the proteins we've looked at so far, they only bind the single other protein they're supposed to bind.

Some of them are less specific.

So we're trying to work out what are the properties that make them more specific.

The more recent-- the things we've made more recently tend to be more specific than the ones from four years ago, which is a good sign.

But that's particularly important when you think about proteins binding DNA.

So we're doing a lot of work on designing proteins to bind to specific DNA sequences.

Turns out, very easy to get proteins that will bind to DNA.

Get things that bind with very high specificity is harder.

So we're still learning how to do that.

AUDIENCE: I just have a question.

What's the success rate for this model?

DAVID BAKER: The success rate?

Well, it really depends on the problem and where we are on it.

So of what I showed today, for example, the proteases, which were the enzymes that cut, those were probably the most complex designs that I told you about today.

Those were-- we tested-- I think our success rate was probably around 5% to 10%.

And the switches are also complicated.

Those were also-- we typically, because of the way that DNA manufacturers work, we typically will get to test 96 designs.

And so for other problems, there are some problems that we've tried that we still haven't succeeded on at all.

So there, the success rate is 0.

And there are other problems which have really been quite become quite easy, where the success rate would be higher.

Yeah.

AUDIENCE: So following up on the proteases, so the ones you described like the target Tau beta.

DAVID BAKER: Yeah.

AUDIENCE: With a seven layer [INAUDIBLE] sequence.

Is it easier to make ones that are less specific?

Meaning they just recognize maybe a two amino acid pair or something like that?

DAVID BAKER: Yeah.

AUDIENCE: Targeted.

DAVID BAKER: Right.

I think what we're worried about right now is that when you put them into the human body, they won't be specific enough.

But you're right.

For some applications, you want less specificity.

It's pretty straightforward.

I mean, what gives-- so we position the catalytic residues.

But then we have a binding pocket.

And so the binding pocket could be short and just have room for two amino acids, in which case you'd expect only to have specificity for two.

Or these ones have a more extended binding groove, where there are pockets for multiple amino acids.

So I think the extent of specificity can be modulated in that way.

Yeah.

AUDIENCE: Could you say something about the risks of this new technology, in terms of-- DAVID BAKER: Yeah.

AUDIENCE: You're a good actor, but there are bad actors.

DAVID BAKER: Right.

AUDIENCE: What can they do with this?

DAVID BAKER: Right.

I mean, for good for-- you know, the truth of the matter is that nature has kind of perfected ways of causing death and destruction on really massive scale.

And it's kind of interesting, the 1918 Spanish flu is the most dangerous virus that's ever existed on Earth.

And there was a decision-- there was a big debate, maybe 20 years ago, about whether or not to publish the sequence.

And it was published.

So now anyone can make the 1918 Spanish flu.

The sequences of Ebola virus are known, botulinum toxin is known.

So anyone can potentially make those.

And those are considerably more dangerous than anything one can design now.

But the methods will keep getting more powerful.

So I think it's something that one definitely has to think about.

We actually had a workshop that actually the National Security Council asked us to convene last year here at the UW.

And I think the consensus was that the methods right now, there is much more potential to do good.

It's easier to block a virus than make a brand new virus.

The thing that I think is really important, that will require international collaboration-- which might be a little bit hard in the current form-- but to actually make any of these, there's the computer calculations.

But the key step is when you go from the amino acid sequence of the protein to the gene that encodes it, the physical DNA that encodes it.

And that's done using DNA synthesis, typically a machine, a DNA synthesis machine.

So what I think would be really important, what should be done, is to ensure that these machines always have to be hooked up to the internet.

So every time a sequence gets made, then it's uploaded to a database which is encrypted and totally private until there's some emergency.

And then it can be queried to see where that sequence was made.

So you'd have attribution, and I think deterrence.

Because the truth of the matter is making any virus, it's going to be-- there's going to be many, many false starts.

But something suspicious will turn up.

And so logging DNA synthesis, I think, is the best way to protect.

It's not only against designs, it's against if someone resynthesizes the Spanish flu, you'll who it is.

Yeah.

AUDIENCE: Speaking to infectious diseases, I was wondering about, you mentioned on this slide pandemic preparedness as an application for protein design.

I was just wondering if you could speak-- DAVID BAKER: Yeah.

Well, during the pandemic, we made vaccines.

Neil made vaccines for COVID, too.

We made very effective therapeutics.

And there was a lot of enthusiasm at high levels in the government about really doubling down on pandemic preparedness afterwards.

And so, we actually then set out to make proteins to bind-- to block the major respiratory viruses.

But I think everyone knows what happened.

Within about three months of the pandemic ending, there was a complete loss of interest in anything having to do with pandemic preparedness.

And so now, it's actually kind of funny.

So we made all these proteins which are really good at blocking things like MERS and RSV and Nipah and Hendra viruses.

But VCs have no interest in those.

Pharma companies have no interest in those.

And so the things that get picked up on are like the inflammatory disease, the autoimmune, the cancer.

So I think there, it's more we're at a society level problem of how important is pandemic preparedness?

Yeah.

AUDIENCE: Do these diffusion models account for how the proteins get cooled as they're generated, or the different interaction?

DAVID BAKER: No.

And that's a really profound point.

So you might have thought that this should never work.

Because we're just specifying the final state.

And this denoising trajectory is not at all a representation of how the protein folds.

So it could have been that what was-- that you really had to encode the folding process itself in the sequence.

But here, I mean, we've probably designed-- we've designed many hundreds of thousands of proteins.

And we've never paid any attention to the actual folding process.

It's kind of ironic because when I came to the UW many years ago, it was like, at that time, it was thought how proteins fold is really critical.

But it turns out that if you can think of a folded protein as being a really deep energy well, and if that well is deep enough, then the protein will fold to it in the same way that ball sort of rolls downhill.

So if there is a profound result from this on that point, it's simply that simple-- simply being in a really low energy well, minimum, is sufficient to guarantee folding to that.

So thermodynamics determine kinetics in a way.

Oh yeah.

AUDIENCE: Thanks for the session.

From the AI and ML industry, what is needed that can give energy boost to this work?

DAVID BAKER: Well, I think everyone here who's a practicing ML or AI researcher knows that there's one thing that you really, really need that's really expensive, and the whole world is trying to get more of it and doesn't want to give you any.

Which is compute.

Yeah, so that is the limiting resource on the methods development.

I mean, so a lot of the stuff that we were doing was going back and forth with DeepMind's.

And the problem was that, suppose you're trying to improve an algorithm and you've got five different architectural changes you could make.

And there are different data sets you could use.

And if you're sitting within DeepMind, you could try each one of them one at a time, do it in triplicate, and then see what the effect is and then take the things that work.

For us, with 1/100 or 1/1000 of Compute, we just had to guess, and so it's just really hard.

It makes it much harder to develop.

So with more Compute, can try more ideas and move much more quickly.

So that's the limiting thing.

Yeah.

AUDIENCE: You talked a little about using proteins to actually do computation.

DAVID BAKER: Yeah.

AUDIENCE: Could you say a little more about what are the biggest open problems?

Like, is input/output the hardest problem?

DAVID BAKER: Yeah.

Input/output is the hardest problem.

So the attractive thing about this idea is the same thing that makes it hard.

So proteins are about-- these protein switches would be about 10 nanometers.

Let's say that you have a 10-millimeter crystal.

So that's a million copies of the protein on each side.

So that means you've got-- you have a very large number of-- so you have a 3D array of a very large number-- a very, very large number of transistor-like molecules, which in principle, incredible computing capacity.

But to address it, if you want to set the state of each of those transistors on the input surface, you're going to need an array.

You're going to need something which can individually address those, either with light or electrically.

And then to read out the result of the computation on the other side, you're going to have to have something that can convert, can read out there as well.

So I think it's how you do those interfaces between proteins and electronics, I think is the key to a lot of these problems. Yeah.

AUDIENCE: You mentioned there are hundreds of known sequences used as training data.

Compared to all these language models today using billions of tokens, that seems a very tiny small data set.

Is that limiting factor on your model?

DAVID BAKER: Yeah.

So the number of structures is about 100,000, 200,000.

But in that distillation experiment I was describing, we were using 10 million sequences.

So we can convert those sequences to structures by making predictions.

So yes, the 200,000 structures are definitely a limitation.

And that's why distilling becomes so important.

So if you can train a good enough model, then you can start taking things like sequences and turn them into structures and then train on those.

So yeah, we're definitely limited by data set size.

And for things like protein small molecule interactions, which is really critical for drug discovery, the data is particularly limiting.

So creative ways of using lower resolution, experimental data are really critical.

Yeah.

AUDIENCE: So I find it curious how modeling the proteins with the single atoms was more accurate than using the side chains, considering that the side chains of amino acids are so crucial to tertiary structure unfolding.

So do you have any idea about why the single atom prediction was more successful?

DAVID BAKER: Yeah, I think there's a general trend overall in machine learning that just simplicity is usually better.

I mean, so if you go with all the atoms-- and the transformer's at the heart of this-- it's just really, really simple.

Because you're just noising the atoms. And it's a standard, vanilla sort of denoising thing.

So think about the problem where you're trying to get-- you're trying to position, say, seven atoms very precisely in space.

One atom here, one atom here, one atom here.

You know exactly where you want them.

If you're doing the diffusion at the residue level, then you have to do something fairly complicated to translate those conditions back to where the backbone-- to guide the backbone.

Whereas if you're diffusing atoms, it's really trivial to implement that constraint because you exactly how far away the atoms are.

What we do, what Jasper developed is a way of doing this basically unindexed.

So you don't have to specify which atoms are there, but it can be-- but still, doing it at the atom level, it makes the conditioning much more straightforward, I think is the answer.

Yeah.

AUDIENCE: So firstly, I want to thank you for your presentation and I have a question.

When studying chemistry, one of the problem is the use of catalyst to convert carbon dioxide into ethanol.

DAVID BAKER: Yeah.

AUDIENCE: So do you think that we can use-- create a protein combined with metal to solve the problem?

DAVID BAKER: Yeah, so we're working on that with-- we're collaborating with people who are trying to improve carbon fixation.

And there's a number of different routes.

What nature does for carbon fixation is very complicated.

It's done by a protein called rubisco, which is not a very good enzyme.

And with our collaborators, we're trying to make proteins that can do this more efficiently.

It's a tall order because nature has obviously been working on that problem for a long time.

But that is a very interesting application for protein design.

Yeah.

AUDIENCE: What is the thought process of what the lab actually ends up working on?

And I'm curious where those ideas come from.

Is it mainly, like, Gates Foundation pharma companies are interested?

DAVID BAKER: It's really all over the place.

I mean, some of it is external like that.

Sometimes people come to my group with a specific problem they want to solve.

A lot of it just bubbles up internally because we're this communal brain idea, where everyone's talking to everyone all the time.

And so, people will be talking and say, hey, we could do this.

I get a ton of email every day with people suggesting collaborations.

And people go to meetings.

It's a very good question because there's so many things you can do now with this technology.

I'm sure we're not necessarily optimized to answer that.

But yeah, I mean, it's like-- I think the general answer to that question, ideas come from all over the place.

So yeah.

AUDIENCE: Can I actually build on that question and ask a different question?

When you started at UW, what made you pick this specific-- DAVID BAKER: Well, I didn't.

Yeah, so for all the undergraduates in the audience I had no-- I mean, I had no concept.

This would have been much more foreign.

I wouldn't have understand a word of what I said.

And when I came to the UW, I wasn't even thinking about proteins.

I was thinking about the silly problem that I mentioned earlier, about how proteins fold.

We were looking at it experimentally.

So it's just one thing led to another.

So yeah, you don't plan too far ahead.

I think that's my main piece of advice.

SHYAM GOLLAKOTA: Maybe we'll take one more?

One or two more questions and then.

DAVID BAKER: Yeah.

AUDIENCE: Yeah.

So you mentioned earlier that there are harder and easier problems to solve.

I guess I'm kind of wondering, how do you define the solvable set of problems?

DAVID BAKER: Yeah, well, I think that's sort of my job.

You sort of have some intuition for-- there's also the problems that are easy, that or the problems that have already been solved.

We really don't want to work on our problems that have already been solved.

But people still do that.

And you probably know people who work on problems who are already solved.

So you definitely don't want to do that.

And then there are the problems which can't be solved today, and you just don't want to work on those either.

Well, you don't really know.

But you don't want to invest too much on the problems that are probably unlikely to be solved today.

So you want to identify the adjacent possible, the things that aren't solved, but if you're smart, you work hard and you talk to people all the time, you're likely to solve in a year or two.

And that's my job when I'm working with my students is, OK, well, I think you'll solve this problem.

And that's really important.

That's like, the key part to all of this.

So yeah, that's probably the hardest part of doing science is picking the problems that aren't solved, but can be solved and are important, so that when you solve them, it's an important step forward.

SHYAM GOLLAKOTA: One last question?

DAVID BAKER: Yeah.

AUDIENCE: So you kind of talked about affinity and specificity when talking about designing proteins.

When you think of medicines, like I'm thinking a vaccine, another factor would maybe be how stable as a function of different temperatures.

Is there a limit to how many objectives you can optimize for in these?

DAVID BAKER: Well, it's interesting.

Stability.

One of the things that's surprising that we found when we first started designing proteins, the proteins that we design are almost all really, really stable.

You can boil them and they're fine.

So that we got for free.

But in terms of multi-objective optimization, I think the main issue is for drugs, there's something called developability-- how likely it is that you'll be able to manufacture at a large enough level and behave well enough when it's put into a syringe-- where we don't really have a lot of data.

And so we don't, really.

But if we had a lot of data-- so I think it's getting those objective functions.

I think doing the multi-objective optimization is not so hard.

It's just standard.

That's standard stuff.

SHYAM GOLLAKOTA: All right.

So I think David is going to be here for the next 10 to 15 more minutes, if anyone has questions.

But let's thank him for this incredible talk.

[APPLAUSE] DAVID BAKER: Yeah.

And again, we're really excited about collaborating.

So yeah, let me if you're interested in anything.

Yeah.