How We Scaled Kimi K2.5 | Zhilin Yang's full GTC 2026 Keynote

Hi everyone. Thank you so much for the introduction. It's great to be here. Uh

introduction. It's great to be here. Uh

it's great to have this opportunity to share with you guys uh some of our latest progress and efforts.

So uh one of our major pursues uh is to build better open models and we believe in democratizing intelligence. With open

models you can deploy anywhere. You can

be on your local servers, you can be on the cloud and you can access every single bit of the weights in the modeling instead of just uh using a

black box. And this is one of the slides

black box. And this is one of the slides that I took from Jensen's uh talk earlier this year at CES. So as you can see, open models are quickly uh closing

the gap with uh proprietary models and it's reaching the frontier. And we

believe that with better and better open models, we're going to u make intelligence more accessible to anybody in the world in every corner of the

world. But open models cannot be just

world. But open models cannot be just open. they have have also to be great.

open. they have have also to be great.

So uh in this talk we're going to discuss how we make open models great.

Uh so as we know uh scaling is a primary driver uh for a lot of progress maybe all of the you know major AI developments that we have witnessed in

in the last few years and here we're going to discuss how we scale our model in different dimensions. So on the left hand side the first figure you see here

is kind of the the standard scaling law.

So on the x-axxis you have the uh log of the number of training tokens and on the y-axis you have the log loss and as you scale the number of training tokens you

get a lower loss. But here the point is we're not going to just scale the number of training tokens but we also want to want to improve uh the token efficiency.

uh meaning that we want to move this curve to the left hand side so that we can achieve a lower loss a much lower loss using the same number of training

tokens and this can be achieved by having better architectures and optimizers as we'll discuss in in our later slides and the second scaling

dimensions that we're very interested in is to scale the context lens so as you can see in the second second figure uh if we increase the context length then

we can uh have a much higher accuracy in terms of predicting uh the token loss at a given position. Uh and this means that

we can increase the capability of the model to achieve more complex task by increasing uh the context. So this is the second scaling dimensions that we're

going to talk about. And the third scaling dimension is the number of agents. So we introduced this new

agents. So we introduced this new learning paradigm of agent swams where we can we don't just rely on a single agent but we also uh orchestrate a swarm

of agents that can accomplish the subtask in parallel so that we can increase the task capacity and we can translate all of this in into the language of agents. So if you look at

token efficiency, it's mostly about having a stronger prior so that you can uh have more efficiency when you do agent RL to search for a uh better

solution. And you think about long

solution. And you think about long context is it's mostly about increasing the context length so that you can have a longer running agent. It can probably run for days or even weeks or months to

accomplish more uh more more tasks more uh complex tasks and about and for agent swarms is is another dimensions that that add to it and at the end of the day

we're going to have a swarm of agents that each of them have a super long context and each of them have a very strong prior for us to uh search uh in

this entire agent RL system.

All right. So we're going to start uh from token efficiency. So this is one of the most you know classical figures in the history of machine learning. Right.

So it's taken from cap Kaplan Adel and it basically says that if we scale proportionally the number of training tokens the model parameters and also the

amount of compute we can get lower and lower loss. And this is you know one of

lower loss. And this is you know one of the major breakthroughs that that the entire community uh has achieved in the last few years to to get uh better

intelligence. But here what what we're

intelligence. But here what what we're interested in is is to have better and better uh token efficiency. And uh

here's the thing. So one thing that I would like to emphasize is that token efficiency is not just about efficiency.

is actually also about improving the upper bound of intelligence. So here

here's why. So uh suppose you have uh say 50 trillion tokens, 50 trillion high quality tokens and then you apply this

new optimizer maybe the m optimizer and then all of a sudden you have a two times token efficiency. So it means that it's almost like magic that you get uh

equivalently 100 trillion tokens. Uh and

nowadays we are scaling towards the data war and we're hitting you know the data war and the amount of high quality data is quite limited and if we suppose that

is a constant amount then we increase the token efficiency it means that uh we're going to get better intelligence out of it. It's not just about infrastructure efficiency. It's about

infrastructure efficiency. It's about you know better uh intelligence. So so

this is why we spend you know a lot of efforts in this aspect because it's going to push uh the frontier of of intelligence and m optimizer is one of

the things that we have heavily invested in uh since last year. So it's a second order optimizer and basically every

single gradient update is transformed in a way that each entry is orthogonal to each other and this is very different from the traditional atom optimizer and

if if you implement this optimizer properly you can get a two times token efficient efficiency improvement. Uh so

we we are one uh we are the first uh work we published the first work to demonstrate that SM optimizer is actually scalable for LLM training and

these are two key techniques that it we employ to make it effective for large scale training. So one of them is decay

scale training. So one of them is decay it it is critical for scaling to larger models and the second is uh we want to ensure a consistent RMS updates compared

to Adam. So we have this uh adjustable

to Adam. So we have this uh adjustable coefficient that is applied uh to each update so that uh the uh resulting RMS

is going to be comparable to Adam and uh to make MIUI memory efficient across all this uh Nvidia GPU clusters. We also

develop a distributed mumm optimizer implementation that partitions the states across uh the data parallel group so that we can have a very efficient

imple implementation for the mum optimizer and these are some of the results uh that we presented in the paper. So as you can see with with the

paper. So as you can see with with the same number of parameters and the same number of training tokens we just replace the original AdamW optimizer

with the new MU optimizer is going to to improve the performance across the board uh sign uh significantly.

But there was uh this new challenge that we encounter when we try to scale it up further. when we try to scale muon for a

further. when we try to scale muon for a one trillion parameter model we encounter a new issue uh about training instability. So as you can see on the

instability. So as you can see on the left figure we uh we observe that the max logits quickly explodes and quickly

exceeds 1,000 and the typical values for uh for uh training for this max logics is about say 50 or maybe less than uh

100 but for uh for m it quickly exists 1,000 and at the same time we observe training uh divergence on the left hand side. If you look at the trading loss,

side. If you look at the trading loss, uh it goes down a bit but then at the end of the day it explodes and it cannot converges uh as expected. So this is one of the technical challenges that we have

to to address. Uh and the solution to this is to introduce this new technique called uh QK clip. So basically what it says is that for each attention hat in

this entire neuronet network we're going to in the forward pass we're going to compute the max logit and then we're going to calculate a dividing factor

that can be applied to each uh key projection as well as the query projection so that we can uh sort of clip the maximum uh value of the query

and the key to to sort of constrain it into a a given range so so that uh we're not going to have exposion anymore. Uh

so these are some of the empirical results. On the left hand side there are

results. On the left hand side there are two curves but there are strictly overlapped with each other. So these are the training curves before and after applying the clipping technique. So you

can see uh the clipping technique does not affect uh the training loss uh uh uh decrease at all. But on the right hand side if we inspect uh the intermediate

metric if we inspect the max logit it's going to be effectively uh constraint.

So it first expose as before but at the value of 100 it's going to be clip at a constant value for a long time and then after a certain number of steps will

just naturally go down. uh so the neural network sort of uh find a way to uh constrain the maximum value of the uh

max logic to ensure a stable uh training process and at the same time it doesn't affect you know the training convergence as shown in in the in the left figure.

So we employed this technique in our uh K2 model training and successfully scaled it to one trillion parameters.

And this is uh the first example of a large scale mong training in the history of machine learning.

And the second dimension that we're very interested in is isl context. So this is another figure. It's probably less

another figure. It's probably less known. uh is is one of the hidden gems

known. uh is is one of the hidden gems in these papers. Uh so instead of just you know pushing down the training loss by training on more tokens, it has some

uh uh insights from another perspective.

So as we can see this is a comparison between transformers and LSTMs. So on the left hand side we can see that transformers achieve a lower training loss given the same number of parameters

and the same number of training tokens as expected. And this is why

as expected. And this is why transformers become you know the you know the sort of the de facto architecture that people are using right now. But on the right hand side it's

now. But on the right hand side it's really interesting to see that transformers are actually better because it can improve through the whole context. So the x-axis is the token

context. So the x-axis is the token index in context. And if you increase the token index, you can see that the training loss of transformers actually

drop by a lot. If you just continue continually increase context lens, the loss just continuously drops down. But

if you look at you know the curve of RSDM, it just is saturated after a certain uh number of tokens. It means

that transformers have this better capability of capturing longer context and this is this is what makes it uh better uh because if it if you go back

to like 10 years ago uh people use LSTN for tasks like machine translation but it is not good for for example understanding entire codebase or running

a super long agent trajectories to solve uh a a uh for example writing Linux kernels uh from scratch. is not going to be accomplishable by STMs. So this is a

very uh much neededed capability in the era of agents because tasks are becoming harder and harder uh and we need longer and longer context. Uh so the research

idea here is to develop a better architecture so that we can uh efficiently scale to a longer context length and at the same time achieve a

lower per token loss at uh larger token indices. And this is the motivation uh

indices. And this is the motivation uh for which we introduce this new architecture called uh Kim linear. And

it's uh contains this new uh linear attention variants called uh Kimmy delta attention which improves the original

uh delta rule GDR uh by improved recurrent memory. I will show the

recurrent memory. I will show the details later. And at the same time,

details later. And at the same time, we're going to mix linear attention layers with full attention layers using a one to three ratio so that you can balance between uh this long context

capabilities and at the same time having a more efficient uh implementation.

So this is uh some of the formulation.

Uh the basic idea is simple. If you look at linear attention uh in the original formulation uh the memory is going to be global. So there is a global single

global. So there is a global single decay factor that is applied along the way. So it means that uh basically if uh

way. So it means that uh basically if uh there are only two cases in one case is in one case you're going to forget basically everything and you're not going to retain any information and in

the second case you can choose to retain you know almost everything but at the same time you don't have the capability to leave out some of the unnecessary information in this long context. So we

introduce this key idea of having a fine grain uh decay factor as shown in this highlighted alpha term. So it's going to uh instead of being a scalar, it's going

to be a a diagonal matrix which controls uh the decay rate for each channel. So

that we can have two possibilities. For

some of the channels, we can uh decay really really slow, meaning that we can retain this long context information across a very long uh range. And at the

same time, for the other channels, we can sort of quickly forget uh the information from the past indices to refresh it and observe new information.

And this is uh to increase the expressivity uh of this model.

And of course to leverage modern GPUs we have to use this chunkwise uh formulations so that we can uh paralyze the computation on modern GPUs. So the

first equation here is the chunk uh chunkwise um formulation of uh Kimaria.

But as you can see this is going to bring massive infrastructure uh challenges because of this newly introduced alpha term. Uh because now it

is a matrix instead of a scalar it cannot easily be factored out. So to

achieve a efficient implementation uh we rewrite the entire equation into the the bottom three equations. Uh so it we introduced this matrix inversion

operation as well as introducing uh the uh cumulative decay factor so that we can implement this entire thing in parallel without sacrif sacrificing uh

any efficiency and more importantly this is not an approximation. is an exact uh mathematically equivalent formulation so that we can achieve much efficient

implementation without sacrificing uh any loss in terms of uh performance. So

it's going to be as efficient as uh you know previous linear attention variance but at the same time much more expressive. So these are some of the

expressive. So these are some of the results that we obtain uh using a fair comparison. So on the left hand side we

comparison. So on the left hand side we see the performance on two different types of task. So MMAU is a short context task. So for short context task

context task. So for short context task uh Kim achieved a better performance compared to MLA and GDN. And at the same

time for longer context task such as ruler uh Kim linear is uh also better than the varants uh the other variants while being much more efficient compared

to MLA.

And when we scale the contact lens further to for example 1 million tokens or even longer uh it's going to be much more efficient uh compared to uh the

baselines. And this is also uh the first

baselines. And this is also uh the first architecture that can outperforms full attention across across the board uh including short context task uh long

input task and long output task.

So these are two uh key dimensions that uh we are interested in and the third dimension uh is the agent swarms. So

here's a diagram to showcase how we design this agent swarm paradigm to solve some of the more complex task

compared to single agent paradigms. So here we have an orchestrator or you can call it a main agent. It's responsible

for orchestrating task. It has different options. For example, we can spawn uh a

options. For example, we can spawn uh a group of sub aents and assign new task to these sub aents or you can collect the results uh from the return of these

sub aents and you can sort of uh performing this process in an iterative way and at the end of the day you can ac accomplish a more uh complex task

compared to using one single agent and it's analogous to to human society. For

example, if we build a a company, uh we need different roles and we need for example uh orchestrator or maybe we need a CEO to to decompose and assign the task to different rows and then at the

end of the day uh the entire organization is going to have uh to move towards this same goal. And here for example in this case we have maybe you

have the AI researchers uh you have the web developers you have physic researchers and they can study different topics and at the end of the day you just collect the results and spawn uh a

group of fact checkers and web developers and file downloaders to um to assemble the results to into a a a single report.

And this is another pers perspective to to look at uh this new paradigm. So the

x-axis is the complexity of the task uh and the y-axis is is the execution time and the complexity of the task is

measured by the accuracy of uh a group of models uh on on such task. So we can see with agent swamps it's going to uh

substantially uh uh in uh reduce uh the execution time uh compared to uh uh compared to single agents. Uh it's going to be more effective. Uh and this means

that we can scale uh this agent swarm paradigm to uh for example if you run this uh agents roams with 100 or maybe even 1,000 sub agents you can accomplish

a complex task within a certain period of of time that is tolerable for um for to producing real economical value

and we can we can certainly scale it in different dimensions. We can scale the

different dimensions. We can scale the inputs. For example, we can download and

inputs. For example, we can download and read hundreds of sources or even maybe thousands of doses in parallel or you can output uh write a 100page literature

review uh in uh in parallel or you can take actions at scale. You can perform data analysis uh for 10 different tasks and also it is orchestration at scale.

You have to learn to design subtask and aggregate the the results.

And technically uh we define some new objective functions to guide the learning process of our agents realm system. So there are three rule

system. So there are three rule functions uh reward objectives that are uh considered here compared to uh the

conventional single agent IO learning.

So the first term is what we call the instantiation reward. It incentivizes

instantiation reward. It incentivizes sub Asian instantiation to prevent uh uh this uh serial uh collapse uh phenomenon

from happening. So basically we don't

from happening. So basically we don't want it to default to single agent execution. We want to encourage uh the

execution. We want to encourage uh the parallel executions especially uh when we when when it's early stage in training. uh and of course we can decay

training. uh and of course we can decay the weight for uh this instantiation reward time over training course because uh when it learns to uh learns uh par

execution we can reduce uh the weight and the second term here is is finish reward uh and it is used because we observe one of the things in training uh

that uh some of this uh subtask are just created but never finished. So it's

almost like it's going to hack the first term by just spawning a bunch of sub agents and the task might be too complex or maybe the task just doesn't make sense. And here we use this finish

sense. And here we use this finish reward to basically encourage that each of the subtask should have a relatively

high ratio of uh completion instead of just spawning a bunch of uh pseudo tasks. We need it to be uh meaningful.

tasks. We need it to be uh meaningful.

Uh so this is the second term that we use and of course we use the same you know decay strategy. We use a relative highway at the beginning of training and we decay it to a relatively low weight

at the end of training. And of course the third term is the standard term uh is is the outcome reward. Uh it's going to measure whether the entire task uh is

completed. Uh and then we're going to

completed. Uh and then we're going to add these three terms in our uh reinforcement learning uh system. And of

course we have to build you know the entire infrastructure because uh right now you need to support the parallel execution and then you need to support different reward functions uh and and to

you know maximize the efficiency of the entire agent swarm IO system.

So here are three uh different things that that we have uh tried scaling. Uh

the m clip optimizer uh improves token efficiency and Kimmy delta attention uh in the Kimmy linear architecture improves long context and we also have

the agent swarms paradigm to further uh create a new dimension of scaling and all of this put together uh we created

uh Kimik K2.5 a new model that we just released uh over one month ago. Here's a

short video to demonstrate some of its capabilities.

Heat. Heat.

So yeah, there are a lot of interesting things capabilities that we discover from the model. For example, uh it merges the visual capabilities with

coding capabilities. So a lot of new

coding capabilities. So a lot of new things just emerge out of it. It can

read a video and then produce a website that uh sort of replicates or style transfer the original video and all of

this are due to successful uh and stable training uh at the pre-training stage.

So this is also one of the most beautiful curves that uh I observed in my life. Uh so this is the training

my life. Uh so this is the training curve of the K2.5 based model. Uh so as you can see it went through over 15 trillion tokens and of course in K2.5 we

additionally train another 15 trillion tokens and the entire uh train uh the training process is just so stable.

There's no loss spike especially when we introduce this new m optimizer. We

didn't observe any spike and this smooth stable training process produces a very stable outcome that uh a very strong base model that we can fine-tune on top

of it uh to achieve you know new capabilities as as we introduced and saw in the video uh the video and this is

also of course uh trend on Nvidia H800 GPUs and each node in this H800 cluster contains two uh TB RAM and AGPU use uh

they are conducted by MV link and uh one of the another you know key innovation of K MK2.5 is that it is the first open

model with native joint vision uh text capabilities. So if you look at previous

capabilities. So if you look at previous open models, usually there are visual capabilities added on top of a text space. Meaning that for example, if you

space. Meaning that for example, if you train the text models for 20 trillion tokens and then on top of it, you do another two trillion sort of a post

training process to add additional visual capabilities on top of it. Uh but

for K2.5, it's different in the sense that we feel the training process of vision and text from day one. So it's

called early fusion here. We start from you know 0% of the progress. So from day one we're going to merge uh the vision and text tokens and as shown in our pre

preliminary experiments it outperforms uh late fusion and some of the new capabilities that we observe also come from this training recipe. Uh for

example, if you want to do vision to code, you really have to merge vision and text into a single brand to achieve that. If you separate these two brands,

that. If you separate these two brands, it's not going to happen. You have to align these two modalities into a share embedding space uh entire uh a share

representation space so as to uh achieve this. And another interesting thing that

this. And another interesting thing that we observe is that these two modalities can actually enhance each other. So

that's uh that's been long been a challenge that if you add vision capabilities into a text model, it's going to somewhat uh uh hurt the text

performance. But here we found that if

performance. But here we found that if you train it properly, these two modalities can actually enhance each other. So this is one of the key

other. So this is one of the key findings that uh we observe in in in our training. Uh so first vision improves

training. Uh so first vision improves tax. So this is so interesting. So

tax. So this is so interesting. So

before vision RL the performance uh in the first column and then we have the performance after vision RL. So here

vision RL refers to a process that we only use vision task. So there is no text task involved here. We only have vision task. For example, we teach the

vision task. For example, we teach the model how to how to count uh how to answer some of this visual QA uh problems uh without any for example math

any coding problems in in in this space.

But we observe that it's going to improve the performance for even you know reasoning heavy text task and on the other hand text also improves vision. If you have a very strong text

vision. If you have a very strong text base, you you actually don't need any vision SFT data in the training process.

And this is the approach that we adopt.

So it's called zero vision SFT. Uh

basically we don't have we have basically zero vision SFD data. And the

only SFT data that we have is the text SFT data. And then we do a joint IR over

SFT data. And then we do a joint IR over text and vision. And you can see that we can achieve uh almost state-of-the-art performance across the board on on vision task without any vision data. So

it it's clear that uh if you have a strong text base is also going to improve uh the vision if if you uh align these two modalities into a shared space

in your in your pro training.

And also uh these are some of the examples of uh uh yeah as I was showing the video. So

it's it demonstrates strong capabilities of uh uh visual design and front end coding and this also emerges from our uh vision text for training.

So uh after all this so this all about Kim K2.5 and as probably uh you probably know we released our new architecture uh

yesterday uh in our tech report it's called attention residue. So here I'm also going to briefly talk about our new work uh which serves as a a sneak peek

into our next generation architecture that we're probably going to adopt in in our later models. Uh so here the motivation is is quite simple. Can we

apply some of our techniques that we use in in the in the temporal dimension and we we just take some of the inspirations

and we apply to the death uh dimension.

So uh and it starts from this residual connection. So I still remember uh

connection. So I still remember uh listening to to uh Kami's talk uh at a tutorial in in ICML 2016 10 years ago.

Uh so it was a brilliant idea. So

basically before RestNet uh nobody was able to train deep networks. If you

increase the depth, if you increase the number of layers for neural networks, nobody was able to train it because you observe this gradient exposure, gradient vanishing, all this, you know, stability

issues. But then after the introduction

issues. But then after the introduction of of RestNet, we can train, you know, an arbitrarily large number of layers.

You can stack as many layers as as you want and you you you don't have to worry about the training stability issue uh and stuff. And as discussed in Ilia's

and stuff. And as discussed in Ilia's talk uh two years ago, it basically says that residue connection is a variant of

RSTM but just rotated 90 degrees. So how

do you understand this? If you look at LTM is is a variant of recurrent net, right? And it's a recurrent u model

right? And it's a recurrent u model process. So we're going to take the

process. So we're going to take the hidden states from the last step and then we're going to have some gating mechanism some function to produce the

current state right and if you look at the uh the depth dimension residue connection is basically the same we're going to take the output from the last

layer and then we're going to apply some sort of function on top of it to produce uh the current uh uh the current output of the current layer.

It's just the formulation is different.

For example, for residual connection, we're going to use a fixed addition.

We're going to have this uh we're going to add uh the previous hidden states uh with the current output. It's just the formulation that's different, but the

basic idea is the same. It's a recurrent net applied uh in the dimension of death. And but on the other hand we can

death. And but on the other hand we can think about uh reformulating uh this uh this function instead of having an STDM

can we have an attention in the dimension of that and it's going to create new possibilities because attention have has been demonstrated to be so successful uh in the transformer

era. So what we're going to do is not

era. So what we're going to do is not just to take the last hidden state, but we're going to consider all the previous hidden states and use the attention operation, the attention mechanism to

assemble and aggregate all of these previous hidden states to compute the current state. So this is exactly

current state. So this is exactly attention rotated by 90°. uh is sort of we view it as a natural generalization

of residue connections uh in the ASM analogy. Okay. And here is the detail

analogy. Okay. And here is the detail formulation. So on on the left hand side

formulation. So on on the left hand side is a standard residue as I said it's basically rotated by 90° and the second figure is

attention rotated by 90°. So what we do is to collect all the previous hidden states and have a simple attention operation on top of it to produce the

current layers outcome. Uh and of course to increase the efficiency to reduce the infrastructure for example communication and memory overhead. We also design a

new variant called uh block attention residue on the right hand side. So

basically the ideas is also simple.

We're going to divide uh all the layers in the neural networks into multiple blocks. For example, each block can

blocks. For example, each block can contain say 16 layers or it can contain maybe four layers. And then for each block, we're going to uh apply uh this

attention residue only on the output of each block. But within each block, we

each block. But within each block, we also we still adopt uh this standard residue. So this is going to reduce a

residue. So this is going to reduce a lot of overhead while having minimal uh loss in terms of training accuracy.

And these are some of the impressive results that we achieve uh uh on this new architecture. Uh so on the scaling

new architecture. Uh so on the scaling law we can uh improve the token efficiency by uh 24%.

uh meaning that if you have 50 trillion highquality tokens now you just magically have uh uh over 60 trillion uh

tokens and then for the validation loss you you can also observe that uh it's uh consistently lower than uh the original

curve u demonstrating uh this stability across optimization and also achieves the best improvement uh on some of this

coding math and reasoning heavy task as shown in the benchmark results of GPQA uh math and human val so the entire community uh keeps moving

forward uh and we're happy that we can we're able to contribute to to the community with uh you know new technologies and some of this are h you know some of these technologies have

been sort of standard and de facto for a long time but as you can see we still see a lot of opportunity unities to improve it to uh to have revolutionary

new design to achieve better performance. If we multiply all these

performance. If we multiply all these scans together, you can actually have a much better model. So, Adam was invented in 2014 uh and now we scale an open

source mirror and click a a dropping replacement for uh for Adam. And I'm

sure that uh if you train a transformer LLM uh it's going to be much better if you use Mong clip instead of atom. And

attention was invented uh over eight years ago. And then now we have Kim

years ago. And then now we have Kim linear which is a linear version. We

don't have to use full attention across all layers. Uh we can have linear

all layers. Uh we can have linear tension that performs better on short long context at the same time. and also

residue connections are now uh also challenged uh we scaled in open source attention residue. So I think one of the

attention residue. So I think one of the uh interesting things about our error is that we sort of adopt a different mindset for for doing research. So if we

go back to 10 years ago it's mostly about publishing a new idea. uh but then I think the lack of the rigor of the

experiments is very hard to produce uh solid experimental results. But now we have this scaling ladder. We have uh enough resources to you know trend the model and run it on different at

different scales. We can have you know a

different scales. We can have you know a whole set of benchmarks to measure the progress. So it is it becomes easier to

progress. So it is it becomes easier to make confident and solid conclusion out of it. And this is one of the reason why

of it. And this is one of the reason why we are observing you know new progress uh on this ancient techniques and I'm sure that we'll see more and more

especially in the open-source community.

I think we're going to have more and more even better architectural and you know optimization uh improvement in in the next few years.

All right. So to summarize, we're going to uh keep scaling our models. And so

these are three dimensions. Uh for

example, we we see uh we we see uh different architectures and optimizers that optimize all three, you know, dimensions and we'll keep you know see

new dimensions for scaling. Uh agents

ROMs is is not the end. And we are glad that we can uh move forward with the entire open source community to achieve better and better uh you know

intelligence. Thank you so much.

intelligence. Thank you so much.