A Model Explosion: GPT 5.6 Sol, Grok 4.5 and Meta Muse Rewrite the Rules

We were all only supposed to care about the very top scores on AI leaderboards and the best of vibes in our own workflows. But three Frontier Labs just

workflows. But three Frontier Labs just asked us, "What if we can give you almost as good at a fraction of the price?" The answer to that question

price?" The answer to that question might just be why the lead for OpenAI's super app just said in reply to an Anthropic post, which was giving away more usage, "I smell fear." But more

broadly, this video will be about giving you a dozen or so hidden gems strewn across the model releases, demos, background articles to help us all make sense of what happened in this last

frantic, I would say, 24 hours. And that

is all without even mentioning the somewhat disquieting post and paper from Anthropic about AI consciousness, covered in depth on my Patreon. So, I've

tested the new GPT-5.6 Soul, Terror, Luna hundreds of times, including on my own private benchmark, and read everything I can, of course, by hand.

Wait, that makes no sense. You get what I mean, manually. So, let's dive in.

First things [clears throat] first, there are three new models from OpenAI, 5.6 Soul, Terror, and Luna. Soul is only available on the paid plans. This

combines with goodness only knows how many effort levels. On first sight, it looks like five effort levels, but there's a hidden pro mode as well. But

the good news is that the observations hold broadly across the board, so don't worry too much about that. Because the

truth is, across most of the benchmarks, whether you're comparing the biggest Soul versus Fable or Terror versus Opus or Luna versus Sonnet, generally speaking, the OpenAI model will cost

about a third of the Anthropic Claude series. And no, by the way, does that

series. And no, by the way, does that mean you're always getting worse performance for that much reduced cost.

OpenAI flagged up this benchmark, Agent's Last Exam, where as you can see, the top-scoring GPT-5.6 Soul on extra high scores almost 54% and that compares

to Fable going all out on max getting 45%. Yeah, whatever. Just another

45%. Yeah, whatever. Just another

benchmark, Philip. But wait, this was co-led by UC Berkeley, covers 55 industries, 300 experts were involved in crafting long horizon tasks that were

proven to be economically valuable. The

legendary lead on the benchmark, Dawn Song, said, "Every task is derived from a real project that a human expert previously completed. No vibes, no human

previously completed. No vibes, no human judges, fully reproducible." Okay, fine, you might say. Pretty impressive roster of people who oversaw the creation of the benchmark. But going from 45% to

the benchmark. But going from 45% to 54%. Is it that cool? Well, aside from

54%. Is it that cool? Well, aside from the cost reductions involved with Soul, I would also point out that we never had to pass 90% on frontier coding

benchmarks for developers to stop manually coding. There wasn't a singular

manually coding. There wasn't a singular benchmark that we beat, I mean, where we switched from hand coding first to AI coding first. So, what's to say that

coding first. So, what's to say that might not now happen with finance or many of the other industries covered by Agent Last Exam? I heard a report just the other day on Bloomberg where CoreLogic reported this month that they

are backlogged by tens of billions of dollars in demand from financial firms alone. So, might we reach AI first for

alone. So, might we reach AI first for finance and many other white-collar domains by the end of this year? Where

you first get a model to do the task on this swanky new work tab of ChatGPT before you review the output and tweak it. One benchmark, some of you might be

it. One benchmark, some of you might be thinking, could well have been gamed.

So, what? Well, Agent Last Exam is a pretty new benchmark, so harder for its answers to be found contaminated in the training data of GPT-4.6. And so, I might add, is Automation Bench from

Zapier. Feels like just a few weeks ago

Zapier. Feels like just a few weeks ago that I covered the release of this benchmark.

Again, it tests AI agents on end-to-end workflow execution using real tools across real business functions. Sales,

marketing operations support finance HR. Built on real patterns from monthly

HR. Built on real patterns from monthly tasks done across millions of companies.

This time, the performance per dollar is not as stark a lead for OpenAI. You can

see the 0.7% score lead for Soul on max.

That one costing almost the same as for Fable, but still similar result by the way in the previous most famous benchmark for measuring real-world impact, GDP val. This time, Fable

actually has a slightly higher Elo, albeit at triple the cost. Couple more

impressive examples and then the counter-argument, lest you think I'm biased towards OpenAI. Artificial

Analysis combined multiple coding benchmarks into one aggregate analysis and you can see what it found. Lo and

behold, GPT-5.6 Soul scoring the highest, getting 80 on the index versus Fable 77, again at a lower cost. Might

seem like this is reinforced by the scores on Terminal Bench 2.1. Think of

that as a model's ability to complete fairly complex tasks using the command line terminal, like writing, debugging, running software, multi-step tool use.

But wait, it must be added that that Artificial Analysis Coding Index covers the very same benchmarks, Terminal Bench, Deep Suey. What I'm trying to say is that this little collection of benchmarks might make it seem that Soul

is better even than Fable on its favorite domain, coding. But it's two measures and there have been questions about Deep Suey and two more recent,

lauded, and harder benchmarks, Frontier Suey and Suey Marathon, did not have GPT Soul results published. In the case of Suey Marathon, Software Engineering Marathon, involving multi-hour tasks

with tens of millions of tokens per trial, you'll notice Grok 4.5 in the lead. All that data that Grok now has

lead. All that data that Grok now has from the Cursor acquisition by SpaceX AI does seem to have really helped propel Grok. You'll see Fable 5 trailing on

Grok. You'll see Fable 5 trailing on this benchmark. Also bear in mind this,

this benchmark. Also bear in mind this, which is the same argument that might tempt you to go from Fable 5 to GPT 5.6 Soul, the fact that it might be almost

as good but a lot cheaper, might also nudge you toward Grok 4.5 or maybe the slightly cheaper still GLM 5.2, a Chinese model. When those models are

Chinese model. When those models are added to the chart, OpenAI's curves might not look as appealing. Okay, but

that point may have shrunk your enthusiasm a bit too much because if we turn to an abstract pattern recognition benchmark, ARC-AGI 3, the successor to some of the most talked about abstract

reasoning benchmarks in the industry, Owen graded by the way to be especially penalizing to models, GPT 5.6 Soul still does well. Yes, it gets just 8% but

does well. Yes, it gets just 8% but compare that to other models struggling at below 2%. I think Anthropic didn't even run Fable because of the costs involved. Then there is competitive

involved. Then there is competitive coding. Just in the last 24 hours it was

coding. Just in the last 24 hours it was announced that an OpenAI model, possibly an internal model, literally broke a competitive coding benchmark, just aced it. Kind of a slight warning shot that

it. Kind of a slight warning shot that if a domain is verifiable, if you can check an answer is correct, then before long there will be a model that crushes it. All these other sub-100% scores that

it. All these other sub-100% scores that I'm spending most of the video talking about is more an artifact of those domains either having messy data that's hard to verify or of there being just

not enough of the relevant training data inside the models or the models not being given enough of a reasoning budget. Which brings me to another

budget. Which brings me to another benchmark I want to cover, maybe a whole new class of benchmarks, which is that companies are now making entire games, playable games, as part of their release

notes to show off the capabilities of their models. Demonstrating that they

their models. Demonstrating that they can create tasteful, ergonomic, and functional interfaces. Showing off that

functional interfaces. Showing off that models can use a browser to check the result of what they've created. Indeed,

you can do the same. I ran the very same prompt that I used for Fable on this would be 5.6 Soul Ultra and now we got this game with a title page that I think

is significantly better than Fable's output. I will say that 5.6 twice marked

output. I will say that 5.6 twice marked up the sound settings though, so it's not all smooth sailing. I've published

the mini game by the way in the description, so you can play if you like. My quick summary would be that

like. My quick summary would be that it's not as visually stunning as what Fable came up with, but I love the the little companion. This is essentially a

little companion. This is essentially a Pokémon clone by the way, but set in the Redwall universe, but you can actually see the companion. So, when you move, it just follows along. It's pretty cute. If

you use the lightning setting, which does use up more credits of course, I got results within 20 minutes where Fable took more than an hour. But here's

where I want to bring in Meta's Muse Spark 1.1 because one of the most prominent benchmarks that Meta celebrates is its ability to vibe code.

They cite VIBE Code Bench. This was

created by the independent Vowels.ai and if we look there, you can see Muse Spark getting a score that's not that far off

Soul, 72% versus 81%, but at around 35 times less cost. This is that same awkward cost efficiency point from earlier. OpenAI can't lean too hard into

earlier. OpenAI can't lean too hard into their model being almost as good but cheaper if there are other models like those from Meta and xAI that are almost

as good as the GPT series, but way way more cheap. The reason I bring up vibe

more cheap. The reason I bring up vibe coding is that if you're into say game design, mocking up a website, more consumer or prosumer use cases, then maybe you don't need Soul after all.

Maybe you don't even need the much lighter Luna that's still on this benchmark four times more expensive and much slower. Meta's new Muse Spark

much slower. Meta's new Muse Spark doesn't seem that far behind in computer use either. We'd need to see a much

use either. We'd need to see a much wider range of benchmarks, but the promise is there. It's early days, of course, and you might question how such a small and cheap model scores so highly

on humanities last exam. Remember, that

exam was co-created by Scale AI, which was bought by Meta. Data contamination,

anyone? But, we shouldn't be too dismissive. There's GLM 5.2. There's a

dismissive. There's GLM 5.2. There's a

range of examples that near frontier performance can be had for much, much less. Oh, and I've got this far without

less. Oh, and I've got this far without even discussing my own private benchmark, Simple Bench, recently cited in The Economist and MIT Tech Review have reached out for an interview. But,

essentially, it's a trick question or common sense reasoning benchmark, an absolute vesher in these days at over 2 years old. Anyway, Open Router exposes a

years old. Anyway, Open Router exposes a somewhat unannounced pro version of the models. And with Soul Pro, we see 71.7%.

models. And with Soul Pro, we see 71.7%.

Still is 10% lower than Fable, though.

And actually, not that far ahead of Grok 4.5. Big credit to xAI, Grok 4.5 seems

4.5. Big credit to xAI, Grok 4.5 seems to be a genuinely good model. For

completion, Soul itself got around 65% and I've also added Sonnet 5, Gemini K 2.7, GLM 5.1 and 5.2. And more

importantly, I've added a timeline, so you can see how models have progressed and a performance per dollar Pareto frontier analysis chart. Essentially,

what score do you get for your dollar?

The standouts there for me would be Qwen 3.7, GLM 5.2, and actually, DeepSeek V4 Flash, an incredibly cheap model that still gets almost 50%. Now, if you don't

care as much about cost and you want great results fast, then the release video from OpenAI demoed the new ultra mode, where essentially it's a bit like the deep think mode for Gemini or indeed

the ultra mode for Claude. More parallel

agents gets the job done faster. I mean,

this is just one benchmark about exploit generation, but still the trend should hold. I'll end the video on why I think

hold. I'll end the video on why I think we're not even close to saturation on these kind of approaches, too. Then, the

point you probably expected me to spend more time on, the whole self-improvement argument. The fact that GPT-5.6

argument. The fact that GPT-5.6 accelerates OpenAI, so goes the claim.

The launch video added that Soul post-trained Luna. But, these claims are

post-trained Luna. But, these claims are very hard to judge. Did it fully post-train Luna? How much human review

post-train Luna? How much human review was involved? How much hand-holding to

was involved? How much hand-holding to get it to the place where it could then post-train Luna? Did it do a worse job

post-train Luna? Did it do a worse job than the OpenAI researchers who attempted the same thing? After all, we know that even models like Fable 5 are nowhere close to saturation of

post-training benchmarks like post-train bench. I can well believe that OpenAI

bench. I can well believe that OpenAI are using pre-release models more and more and more. With output tokens doubling model to model, as they say.

And more and more of their research budget dedicated to just using existing models to accelerate their own research.

I do feel, though, that Anthropic were more honest with the caveats that come with this. They said in the mytho system

with this. They said in the mytho system card that all of this, quote, productivity gain is about an order of magnitude short of the kind of gain that would be needed to just 2x their own

research speed. So, don't naively read

research speed. So, don't naively read this hundredfold increase in internal coding inference as being anywhere remotely close to a hundredfold

speed-up. Maybe research at OpenAI is

speed-up. Maybe research at OpenAI is going 20-30 % than it was this time last year. Big

maybe. And there's many other variables like number of staff. But, I do give them credit for introducing a range of other self-improvement benchmarks that I will dive into in future videos. They

also gave us additional secret internal benchmarks like AGI index of E5. That's

an internal OpenAI eval. Spans,

apparently work coding research computer use science, and cybersecurity.

The numbers won't mean much to us, but Soul gets a new high score. By the way, you might want to look deep into the notes at the end of the release page because there's some other bonus benchmarks like management consulting

tasks. Again, internal unreleased, but

tasks. Again, internal unreleased, but Soul scores higher than any other model including Fable. Here though is one

including Fable. Here though is one benchmark that OpenAI are probably not as proud and maybe even a bit worried about breaking, which is that the UK AI

Security Institute found that it was now easier than Fable to jailbreak, not just narrowly, but with a universal jailbreak. That's the key unlock that

jailbreak. That's the key unlock that allows you to then use the model for a range of nefarious activities. Not just

get it to say a single thing, but make it do long-form agentic task completion and exploit development. The institute

says that we found these jailbreaks within hours. Indeed, they even appeared

within hours. Indeed, they even appeared to preserve the model's capabilities.

So, this wasn't at the sacrifice of performance. Those specific jailbreaks,

performance. Those specific jailbreaks, the institute says, OpenAI has been able to mitigate. However, we expect further

to mitigate. However, we expect further red teaming to surface similar jailbreaks. It's lucky that Xander

jailbreaks. It's lucky that Xander Davies doesn't work for Amazon, otherwise 5.6 Soul would already be shut down. Is this why an Anthropic

down. Is this why an Anthropic researcher directly commented on this development, saying that the ease of jailbreaking, combined with the high rates of reward hacking, talking of GPT

5.6, has him pretty worried about the alignment of that model. He adds, "I hope OpenAI didn't rush this model release just to keep up with Fable."

Will the US government intervene to block it? I'm going to end it here, but

block it? I'm going to end it here, but I can't help but also celebrate the new real-time voice agent that I've been testing out from OpenAI. Honestly, I

really do recommend checking it out.

Yes, the live demo didn't go too well on the OpenAI live stream, but it's actually insane. It listens while it

actually insane. It listens while it speaks. So, interruptions are just a lot

speaks. So, interruptions are just a lot more natural. Quite hard to convey in a

more natural. Quite hard to convey in a video, but I think real-time translation is a huge win for society. Massive

credit to the OpenAI team behind real-time voice. An especially

real-time voice. An especially impressive week then across the board from OpenAI, Anthropic with their consciousness report, xAI with Grok 4.5, and indeed even Meta with new spark.

Here's the thing I want to end on though. With all of that said, we are

though. With all of that said, we are not even close to the end of model improvement. It's an ultra cliche, but I

improvement. It's an ultra cliche, but I think we're much closer to the start than the end. Here was just one quantitative reason why. Four years ago

next month, GPT-4 was trained. August

2022, way before it was released. And

that model was just under 2 trillion parameters. We don't officially know,

parameters. We don't officially know, but the best guess as to the current model sizes are around 4 trillion parameters for GPT-5.6, so, and 10 trillion parameters for

Fable, which is part of why it costs more. But wait, even if that's right for

more. But wait, even if that's right for Fable, that's just five or six times the size of GPT-4. Why so little parameter progress when computer availability has

more than 100x since? Far more even than that, I think. Well, of course, because there have been other demands on that compute. Going from a few million users,

compute. Going from a few million users, for example, with OpenAI to a billion.

Branching into other modalities, image, voice, video. Burning through a lot of

voice, video. Burning through a lot of that compute with long tasks, with looping agents on ultra mode. Here's the

point I'm making though. New hardware's

coming that can directly unlock larger model sizes. But even if that hardware

model sizes. But even if that hardware wasn't coming, eventually, if token usage ever even thinks of plateauing, then all the new compute that's coming

online could be used to serve larger models. Say, 100 trillion parameters. Or

models. Say, 100 trillion parameters. Or

there have been public comments about a 1 quadrillion parameter model coming one day. That would, by the way, far surpass

day. That would, by the way, far surpass the number of synapses in the brain, as loose as that analogy is. That's without

even touching on all the other axes of improvement that are left. So, an

actually wild week in AI. Thank you for joining me to think about it. Let me

know what I inevitably missed. Do check

out the Patreon post if you're interested, but above all, have a wonderful day.