Learning to Reason with LLMs
Scaling test-time compute through search and reinforcement learning dramatically improves LLM reasoning capabilities, offering a viable alternative to traditional pretraining scaling walls.
Highlights
7 clips · 14 min of 52 min-
- Adding 30 seconds of search reduced distance from Nash equilibrium significantly.
- Search provided the equivalent of scaling parameters by 100,000x in poker.
-
- Cultural factors in game theory favored precomputed Nash equilibria over search.
- Academic competition constraints penalized test-time compute usage.
- Researchers drastically underestimated the performance gains of search.
-
- Raw neural nets alone in Go were not superhuman without search.
- Monte Carlo tree search provided massive elo gains equivalent to huge scaling.
-
- o1 scales test-time compute to dramatically improve math competition performance.
- Combining reinforcement learning on chain of thought with test-time compute yields compounding gains.
- Test-time compute scaling on challenging math exams drives accuracy from 20% to over 80%.
- Applying reinforcement learning to generate a chain of thought creates a compounding performance effect as training compute increases.
-
- Reasoning problems like math and Sudoku have a property where verifying a solution is much easier than generating one.
- When models develop implicit verification capabilities, they can recognize poor paths and actively search for correct solutions.
-
- Research in board games shows that increasing test-time compute can directly substitute for training compute scaling.
- Inference scaling provides an escape hatch against approaching pretraining data and cost walls.
-
- Classical logic and SAT solvers are efficient for specific domains but often lack broad generality.
- The true value of o1 is its ability to generalize reasoning across completely disparate real-world tasks unlike prior domain-specific search models.
That’s all the highlights.
Questions this video answers
6 · every answer cites the moment it was said01 Why did early poker AI struggle against top human professionals despite massive parameter scaling? 3 sources · 3:16
Early poker bots relied on massive pretraining through self-play and acted instantly, but lacked the ability to think carefully during difficult decisions. Human professionals, conversely, would pause and deliberate on complex problems, allowing them to outperform the instant-acting AI. Adding a search mechanism to let the bot think for 30 seconds provided the equivalent performance boost of scaling parameters by 100,000x.
02 Why was search largely ignored by the poker AI research community for years? 3 sources · 10:24
Cultural factors in the game theory community prioritized precomputing Nash equilibria rather than using search. Additionally, scaling inference compute made experiments painfully expensive and slow, especially given poker's high variance. Researchers were also heavily incentivized by competitions that enforced strict time limits on test-time compute.
03 How does simple consensus fall short when scaling inference compute in large language models? 3 sources · 18:48
Consensus involves querying an LLM multiple times and taking the most common response, which yields initial accuracy gains on math benchmarks. However, the performance quickly flatlines after querying the model multiple times, failing to provide additional returns. Thus, simple consensus is limited as a generalized method for scaling inference compute.
04 Why does inference compute work so well for certain reasoning tasks? 2 sources · 29:08
Inference compute succeeds on tasks where verifying a good solution is much easier than generating one, such as math proofs, coding, and puzzles. Large language models can implicitly perform verification during their chain of thought to recognize mistakes and try alternative strategies. This generator-verifier gap mirrors techniques successfully used in games like chess and Go.
05 Can scaling inference compute help bypass traditional pretraining scaling walls? 2 sources · 33:19
Increasing inference compute offers a viable alternative to escalating pretraining costs and data requirements. Research shows that increasing test-time compute by 15x is equivalent to a 10x increase in training time compute. While inference costs and latencies will rise, users are often willing to pay that price for breakthroughs on difficult problems.
06 What is the primary advantage of o1 over domain-specific planning techniques? 2 sources · 47:05
While domain-specific planning algorithms can achieve superhuman performance in restricted environments like poker or Hanabi, they often fail to generalize outside those domains. In contrast, o1 provides a general-purpose reasoning technique that successfully applies across diverse complex tasks. This broad applicability makes it a powerful tool that can save users days of work.
Section-by-section summary
11 parts
Scaling parameters alone failed in poker
Early poker bots relied on massive pretraining and self-play scaling without test-time search. Despite increasing parameters year over year, these bots still lost heavily to top human professionals.
Test-time search outperforms massive parameter scaling
Adding a search algorithm to let the poker bot think for thirty seconds drastically improved its strategy. This test-time compute yielded performance gains equivalent to a 100,000-fold increase in model parameters.
Cultural norms and competition rules hindered search
Game theory traditions and rigid competition constraints discouraged researchers from exploring test-time search. Furthermore, the community severely underestimated the massive performance multiplier that search could provide.
Monte Carlo tree search powers AlphaGo Zero
AlphaGo Zero achieved superhuman performance not through raw neural network scale alone, but by combining it with Monte Carlo tree search. Rules of thumb suggest bypassing search would require astronomical scaling to match those results.
Scaling inference compute in large language models
While consensus sampling offers limited improvements, OpenAI o1 introduces a more general approach by using reinforcement learning for chain of thought. This enables effective scaling of test-time compute, resulting in massive accuracy gains on complex reasoning benchmarks.
o1 shows massive jumps across diverse complex domains
The o1 model achieves breakthrough scores on expert-level benchmarks spanning competition mathematics, coding, and PhD-level science questions. These performance boosts extend surprisingly beyond STEM into subjects like the LSAT and literature.
Chain of thought enables systematic error correction
Unlike prompting models to output answers immediately, o1 uses large-scale RL to produce a prolonged, high-quality chain of thought. This allows the model to break down complex problems, test multiple strategies, and actively correct its own mistakes.
Verification gaps make inference compute effective
Reasoning tasks like math and coding share a property where verifying a solution is much easier than generating one. When models develop implicit verification capabilities, spending more inference compute allows them to systematically search for correct solutions.
Inference scaling bypasses traditional pretraining walls
Research in board games shows that increasing test-time compute can substitute for training compute scaling. This effectively bypasses concerns about hitting pretraining data walls, justifying higher latency and cost for high-value problems.
The bitter lesson favors compute-scaling over heuristics
History shows that general algorithms scaling with compute consistently beat clever human-crafted heuristics. Academic research must shift away from short-term publication hacks toward techniques that effectively leverage massive computational scale.
Q&A explores academic barriers and search generality
Discussion addresses the stark compute disparity between industry and academia, as well as comparisons to classical logic and SAT solvers. The speaker emphasizes that o1's primary advantage is its broad domain generality rather than narrow search rules.
Connects to your library
by shared topics
n next · p previous clip · t highlights/full
< > slower/faster · / search what was said
-
Counterfactual regret minimization1× · first 2:14
The underlying algorithm used by early poker bots trained via self-play.
-
Monte Carlo tree search4× · first 13:13
A search algorithm combined with raw neural nets to leverage extra computation at inference time in Go bots.
-
Chain of thought7× · first 21:43
A step-by-step reasoning process in natural language generated by a model before providing a final answer.
-
Test-time compute18× · first 10:56
Additional computational resources expended during inference to improve model reasoning and problem-solving.
8,694 words · about 35 min to read
SPEAKER: Noam worked on Cicero, which was the first AI to achieve human level performance in Diplomacy. And also, he built the first AI to be top poker players a long time before. Noam is clearly an expert on reasoning and gameplays. And we're really excited to hear what he has to tell us about reasoning with LLMs. So take it away. NOAM BROWN: Thank you. OK hi everybody. My name is Noam Brown. Mostly, I'm going to talk about our o1 model that came out of OpenAI recently, but I also want to give a broader perspective on reasoning in AI in general.
To start with, I should say-- so I'm going to give this talk. And I'm also going to talk about some of my background working on reasoning in games. The o1 model is the product of a big team effort. All these people were working full time on this for a very long time, me alongside them. There are actually more people that should be in this picture that are not because, actually at the time that this was filmed, they were busy running models for more experiments and couldn't step away.
So there's a lot that goes into this. And so by no means-- I don't deserve nearly as much credit as I tend to get for this kind of stuff. It's just like, if you have some successes before, people look to somebody that has a lot of Twitter followers or something to get credit for this kind of thing. So anyway yeah, the whole team deserves
Scaling parameters alone failed in poker
a lot of credit for this. So I want to start by talking about my experience when I was in grad school. I started grad school in 2012. And I was working on AI for poker. And at the time in the AI poker community, people felt like they had figured out the solution to getting to superhuman poker bots.
So they had this outcome that worked, and all they needed to do was just scale it up a ton. And there is this thing called the Annual Computer Poker Competition. And every year, all the different research labs that worked on AI for poker would get together, and they would play their poker bots against each other to see which one was best. And because people felt like they had the algorithm figured out and they just had to scale it up-- and by the way, the algorithm is called counterfactual regret minimization.
But that's actually not really going to be the focus of this talk. Because they felt like they had this figured out and they just need to scale it up, the Annual Computer Poker Competition really turned into a competition of scaling these models farther and farther. So when I started grad school in 2012, here you can see I have this plot of the number of parameters over time that went into these models. So when I started in 2012, it was still pretty small. 2013, it went up like 5x, went up another 5x every year.
And every year, the models get better. And every year, they beat the previous year's models. And in 2014, we actually-- the Carnegie Mellon team-- won the Annual Computer Poker Competition. And we decided to play the bot against top pros to see how it would do. So this was the 2015 Brains Versus AI Poker Competition. We challenged four top poker pros to an 80,000 hand poker competition. And there was $120,000 in prize money to incentivize them to play their best.
And our bot, Claudico, ended up losing by a pretty sizable margin, 9.1 big blinds per 100. That doesn't mean much to you unless you're a poker player, but it's a pretty substantial amount. And so that was really disappointing. But when we were doing this competition, I noticed something interesting. You see the way these bots were trained they would spend three months or so doing self-play leading up to the competition on thousands of CPUs.
So you would just have the bot for months playing through about a trillion hands of poker in self-play and learning this really complicated strategy. And then when it came time to actually play the game, they acted instantly 10 milliseconds per action at most. And the humans, on the other hand, they had only played maybe 1 million or 10 million hands of poker in their life.
But when they were faced with a very difficult decision, they would sit there and they would think. And they would think for a few seconds. They would think for a few minutes if it was a really difficult decision. But they would sit there and think instead of acting instantly. And it made me wonder if that ability to think about the difficult problems
Test-time search outperforms massive parameter scaling
was what allowed them to do much better than AI. So after the competition, I decided to look into how important this ability to think before making a decision was. So here, I have two lines. So the x-axis is the number of buckets. This was before neural nets really took off.
And so we were actually using k-means clustering. So this is the number of clusters times some constant. And the y-axis here is distance from Nash equilibrium. Again, this isn't really a poker talk. So the important thing to know is that, as this number goes down, you get a better poker strategy and your poker bot is better. And so the lower that number is, the better. And you can see-- let's start with the blue line.
So the blue line is that there's no search. It's just you have the bot acting instantly at all decision points. As you increase the number of parameters by 100-fold, you get a pretty nice reduction in distance from Nash equilibrium. So you cut it down by about a half.
1 Highlight · Search vs. Parameter Scaling in Poker
And that's great. It's a much better bot. And this is what happens if you add search. And specifically, what I mean by search in this case, there's a domain-specific poker algorithm that allowed it to compute a better strategy when it reached the end of the game. And so when it reached the last phase of the game, it would think for about 30 seconds and compute an even better policy.
So this is what happens if you add search. And you can see just by having it think for 30 seconds when it reaches the end of the game and computing a better policy, you've managed to reduce the distance from Nash equilibrium by 7x, even at 200 buckets. If you were to extend this blue line and see how far would you have to scale up the number of parameters to match the performance that you're getting by search, you would have to scale up the number of parameters by 100,000x.
So just by having it think for 30 seconds, you're getting the equivalent of scaling up by 100,000x in poker.
Now, when I saw this, I was shocked. Over the course of my PhD up until this point-- this was three years of PhD work-- I had figured out how to scale up these algorithms by about 100x, which I was very proud of. I think that was great work, and I wrote several papers on that. Here, after seeing this, I realized that was all going to be a footnote in my thesis. And the only thing that really mattered was scaling up search.
So after seeing this, we changed the approach to our poker bots completely. And it all became about how to scale up search. By the way, we're going to take questions at the end. Yeah. OK. So yeah, this led to the 2017 Brains Versus Poker AI Competition. Again, we played our new bot, Libratus, against four top poker pros. We had $200,000 in prize money this time.
And they played 120,000 hands to reduce variance as much as possible. And the bot won by 15 big blinds per 100, whereas our previous bot in 2015 lost by 9 big blinds per 100. This was a huge shock to everybody involved. It was a huge surprise to us. It was a huge surprise to the poker community. It was huge surprise to the poker players. And in fact, I think what really illustrated this is, when we announced the competition, the poker community decided to do what they do best and decided to gamble on who would win.
And so they set up this betting market on the outcome of the competition. And before the competition started, the betting odds were about 4 to 1 against us. After we had won the first three days of the competition straight, it was still about 50/50. And by the eighth day, you could no longer bet on the outcome of the competition. You can only bet on which human would lose the least on individual days. So I think this really illustrates just the power of being able to plan, to think before acting.
We followed this up in 2019 with another poker bot that played six player poker this time. So the previous bots were two-player poker. Again, we played 10,000 hands over the course of 12 days. We used variance reduction techniques this time to reduce the luck factor. And again, our bot won with statistical significance. What's really special about our 2019 Pluribus bot is that, despite going to a more complicated game of six-player poker, it was actually a lot cheaper. So Pluribus only cost less than $150 to train if you were to train this on cloud computing services.
And it ran on 28 CPU cores at test time. The reason why it was so cheap is because we figured out how to scale up and make search even better. And I think what this really shows is that these poker results wasn't due to breakthroughs in computing technology. This all could have been done in the '90s if people had taken the right approach.
Cultural norms and competition rules hindered search
Yeah. And so I think this raises an obvious question about, well, why wasn't search considered important in poker before? And by the way, when I say search, I'm going to use the term search and planning kind of interchangeably. So I've used search as a form of planning. The really important part is that search, planning, all these techniques just spend extra compute at test time or inference time instead of acting instantly according to a precomputed plan.
So if search was so powerful in poker, why wasn't it considered earlier? And there's a few reasons. One is cultural factors. A lot of the poker AI community came out of the game theory community. And in the game theory community, you want to have a Nash equilibrium or see how close you are to a Nash equilibrium. And if you don't precompute everything, then you can't say how far away you are from a Nash equilibrium. So there were just cultural factors that discouraged researchers from pursuing this direction.
And I should say, it's not like researchers weren't working on search at all in poker. It's just there were people working on it. They did make some progress. But it really wasn't given the attention that I think it really deserved, considering how important it ended up being. So one was cultural factors. Another is that scaling test time compute, scaling inference compute makes experiments much more expensive and just more painful to work with, especially in a game like poker where you have extremely high variance.
2 Highlight · Why Test-Time Compute Was Ignored in AI Research
You have to play like a million hands to really reduce the luck factor enough that you can have a clear signal on which bot or which human is better. If you have to wait 20 seconds per hand, that's just a really painful experiment to run. So a lot of researchers were discouraged from pursuing this kind of research for that reason. Another big one-- I think this is actually really relevant for a lot of the academic community, especially in machine learning these days-- is incentives. I mentioned that all of the researchers in this AI poker community were focused on winning the Annual Computer Poker Competition.
And the Annual Computer Poker Competition was structured in a way where the bots could only think, could only use, two CPU cores at test time and, at most, 7 seconds per hand on average. Now, that really constrains how much thinking they could do at test time. And because of that, because the researchers were so focused on winning the competition, they pursued other directions that would allow them to win the competition instead of the more important goal of beating top humans.
And in fact, if you look at our bots, Libratus and Pluribus, they actually didn't even qualify for the Annual Computer Poker Competition because they used too many resources at test time. But they beat top humans, which I think is the more important result. And finally, and perhaps most significantly, I think people just underestimated how much of a difference it would make. If you had asked me before I saw my results on this how much
of a difference would figuring out search make in poker, I would have guessed something like 10x. It would be the equivalent of scaling up the models by 10x. I did not expect it to be 100,000x. And I think a lot of people felt the same way. This pattern is not unique to poker, by the way. So the same story also holds in Go.
Monte Carlo tree search powers AlphaGo Zero
This is a figure from the AlphaGo Zero paper. And AlphaGo famously beat Lee Sedol in Go. It's a landmark achievement in AI. One thing that's underappreciated, I think, about AlphaGo is that it used a search algorithm, Monte Carlo tree search, when it was playing. So this is a follow up to AlphaGo. It's AlphaGo Zero. Here, we have the different bars are different versions of AlphaGo and different bots.
The y-axis is elo rating, which is a way to compare different bots and bots to humans. And here, for example, this bar is AlphaGo Lee. You can see it's around 3,600 elo. And superhuman performance is around 3,500 elo, so it's just over that line. AlphaGo Zero is around 5,200 elo, which is much better. But like I said, AlphaGo Zero and AlphaGo used Monte Carlo tree search. They're a combination of a raw neural net plus this Monte Carlo tree search algorithm that allowed it to leverage extra computation and inference time.
So it would think for about 30 seconds before making a move. So what's the performance of just the raw neural net? Well, it turns out it's around 3,000 elo, well below top human performance. So what this means is that the raw neural net of AlphaGo and AlphaGo Zero and all these Go bots are not superhuman. And in fact, AlphaGo happened in 2016.
It's now 2024. Even now, nobody has made a neural net that is superhuman in Go. Now, when I tell this to people, their response is that, well, surely you can scale up this algorithm, the neural net and the training. And you could make it stronger to the point where it's superhuman in Go. And the answer is yes, in theory, or at least in practice. But how much would you have to scale it up by in order
3 Highlight · Inference Compute in AlphaGo Zero
to reach the performance that you would get by adding Monte Carlo tree search and getting the full AlphaGo Zero bot? So there's this rule of thumb in Go that, if you want to increase your elo by 120 points, then you either need to scale up the model size and training by 2x, or you need to scale up the test time compute by 2x. If we follow this rule of thumb-- and there's a lot of asterisks around this that it's probably not perfectly accurate.
If you follow this rule of thumb, then to scale up the raw policy net from 3,000 elo to 5,200 elo, you would need to scale up by 100,000x. And by the way, that's still assuming that you're doing Monte Carlo tree search during training, which, again, if you want to take that out, this number would be even more astronomical. Now, I think, in reality, the number is smaller. My guess would be somewhere between 1,000x and 10,000x because these elo ratings are a little misleading.
The 5,200 elo is calculated by having AlphaGo Zero play against earlier checkpoints. And so because it's trained through self-play, those numbers are artificially inflated. But the takeaway is still you're getting a huge, huge benefit from being able to do search at test time, like being able to leverage extra computation at test time. So these are all games.
Poker and Go are great, but they're not actually really useful in the real world. Why am I talking about this? Well, I think there's a similar story or there has been a similar story in AI up until now with foundation models, large language models. We have seen this pattern since GPT-2 and a lot of these early models that, every year, the models become bigger, train on more data for longer.
And every year, they do better than the previous year's models, a lot like what I mentioned with the poker work back in 2012 and 2015. But the amount of time they spend at inference time is still quite low. If you log on to ChatGPT and you use GPT-4 and you ask it a question, it responds almost instantly. And if you tell it, hey, you have 5 minutes to give me a better response, it's not going to know what to do with that time.
So yeah, we've been in this paradigm where pretraining costs have ballooned. And the exact numbers I honestly don't know. I wouldn't be able to share them if I did know. But I think it's fair to say that it's going north of $100 million. But the inference is still very cheap. And maybe it doesn't have to be that way. And that was the motivation, at least my motivation, for working on something like o1.
Scaling inference compute in large language models
And I think a lot of other people saw this as well and believed that there could be a different paradigm. So that leads to the question, is there a general way to scale inference compute in large language models? Now, this has been a question that the research community has been working on for a while. And there's been a lot of papers on different ways to scale inference compute in language models.
So o1 is not the first technique that has been introduced to do this. But I think it's the most general, and the most powerful, and the most significant. I think that's a fair statement to say. So you can look at previous techniques, and I think one example is consensus. This is actually the simplest way to scale up inference compute. So I think the way to think about consensus is imagine you're taking a math test. Imagine you're not as good as math as you all are.
But imagine you're taking a math test. And you have access to GPT-4, and you want to do as well as possible on this exam. What would you do? Well, you could plug the questions into GPT-4 and get a response. You can write those down. But something you could do that's even better is you could just ask GPT-4 multiple times the same question, see what the most common response is, and then write that down. So that's called majority voting or consensus, and it actually works quite well. So the Minerva paper, which managed to get the performance on the math benchmark, which was this really popular benchmark for measuring the ability of large language models to do complex math, managed to get their performance from a 33.6% accuracy on this benchmark to 50% on this benchmark by using consensus.
Now, I have here a bunch of plots from this paper, large language monkeys, scaling inference compute with repeated sampling. And the line to look at-- so the blue line is if you submit k samples, if you query the model k times, is any of them correct? Are any of the responses correct? So that's the upper bound for what is achievable with any these sampling the model multiple times techniques.
The green line is majority voting. And you can see that, as you query it multiple times with llama 8 billion or 70 billion, it goes up initially. But then it just flatlines. So the problem with something like consensus or majority voting is that you get a lift by sampling it 10 or 100 times. But beyond that, you can't get additional returns beyond that point.
So it's not an amazing way to scale inference compute. And there's been a lot of other techniques that have been introduced that do other things, but they're all not getting amazing returns from scaling inference compute.
4 Highlight · Scaling Test-Time Compute in OpenAI o1
And now, I'll get to o1. So this is a figure that we released in our research blog post. And on the x-axis here, you have test time compute on a log scale. And on the y-axis, we have accuracy on the AIME. So this is the qualifier for the US mathematics Olympiad team. It's a very difficult math test.
All the answers are integers. And you can see that, as you scale up the amount of test time compute, the amount of inference compute in a one, you go from 20% to over 80% in this exam. And there's actually no sign of this stopping. I mean, obviously, it's not going to go past 100%. But it does seem like, if we were to push this further, you would get even more performance on this exam.
And also, importantly, this is the plot of training compute. So the way that we achieve this performance in o1, the way o1 works, is we do reinforcement learning on the large language model to produce a chain of thought. And as we throw more RL compute, more train time compute, into the algorithm, we also see the performance, when evaluated at the maximum settings of test time compute, those also increase.
So this has a compounding effect. So this is o1.
o1 shows massive jumps across diverse complex domains
I think the way to think about o1 is that it's a really effective way to scale inference compute. So what do the evals look like? Again, this is from our research blog post. And you can see, on math, GPT-4.0 gets around 13.4%. o1 preview, which is the model that you can log in to chatgpt.com now and play with, that gets around 56.7%. And then the full o1 model, which we are preparing for release, gets over 80% if you use consensus on top of the model.
In competition coding, again, GPT-4 goes from 11% to 89% percentile among humans, among very talented humans. And on this benchmark GPQA diamond, a PhD-level science questions-- these are multiple choice questions, but very, very difficult, multiple choice questions-- GPT-4.0 gets around 56%. Expert humans get around 70%. And by expert humans, I mean people with PhDs or pursuing PhDs in their subject.
And o1 gets around 78%. So, again, I want to emphasize here, I think the evals are very impressive. The more important thing is the trajectory. Here, you can see we're getting huge returns from inference compute. And we can push this further. So these numbers, I can almost guarantee you, are going to go up over the next year or two.
So more evals-- so this is from, again, various subjects. You see gains on many diverse tasks. This isn't just STEM. So for example, on the LSAT, you see improvements from 70% to 96%. What else do we have here? We even have some chemistry. What else? Language. OK, so English language, we don't actually see a boost. But English lit, we see a little bit of a boost.
We were surprised by the diversity of increase that we got. It's not just STEM. OK, so how does o1 actually work? Unfortunately, I can't go into too much detail
Chain of thought enables systematic error correction
about how it actually works, but I can give you the high level. And I think you'll be pleasantly surprised by how much detail we provided in the research blog post. We try to be relatively transparent because we think that this is pretty updating to the academic community, to the industry, research community, and to the world. And so I think it's pretty important that people understand at least the high level of how this works. But yeah, the reality is that things are progressing quickly.
There's a lot of risks in sharing the full details about how this works. The machine learning world is headed in a new direction that's moving away from the academic roots, unfortunately. There's this thing called chain of thought where, before with models like GPT 4, you could just prompt them. You could say, OK, here's a hard math problem.
And if you just ask it to give a response instantly, it would just say, you asked a really hard math problem. You asked for a response. It would just output the answer immediately. And what people found is that, if you actually prompt it to explain its reasoning, think step by step before answering, it actually improved the accuracy by quite a bit. So that's called chain of thought, where the model will output a step by step reasoning in English or natural language before arriving at the actual answer and outputting the actual answer.
So o1 uses large scale RL to generate a chain of thought before answering. So instead of just relying on the pretrained model to output this chain of thought, now we're able to optimize the chain of thought quite effectively. So this chain of thought ends up being longer. It ends up being a higher quality than what's attainable via prompting alone. And it contains a lot of behaviors that you would want to see from a reasoning model. So, for example, it can do error correction.
It can recognize when it's made a mistake and fix it. It can try multiple different strategies if it notices that one isn't working. And it can break down a difficult problem into smaller steps to tackle it in a more systematic way. And what I want to do is actually walk through some of these chains of thought. We have them on our blog post. And if you haven't had a chance yet, I highly encourage you all to look at the chains of thought on the blog post because I think that they are really illuminating.
So here, for example, we have this question that we can give the model. It's a cipher question. OK, here's this hidden code. And we tell it the output of the-- if this is decrypted, it will say, think step by step. And then we say use the example above to decode this. And if you look at what GPT-4 does, GPT-4.0, it tries to break it down.
And it reasons through this a little bit. The patterns involve selecting specific words or transforming them. It tries to decode it, but it's a really difficult problem. And so it proposes some options. Then it realizes this is just going to be too hard. And it says, could you provide any additional decoding rules to help? What happens if we give this o1 preview? Again, this is not even the full o1 model. First, what is going on here?
It restates the problem in its own words. Our task is to decode this. So think step by step, first, break down the letters. It breaks down all the individual letters, and then it counts the number of letters in each word. Our goal is to use the example above to decode this. Option one, try mapping from letters to letters.
Do any letters match? And you can see it's like doing this very, very long chain of thought, alternate letters, very, very, very long chain of thought. It sometimes recognizes that it's having success. It's sometimes recognizing that it's not working out and it's trying different things. Again, I'm not going to go through this whole thing because it's really, really long. But if you have a chance, you should check it out on the website and see for yourself.
And ultimately, what it arrives at is, OK, so it's decoded this part. And then it's finally decoded the last word, strawberry.
Verification gaps make inference compute effective
5 Highlight · The generator-verifier gap explains reasoning success
And then if we put it together, there are three Rs in strawberry. So that's what the decoded message is. So why does o1 work at all? The way that I think about it-- and I don't know if this is the best answer. But the way I think about it, there are some problems that we consider to be hard reasoning problems.
And these are problems where verifying a good solution is easier than generating one. So examples of this include, for example, Sudoku puzzles. It's really easy to verify that you have a solution to a Sudoku puzzle. It's really hard to generate a solution to a Sudoku puzzle. And so you could go through, spend a lot of time thinking about different solutions or working through different solutions. And then at the very least, you could just try a ton of them and see if any of them satisfy the criteria.
I think this is also true for math. It's much easier to verify a math proof than it is to generate a math proof and things like programming as well. There are examples where verification isn't much easier than generation. So one example is information retrieval. If I ask you what's the capital of Bhutan, I don't know. I could sit here for three years thinking about what the answer is, and it's not going-- does anybody actually know? Oh really? What is it? AUDIENCE: Thimphu. NOAM BROWN: Thimphu, OK, great.
But if I was to sit here for three years and think about what the answer is, I would not be able to get it. Image recognition, too, I would argue, generation is just as difficult as verification for something like that. Now, when a generator verifier gap exists-- that is generation is much harder than verification, and we have a good verifier-- then we can spend more compute at inference time to achieve better performance.
And this is kind of what I think happened in games Go and chess and poker. It's much easier to verify that you are in a good state in chess than it is to figure out how to get to a winning state in chess and same thing with Go. And the nice thing about games and the reason why planning and reasoning all these planning techniques have been so effective in games is because you had this ground truth external verifier that could allow you to know, yes, I've reached a winning state.
Or it's relatively easy to say, OK, I've reached a state in this chess game where I am clearly winning. But the actual process of figuring out how to get to that point was really difficult. So I think that's why you saw all these reasoning techniques, all these techniques that leverage a lot of inference compute showing up first or most noticeably in these games. But now, we've reached a point with the progress in things like large language models where the models themselves are actually pretty good at verifying.
And so they can do a lot of that verification implicitly on its own. It can recognize when it's working through a math problem that the path it's going down isn't very promising. And so it can try something different. And if it's like working through a decryption puzzle, it can realize that it's making progress on the problem and when it's not making progress on the problem. So I think that maybe doesn't explain exactly all of why o1 is so effective or what exactly is going on.
Inference scaling bypasses traditional pretraining walls
6 Highlight · Test-time compute as a substitute for pretraining
But I think it gives some intuition about why this is possible and why we're seeing it now. So what are the implications of this? There is this great paper that came out in 2021 where Andy Jones investigated scaling laws in the board game Hex. Hex, it's kind of like Go, but simpler. And he showed this great plot where, on the x-axis, you have the amount of compute that goes into a model during training time.
And on the y-axis, you have the amount of compute that goes into the model at test time when it's actually playing a game. And you have these curves where you can train different models and they will all have 1,500 elo. So you can either throw a lot of compute into the model at training time, but very little at test time, and get 1,500 elo, or you can throw a lot of compute at test time, but very little during training time, and also get 1,500 elo. And obviously, if you throw more compute into both, you just get a higher elo rating.
And so what he found is that you could increase test time compute by 15x. And that would be the equivalent of increasing training time compute by 10x. Now, we are in a world, like I said earlier, where the amount of compute that's going into pretraining for things large language models is very, very high. But the inference costs are very low.
And there was a reasonable concern among various people that we were going to start seeing diminishing returns from AI progress because the costs and the amount of data that you need for pretraining would become so astronomical. And I think the really important takeaway from o1 is that that wall doesn't actually exist,
that we can actually push this a lot further. Because, now, we can scale up inference compute. And there's so much room to scale up inference compute. Now, I think a reasonable criticism of this is that, OK, we can scale up inference compute, but do people really want to-- that's going to increase latency of responses by a lot. And it's going to increase the cost of queries by a lot.
And that's true. Inference costs are going to go up. But I think for very difficult problems, that is a cost that many people are willing to pay. If you think about things like a proof of the Riemann hypothesis, how much would you be willing to pay for that? $0.15? Yes. $1,000? I would think so. $1 million? I think many people here would be willing to pay that.
What about life saving drugs, new life saving drugs? What about breakthrough technology? There is a lot of room to scale up inference compute to get better performance on really important, really difficult problems that would justify that cost.
The bitter lesson favors compute-scaling over heuristics
And there is a lot of room to scale it up. So yeah, I think one of the main takeaways is that AI can be more than things like Chat GTP. It can be more than chat bots. So to close out, one of the things I want to point out, there's this great essay, "The Bitter Lesson" by Richard Sutton. Many of you, I'm guessing, have read it. If you haven't, I highly encourage you. It's very short, but it's very valuable for academics to understand.
In this essay, Richard Sutton, one of the creators of the reinforcement learning field, talks about the takeaways of 70 years of AI research. And he talks about things like AI for chess and language models. And one of the things he talks about is you look at the history of chess back in the '50s and the '60s. A lot of people try to come up with these clever heuristics that encoded human knowledge into the AI. And you would see short-term lifts from doing that.
But all of those techniques are now obsolete. What really ended up being the most effective was general techniques that leveraged the large amounts of compute. And they were effective because compute could increase and it has increased substantially over the past 70 years. So even if you have a technique that isn't as quite as effective as some clever heuristic, if it scales better with compute, then that is a more effective algorithm in the long run.
And the unfortunate reality is that, in the AI academic research community, there aren't a ton of GPUs. And there is a strong incentive to publish papers, to go to a conference and get recognition for work. And that inevitably leads to an incentive to try to push the bar a little bit higher than the state of the art in order to get a paper and go to a conference and be able to talk about your work.
But those techniques might not scale well with increased compute and, in 5 or 10 years, might not be what people are actually relying on. And so I would encourage everybody to try to think about techniques that do leverage more compute and try to think about what will be really useful 5 years from now rather than just trying to edge out the current state of the art.
OK, what's next for me?
Q&A explores academic barriers and search generality
So my background historically has been in multiagent. I've worked on AI for poker for a long time. I worked on Diplomacy for a long time. I actually became a little disillusioned with the multiagent field partly based on the way that we ended up tackling Diplomacy, which is a not very classic game theory multiagent approach, but a very general reinforcement learning and planning approach. And that's why I actually switched to focusing purely on planning because I felt like there was a lot of value in being able to scale inference compute in general rather than just limiting it to two games.
I now have some renewed faith in multiagent as a research direction, and I'm excited to be launching, with several others at OpenAI, a new multiagent reasoning team. By the way, if you're interested, we're looking for strong engineers that are interested in research. And prior multiagent experience is not required because we're going to pursue a different approach than has historically been pursued by the multiagent research community.
So if you're interested, please apply. I mean, OpenAI is hiring. And I think it's the most impactful place to be doing machine learning research today. And so I've been really happy that I've been able to work there and to be able to do great things. I think we're going to continue to do great things, and I think it would be wonderful to work with other people that are interested in pushing the frontier.
So I'll stop there and take questions. Thank you. Yup. AUDIENCE: I had a couple of questions. One is that you showed the different across different domains. Is this related to how well you're able to train your system, how you're able to reward it? Is that the reason why things are different across domains?
NOAM BROWN: I think it's a few factors. So one is just the quality of the evals. Another is I mentioned that these algorithms, they do well on hard reasoning tasks. There are some tasks that just don't benefit from more reasoning, for more thinking time. And so you might see different lifts in different domains partly for that reason. There could also be data distribution effects.
There's a lot that goes into it, and it's not really clear. We haven't really like disentangled what the actual causes are. AUDIENCE: And then the second question is, does context matter any more? Given that you can actually reason, maybe . NOAM BROWN: I think context matters a lot. And if anything, context matters more. AUDIENCE: OK. NOAM BROWN: Yup. AUDIENCE: So you showed the scaling law for train time compute.
And you said that the wall that you thought existed for the cost of train time compute if you don't do search is now gone. And it's going up with the log scale train time compute. You also said the wall for data is gone as well. So what's the story for the cost of data for both problems and supervision for the train time?
NOAM BROWN: I think it's still unclear. I think that basically a lot of the assumptions about why we would be hitting a wall need to be reevaluated completely given that there's this new paradigm. And so we're still trying to figure that out. I suspect that a lot of other people are going to be trying to figure that out. And the answer is, right now, we don't know. But, yeah, I think that looking at the limitations of pretraining and saying that's going to be a blocker on continued progress, I think that is no longer by itself a blocker.
There might be blockers to scaling this up further. I don't think there are for the next year or two. There might be farther down the road, but so far we don't know. Yeah, you had a question? AUDIENCE: Yeah. So you said a couple of times that you think there's a lot that we can learn from this. You just said this is a new paradigm. What do you actually think that there is for the people who don't work at OpenAI to learn from this?
The things that you're talking about, as you said, are things that we use in other fields before. Those things that have been used in language models before, mostly at smaller scale because people have less compute. The things that have been publicly shared have mostly looked like taking things that people are already doing and applying a lot more computing resources to that. I'm not at all surprised that makes benchmark scores go up. So what is there to learn?
NOAM BROWN: That's a good question, yeah. I talked about my experience working on AI for poker. And sorry, where was it? And one of the things I said is that, look, people were working on search in poker before we had Libratus that beat top humans in poker. But one of the things that I took away from this that surprised me when I was working on poker was how much of a difference it made.
It wasn't 10x. It was 100,000x. And if I had known that and if other researchers had known that earlier, I think they would have invested a lot more resources in this direction. And so I think there's a similar story here where people have been working on scaling up inference compute. But it wasn't like the number one priority of the academic community by any means. And I think being able to point to o1 and show people how much potential there is in this direction to really be able to play with it and see how much of a difference it makes I think shows people how high the ceiling really is.
AUDIENCE: But the most powerful model trained by the academic community have leveraged inference time. NOAM BROWN: Sorry, can you say that again? I missed the-- AUDIENCE: Some of the best models trained by the academic community have leveraged inference time compute and trading off higher inference time compute for better performance. The academic community doesn't have the resources to train this model, but the academic community has never had the-- training this model was never on the table for the academic community.
NOAM BROWN: I think there's-- AUDIENCE: So it may be factually incorrect to say this is not something the academic community thought was a high priority. It's something that-- NOAM BROWN: I'm not saying that they thought it was a high priority. But I think that certainly it's like-- I mean, I can say that I think it should be a higher priority in the academic committee. I think it was a high priority. But I think that there's a lot of potential here. And it should be given a lot of focus.
I think there's a more fundamental problem that you're pointing to which is that there's a huge compute disparity between industry research and academic research. And unfortunately, I'm only able to do so much to help with that. But I think that there does need to be a rethinking about how to structure academic research in order to be able to do this kind of research in the academic community. And I think that it should be possible to do this kind of research in the academic community.
And I think that there are options. I mean, I think you look at other fields like physics. There's a lot of very large, expensive experiments that are done in physics by academics in the academic community. And I think that there needs to be maybe perhaps a reimagining of how to structure some aspects of machine learning research in order to enable those kinds of experiments in the academic community.
AUDIENCE: Can I ask my question slightly different way one last time? Or other-- NOAM BROWN: I'd rather we can take it offline if you'd like. Yeah, you have a question? AUDIENCE: Yes. So nice talk. You chose to use the search in finding a better way to use chain of thought, right? I mean, at least from what you said, you said-- I mean, people hack around with chain of thought.
But you can do it kind of more systematically. And that AI would help with that sort of makes some intuitive sense. But an alternative strategy might be, OK, planning problems often can be done by logical inference. There is search in that community. There have been SAT solvers around for decades. And there are people like Subbarao who essentially pushed that planning is basically-- it has a logical framework.
And, therefore, search in that kind of a space is more efficient and a simpler way and more direct way to solve the problem, as opposed to taking this big machinery of LLMs and coercing them to do the right thing. I'd like to get your response to that statement. NOAM BROWN: I think it's a good point that these are not
7 Highlight · Why general search outperforms classical logic solvers
expensive techniques-- sorry, these are not cheap techniques. These are very expensive techniques. And if you look at, individual benchmarks, there are techniques that can do much better on those benchmarks that are much cheaper. But the real value of something like o1 is generality. And I think people can go online, play around with it on ChatGPT and see that generality. AUDIENCE: No, but let me push back on that.
I mean, logic also has generality, right? I mean, in a sense, classical logic in the classical-- going back hundreds years or in the early days of AI, people like McCarthy and so on, their framing was that logic is a general framework. It's not actually general because of probability and uncertainty and so forth. But there are problems which fall in that framework. And I mean, of course, we were working on these problems decades ago.
And we now have much, much more compute. So ideas of search are relevant in this framework, and it will be quite general. It's not like just solving one domain, which I can agree is less interesting than-- I mean, what you have is a general purpose technique. But I'm arguing that doing system to reasoning in a more logic type framework is also a general thing. NOAM BROWN: So, OK, a couple of things.
First of all, I think the way that we're doing o1 does not have to be the correct way of doing scaling inference compute and doing system two reasoning. I would honestly expect that there is a better way of doing it than what we're doing. And I would love to see that. And so actually, like I said, we can't talk about the details about how o1 works. But also, I don't think it's the most important thing to share that because I would hope that people would explore many different approaches that would hopefully end up doing better than what we're doing.
So I agree. Yes, there's a lot of different techniques. There's a lot of different approaches. And I think it's worth it to explore all of that. And I think the real value of o1 is seeing that, if there is a good technique, just how much of a difference it can make. The other thing is that there's been a lot of techniques for scaling inference compute. I mean, I've worked on that in poker. But the techniques that I developed previously in poker were not very effective in something Go.
And I worked on similar techniques. Basically, my academic career for the past several years has been developing planning techniques that would scale inference compute on these individual domains and do amazingly well, but would unfortunately not be able to generalize outside of those domains. So Hanabi, we reached superhuman performance with a technique that worked really well for Hanabi and worked OK in some outside domains, but not amazingly well, and same thing with poker, same thing with Diplomacy.
And one thing I really value about o1 is that this, for me, is the first time that I've helped build something that does generalize outside of any specific domain. And I've had people come to me and tell me that they've used o1. It has saved them days worth of work when none of the techniques that I've developed in the past
have been able to do that. AUDIENCE: Thanks. SPEAKER: Well, we have time for just one last question. AUDIENCE: AUDIENCE: You go ahead. AUDIENCE: OK. Yeah, go ahead. AUDIENCE: Thanks. So like Stella was alluding to before, people in the academic community are saying URL and chain of thought. Basically, as soon as the chain of thought paper comes out, you're thinking of Erik Seligman's star work. It seems like the new thing here is that you have this knob that you can turn that says vary how much you want to think for a test time in a sort of controllable way.
And that's what makes this look like all of those other nice plots you were showing about increasing the number of samples you draw, increasing the number of the depth to which you build your MCPS tree or whatever. What does that knob look like once you're starting to talk about generating arbitrarily long natural language strings? NOAM BROWN: That's a good question. It's one of those questions I can't answer. Yeah. One of the things that we do want to make available to users is the ability to control-- right now, if you log onto ChatGPT and play around this thing, you can actually control the thinking time.
We do want to make that available to the users eventually to be able to actually control how long it takes for. And that is one of the things that, yeah, I think is very, very, very powerful, very useful, to be able to basically tell it how long to think for. And I think, honestly, the nice-- I don't want to call them scaling laws by any means because they're not laws. But the nice curves that we get out of scaling inference compute, that comes from the ability to control its thinking time.
And I think that we wouldn't have noticed this otherwise. Yeah. SPEAKER: OK, we're time . For the next talk, we meet at 7:35.