MIT EI seminar, Hyung Won Chung from OpenAI. "Don't teach. Incentivize."

6,552 words · about 26 min to read

0:00 um today we're fortunate to have himan from open AI he's a research scientist interestingly he actually did his PhD here at MIT but it wasn't on machine learning right think it was on renewable energy and energy systems and since then um he's gone to Google brain and then moved over to open AI hel's very known well known for his work on uh large language models so his previous some of his Works include uh scaling flan flan T5 flan Pam and t5x which was the um training framework you train the P model so I'm very excited to hear him back U hear him from elen he from him the ADV in LM and let's welcoming back with a round of applause thank you uh it's really good to be back and today the title of my talk is don't teach incentivize my research is on developing general intelligence and as opposed to a specialized intelligence and when doing that it's just impossible to enumerate

1:03 everything that we want to teach the model and just there's so many skills we want from them and we don't even know all of them so I think the only feasible way to get to general intelligence in my view is to weekly incentivize the model so that uh the models themselves figure out themselves so before we get to the details of the talk let me share the goal of today's talk talk which is somewhat different from most of the technical talks uh in that

Focus on finding great problems over technical cleverness

1:35 I'm not going to share specific technical like paper uh latest paper of mine and or experimental results instead I want to share how I think with AI being a running example and you might wonder why in my view we the technical people focus too much on solving problem itself in my view we should put more attention to uh finding great problems to solve and the best researchers I've seen in my career are uh not necessarily the most strongest technical people they certainly are but that's not the defining feature for them instead they know how to find the most impactful problem to solve and I think that comes from having a great perspective perspective and I think having this importance of having good perspective is undervalued and I want to use this chance to share mine and by no means I'm

2:37 saying I have a great perspective my hope is that by sharing mine it Sparks interest in this kind of conversation among some of you and we as a community will discuss more of this and hence become better at finding better problems

Leverage exponential drops in compute cost

2:51 to solve so let's get started brief outline of this talk is first we'll build share my perspective and everything I say will be pretty much on scaling and I'll Define what's my my version of the scaling is later but we we'll develop that perspective with very general AI research and with that we're going to specialize into this language models that I'm working on so that's overall uh view let's start with the most important data points I know in the entire AI so this is from Rich Sutton's keynote speak speech last year and on the x-axis we have a time from about 1900 to 2020 and then on the y- axis we have compute power if you the calculations per second you get if you pay $1,000 and this is in log scale so what we're seeing is that compute um power for a given amount of dollar increases exponentially over like 100 years in other words compute cost is going down exponentially and I don't

3:53 know any other Trend that is as strong and as longlasting as this one and whenever I see such an incredible Trend that is way beyond my intuition I think about two things first I should not compete with this second I want to leverage this as much as possible in every aspect of my career and life so we're seeing this Hardware exploding um exponentially so we as a people some of us developing softwares and algorithms should catch up on that and in particular we need more scalable

1 Highlight · Scaling triumphs over clever human structures

4:25 methods that can better leverage ever increasing uh abundance of of compute so more generally what is the job of the AI researcher is to teach machines how to think uh and one very common unfortunately approach is that we teach the machines how we think we think but do we know how we think at the very low level we don't so what we're doing in this kind of approach is that we're teaching what we don't fully understand and in a limited language of mathematics and this approach typically poses a structure to the problem which can become typically becomes a bottleneck

Clever human structures become scaling bottlenecks

5:03 when when we further scale up and bit lesson again from rich suton is summarizes phenomena saying progress in AI over the past 70 years essentially BS down to developing progressively more General method with less structure and add more data and compute in other words scale up that's a very strong statement since we have seen many many different types of progress but all them just summarize into this very strong statement and I totally agree with this and in fact I think this is the single most important writing in nii and I go back to this essay many many times so i' strongly recommend it if you haven't

5:42 checked it out so this is my graphical version uh illustration of the same idea on the x-axis we have a compute uh you can think about a just Compu or data uh and or datais we have a performance you can think of it as a some kind of an intelligence metric very much of a cartoon so two methods we can think of one with the more structure one with less structure and what we see over and over is that the method with the more structure typically starts off quickly uh because the structure itself serves as a shortcut and that um serves well until it doesn't and it doesn't really scale up in contrast the one with the less structure typically starts off as doesn't work because we give too much freedom to the model and it doesn't know how to utilize it until we give enough dat and compute and good algorithm to leverage that and at some point it just gets better and better and we call this more scalable uh solution so um oh one maybe concrete example is uh when we

6:44 have this like classical machine learning algorithms like svm that could be compared to deep learning it'll be more like a more structured version in particular like kernel methods things like that um are structur supposed to learn what kind of representation we should put into the data whereas deep learning we give OKAY model should learn the representation given this problem so that didn't work but it it eventually won because of this more scalable nature and within this deep learning we see a lot of this hierarchical uh one is more um scalable deep learning approaches and one is not so let's make some sobering observations clever structures posed by human researchers typically become the bottleneck when scaled up what is good in the long run almost necessarily looks bad in the short short uh short term because this is we're doing learning based method and we're try to give machines as much as Freedom as possible and so what's going to win in the long term will look bad in the short term so

7:47 we that's kind of a very different Dynamics from uh other scientific fields and computer is getting cheaper faster than we're becoming better researchers and so we should not compete with that instead give machines more degree of freedom and let them choose how they learn it can be different from what humans think of as a human intelligence but I don't think human intelligence is any form of upper bound and we care at the end about the intelligence and the value it creates so uh we should not dictate how that looks like and maybe if this observation might be obvious to you but at least many people find it not obvious and there are many reasons I want to talk about one reason that is less talked about which is that researchers want to add modeling ideas because that's academically more satisfying and some people think just scaling up is not scientific or even interesting and I often hear this phrase that's just boring engineering and I would like to ask this question to such people why are we doing AI why are we

8:48 developing any technology for that matter and I think the ultimate goal is to create value that benefits humanity and that is more important than anything any ego of the science SST and so really we should be focusing on maximizing the value generated by AI Al mizing the downside regardless of which academic discipline achieves the goal and doesn't matter whether the thing I studied for 10 years is not the most scalable now I have to be really uh intellectually honest about what I'm really after which is to create value and if it's not the most scalable method that leverages the compute the best possible way I should rethink and learn and I think this is kind of the approach that I've been taking including so my topic has been changing but in the theme of better leveraging the compute it hasn't changed so I've been talking about scaling in a kind of a loose way I want to define a little bit more precisely what I mean by that because it's a little bit different from the common

9:49 definition that is used often which is that doing the same things we're more more machines my I think this is true but not that interesting my version is that it involves first identifying the modeling assumption or inductive biosis that bottlenecks further scaling and second replacing it with a more scalable one and if we think about this previous example of svm versus deep learning I think of that as a scaling because we want to put in more compute and SPM was just bottlenecking there and we identify that this data representation was the bottom neck and replace it with the learn representation and that I think is scaling in my view and I think a more interesting type of it and many many um instances of this and in my personal Viewpoint what open AI is really doing well relative to other places is that they really researchers there really abide to this kind of idea whether uh consciously or not this is really what we're doing we have this so much compute

10:51 that's available or going to be available soon and the question is how do I put in it's actually quite difficult if you think about if I give you like 100 billion dollars of compute can you use it in the right way now and I don't think anyone knows really well

Next token prediction drives massive implicit multitask learning

11:04 how to do it so we are really finding a way to put ever increasing compute and that's the researcher's job so that's my perspective um everything I'm going to talk about from now one will be based on this scaling and bitter lesson and so let's talk about this language models I put it like large language models llms because it's just a common acronym but I kind of think the both L's here don't really mean much large is very subjective term large models of today will be small models in a few years or even next year and and the other L language I'm going to talk about why I don't think it's a good term so all the llm so far use transform architectures and we don't have to look uh look into the details we're going to take a very functional Viewpoint um and think about this Transformer as a sequence to sequence mapping with just bunch of map moles input is a sequence of uh vectors each of size d and n and n vectors output is the same thing at least during training that's what happened and the interaction is done by bunch of M moles so just because we have a

12:04 probably a diverse um audience I just wanted to bring everyone in the same page and just have a crash course in What's um what what the next token prediction process is doing and go into the some of the implications of that process so we start with the sequence model and one particular sequence we can think of is a sentence which is a sequence of words and the first step is to tokenize it and why because we need to represent the words in in a computer and which requires some kind of an encoding mechanism and here we usually use this um mapping of the integers so we Define a finite set of integers say like um in this particular case 3 30,000 I think uh and each words or token gets mapped to an integer that's going get changed during training so now we have a sequence of integers instead of the string and we can now do an embedding this is not necessary by any means but this is more common ways of doing it now each integer representation of the sequence element in this case a token is represented in the vector of size d and

13:05 now let's think about what we want from a sequence model from first principles we the sequence model is a model that um just models the interaction between the elements in the sequence and so uh the Transformer is a particular way of doing sequence modeling where that interaction is modeled as a bunch of dot dot product so we take the two Vector representation of the elements take the dot product if it's high then we think this is semantically uh relevant related than the two pair the other pair that has a low D product and we don't really dictate what that means semantically uh we can try to understand it but I think it's not really the easy or may be meaningful and so after bunch of do product we get a representation of same size Vector but now this each Vector knows about the presence of the other ones and hopefully some how measures some relevance between them and have a rich deep understanding of what's going on and so the final process of this take

14:08 the loss function we need a scaler just because we want to use a gradient B based uh optimization method and so that we get the scalar now take the gradient of this scalar with respect to all the parameters in the model and then update using some kind of gradient method that's one step repeat as many um times as possible and that's the the training process so next let's talk about this um taking from this Vector representation into a single scalar which is the optimization objective function and in this particular case most large scale training nowadays use some form of next token prediction so I think it's worth uh looking into it more uh in more detail so I put this original sentence back in the in in this slide so let's in what we're doing is essentially given many U the first token try to predict the next one as I mentioned this encoding mechanism uh we start with the finite uh vocabulary which is say like 100k items and the model outputs the probability distribution over them so the numbers

15:08 here should add up to one and our hope is that the model learns about the structure such that the next token words gets higher probability than uh something like intelligence many intelligence doesn't make sense and after that we do again given many words try to predict the next token which is done and and so on so that that's the process you do everything until the end so this is a represent thinking about conditional probability we get given all the previous tokens try to predict the next one an equivalent uh maybe more uh you know just natural to statistics background people is that we this conditional probabilities when multiplied together becomes the probability of the entire sequence um and that's what we're maximizing and this is the maximum likelihood um framework but I think it's more kind of intuitive to think about multitask learning each predicting the next token given everything uh before and crucially each prediction gets the same weight we don't really distinguish which token is what and in in industrial industry scale

16:11 we just do it in the web scale um text Data feed that into the Transformer that's where magic happens so what kind of magic uh do we expect this is a um paper from Palm paper uh when we worked at Google um the reason I picked this is because nowadays people don't really report in the pre-training only so uh if you want to understand the next token predictions p uh performance by itself we kind of have to go back to the old paper here the details don't matter some bunch of big bench tasks and what we're showing is that just by predicting the next token is actually getting better than many human average performance and I I don't know uh but to me this just doing next token prediction and getting better at so many different things is just magic I don't I still find it very surprising and um I think we should not take it for granted so let's make some observations about this next token prediction we don't directly teach any of these like linguistic Concepts like um the next token is a verb next token is whatever

17:11 we don't do any of that but somehow just by predicting next tokens the languages just come uh like emerge and and that I think is a almost like a byproduct of doing such test languages um are learned and I would go even further to say in the best way to teach languages not to teach language and because we don't want to put any structure in the machines learning about the language and so that's uh very interesting byproduct and that's why I don't think language model the L here is a kind of a misn normal because we're not teaching any language language model is about learning some word knowledge and language is one representation of that and so the other interesting thing is that the model can do some reasoning math and coding and all these things even though we don't

2 Highlight · Next token prediction as massive implicit multitask learning

17:58 any do any specific teaching of that so I think uh all the researchers I know have essentially some version of the hypothesis as to why this things work so well and I don't think any there's any consensus in what why why this thing is so good and I do have my own version which has helped me quite a bit in thinking about my research uh uh projects so I want to share this so my version is that it's a massive implicit multitask learning implicit in the sense that we don't directly specify what are the multitasks that we're uh asking the model to do they're just um imposed by the diverse Corpus that we're using and so um actually let let's think about this um we I think now take it for granted this language models are doing good job but um it's probably useful to think about the suffering the model goes through and during training so let me try to illustrate what is it like to do the prediction of the next token and here I'm going to show a sentence and

18:59 the underlined one is kind of what what I'm trying to predict and in doing so let's do with the least effort possible try to do it with the any shortcut as possible and so if I want to predict this the terrible movie was really and then try to predict boring I can probably take the shortcut and say okay terrible as a negative thing so it's going to be something negative and then it's movie so maybe I can use that pattern matching to get the word boring now I give you one more um sentence after the earning call the share price of Google went up by 5% from, ending in 1050 here because I want to do lowest possible effort I think about can I recycle the skill that I learned from the first uh example to do this and it's not because it's irrelevant I have to U learn some kind of a new skill which is a math third example it's um completely different language now so I cannot recycle this uh because I need to learn new language so third skill I learned fourth the first law of theer dyamics which is my favorite topic used to be is of called the conservation of energy now

20:00 in order to predict this we need to know we cannot recycle anything U above have to learn a new skill which is scientific knowledge and so on and so forth when we do it at the whip scale we're talking about billions or maybe even trillions of sentences and that because each sentence contains multiple next token um on the order of maybe thousands we are talking about trillions of task types that can be um framed like this and if you're trying to solve this and at some point just intractable even for large models this doesn't seem like a tractable thing to do and I think if we really want to do it with the lowest possible effort maybe it's more immutable to think about okay what are the general skills that I can learn and apply to as many test types as possible that probably is easier path now if we burden the model enough with trillions of test types and I think that's my my really handwavy way of thinking about multitask learning hypothesis Beyond some scale it's easier to do that uh

21:03 what what I mean by General skills first these are Text data so understand I mean languages is probably like something I want to learn even though we don't teach anything about language so there are maybe like hundreds 100 languages that we put fit into the model and so learning all those probably very useful to at least understand the syntax uh structure of it second after that I want to understand the language so that I can kind of that's applicable to pretty much most of the sentences next token prediction and and third maybe more like uh reasoning to uh compose the concepts I'm just making things up but I think these are like if I'm doing this task I'll probably think about those and at some point that's the the the the pressure given to the model so um that so I think that that's crucially we don't directly teach any of these General skills and they are just uh incentivized by this uh learning objective and the data and these abilities emerge and abilities that emerge are typically more General

22:01 probably because it's um driven from the need to do develop General skills and so you might wonder okay this weekly incentivizing the model requires a lot more comput and hence it's a inefficient process and I would like to think about it this way it's enables adding more compute and we have will have more

Weak incentives induce general capabilities

3 Highlight · Designing incentive structures instead of teaching tasks

22:20 compute this is more scalable training strategy and that I think is why this has uh done so well and maybe let's generalize this uh hypothesis a little bit for a given data set and learning objective there is an explicit learning signal and a set of induced incentives so for this next token prediction the explicit signal was to predict the next token because that's the task given the induced incentive is to learn General skills so that it can do many tasks as possible like understand language reasoning and so on what are some other examples playing chess with 01 reward at the end of the game if you think about something like Alpha go zero the explicit signal is to win the game and we don't teach anything other than that and so the induced incentive in this case is learn what good moves are and the model might not even think about what is good and the notion of being good but it's just that's the incentive it has to know kind of those in order to win the game and if we uh for chess the environment is so n uh narrow that probably doesn't matter but if we use this kind of a process

23:23 what it learns winning the game maybe matters but the the general notion of what is good in this environment is something that is very general and I think we should increasingly care more about that third example it's probably more relevant for nowadays hallucination I think is one of the the core problems that are unsolved for the language models and so here's one um example I got from John uh Sean's talk here and uh adapted for for my purpose here reward structure for let's say we are doing simple question answering and the model hallucinates on this answering simple question we can Define find something like this one reward is one if the answer is correct and unhedged and point five is correct but the model says I'm not really sure but maybe this and got correct zero if the model says I don't know which is an important uh class of answers and third and negative -2 if the answer is hedged but wrong and negative4 if the model is very confidently wrong in this structure what is uh the explicit signal is that

24:24 the answer uh we answer the question correctly that's the explicit signal but what is induced if we do this with trillions of questions let's say is that the model needs to know what it doesn't know uh let me explain what that means so here if I give you like 100 questions and you know you don't know the answer to 50 you can kind of memorize okay this question I don't know the answer and so on but if I give you like trillions of them then it's easier to think about instead of tracking each of those trillions which one I know and which one I don't know maybe think about the general notion of what it means to know something what it means to not know something and that ability is kind of what might be what might emerge out of this kind of in induced incentive structure and that I think is the only fundamental way to solve hallucination um and maybe just to give you a like lose analogy of this process we have this old thing that says give a man a fish and you feed him for a day teach him man man to fish and you feed him for a lifetime so you can think

25:25 about the first one is like the hard coding second one there directly teaching a skill what is the incentive based kind of approach for this problem it's teaching the taste of fish and make him hungry and then this person if it's enabled will go out learn many many skills including fishing and along the way the inducive induced incentive is that he'll learn some other things like patience being patient learning the weather knowing what um bait this fish likes and so on and this these are might be more General skills like being patient is quite General so that's kind of the my lose analogy extending this

Machine scale eliminates human trade-offs between generalists and specialists

26:04 old saying and so looking at this you might wonder okay this is very inefficient why would you even do do that and there exists a trade-off of achieving the goal versus taking uh some so much time and more generally what we're thinking about is a trade-off exists in this kind of situation when when there's a trade-off there's a scare resource that we're thinking about and that is in this case a time required to get this job done so if we plot it on the x-axis for the time required to get it in each method it takes a lot more time to uh incentivize him to learn the teaching skill sorry the fishing skill itself but that limitation is for humans for machines it the the scare of resource we're talking about when thinking about the trade-offs is not time it's time but increasingly it's more about the compute required and we can get away with this and this has a quite Rich uh implications and one of that is um I hear a lot about okay we have a small specialist model that's

27:04 probably better than some generalist large model and when I hear this the hidden assumption here is that small specialist models can win in a narrow domain because there is some kind of a trade-off between being a specialist and generalist I think this is the bias we get from um just being a human why because again tradeoff we want to think about the scarce resource which is time and most humans operate with pretty much the same time budget um and like you may sleep like one hour less or more but roughly like 16 17 hours a day that's kind of what it's given to humans and because of that time spent not being a journalist is time spent on or time spent being a journalist it's not spent on being a specialist and vice versa so that's the trade-off um and but for machines it's not that at all one model gets to enjoy a lot more compute than others so if you like this dragon there's a room of spirit and time where you spend a train one year inside it's a day outside so in this case a person

28:07 access who has access to that kind of thing has 365 times more time than others but when we talk about machines the multiplier is not just 365 it's a lot higher and I think that's the situation so I don't think this trade-off exists at all if anything generalists are actually better at special things and this incen structure uh you might be wondering this is very old idea if anything entire human civilization probably is driven by um incentive and even Evolution might be thought of as a incentive structure for survival so why am I talking about this old idea why is it now relevant that's the thing I need to mention and let me start with this no amount of bananas can incentivize monkeys to do mathematical reasoning so even if the reward is infinite and incentive structure is perfect this just not going to work it's not possible and what that means is some kind of a threshold intelligence is necessary for the given uh problem to work with the incentive structure and currently with

29:09 something like gbd4 we cross that threshold for many many tasks so I think it certainly uh now suddenly makes sense to think about incentive based structures to teach the models what we care about and that brings it to this notion of uh this incentive structure uh depends on the model size or model scale what abilities emerge depend on the model size that's one interesting parameter and if the model is too small model just might just give up on developing this General skills and just

Emergence demands a 'doesn't work yet' perspective

29:40 you know let's leave it the simple heuristics and just fine with the high loss during the training process so that's that sums up the incentive structure part of it and I want to I've been talking about this immersion abilities a little bit in a loose sense I want to talk about that so um in particular we we are talking about some abilities emerging when we scale up and this requires really having the right perspective and in the the field we have some discussions around okay the emerging abilities a mirage or emerging abilities don't happen and things like that I don't think the details matter let me give you one one way to think about it I make the tiniest Transformer the D model is one one two layers tiniest Transformer model and and take some task that works with gp4 and say math and then solve it with this it's not going to work and it works here so there's some kind of inflection point there and for any problem you can find the smallest possible model that doesn't work and if if it works with the current model that's to me that's emergence so I

30:40 think that's very easy to prove that it actually happens and if it doesn't then probably it's not uh interesting enough for me and this is what I mean x-axis some some scale and then it doesn't work

4 Highlight · Adopt the 'doesn't work yet' mindset and unlearn outdated axioms

30:53 and then at some point it SK uh just emerges doesn't work for everything but but in for many interesting things like reasoning this is the pattern we see over and over and and the first perspective that uh really we need to think about this this perspective of yet because now things that don't work now doesn't mean that it will not work later so what we should do is instead of saying this idea doesn't work we should say this idea doesn't work yet and that really is um unnatural for humans to think about and I think there because the bias is so huge in how we operate in this environment we are operating an

Rapid capability shifts require constant unlearning

31:29 environment with underlying axioms not changing often which is kind of the physics so if you run like I don't know thermodynamics experiments now and you get the result you run that three years now from and it will probably be same if for that matter 300 years probably the same and and we're used to this kind of environment so uh because the underlying axioms are the physics we are operating and for language models what is that equivalent concept of the axioms that we're operating under and that is more like the most capable model so when uh in 2020 uh gpt3 came out that serves as an axom for researchers to build intuitions on how to leverage this kind of uh technology gp4 came out last year it changed a lot of the things and many things that built on top of that old model are not relevant it's almost like every couple years we move to different planets and the physical understanding changes and this calls for the need for kind of a constant unlearning process and the the the intuitions and and ideas

32:31 built on top of those incorrect axioms now have to be unlearned but I don't think this is done that often so which acts as an interesting Dynamics for newcomers and in the field I see a lot of times that undergrads with not much research experience come in one year they write papers that change the trajectory of the entire field um and that is not possible if you think about in theoretical math physics I don't think that that's possible um unless you go back to like gaus or something um so so this is a really interesting Dynamics if you come in with not much um knowledge you might think of yourself as having not much baggage to unlearn and you're more flexible at least that's how I uh thought about it and many people at

33:19 openi are not traditional ML phds and they're more like just when it when they see the evidence change and we always adopt to new paradigm and so that's I think this really important Dynamics I um I want to point out so uh how I think about it again one last cartoon and then we can wrap it up is um this emersion ability if we think about some scale on the x-axis and then abilities for the y-axis and I put two models of different size three and four it doesn't matter let's look at the ability one first and uh ability one we have gpt3 and four are before this inflection point so it's not possible but just a little bit better model than gb4 might be able to get um some sign of Life ability two it will take a lot longer ability 3 even gpt3 is able to do it for example this might be like sentiment analysis things like that so what I mean by this when whenever we see this ability because we don't have any theory for imersion abilities there's no way to know which ability I'm

34:19 working on but at least having this kind of visual pictor can um just have I can I can use this to prepare myself so I don't want to work on as a practical person like ability two I want to work on uh ability one and just be prepared for the next model and think about why this is not working can where can I put more compute to to get to the the the sign of life and maybe less so there's nothing wrong with this incremental thing but I'm just if I'm working on things that small models can already do I need to know that it's there's uh you know finite room of improvement let me uh close this talk with some just summarizing what we have talked about the most important uh underlying driving force is the compute cost is decreasing exponentially and AI researcher's job recognizing this is to not compete with that and just leverage it by designing more scalable methods current generation of language models rely on next generary next token prediction which can be thought of as a weak incentive structure which needs uh to learn General skill so that it can

35:21 handle trillions of task types um and more generally we should start thinking about next token prediction has been great but it's not the only thing and we should really think about incentive structures for the models and um that's the new kind of learning Paradigm we should really engage in and emerging abilities happen which is one one of the unique things about the language model so we need to have some right perspective and such as unlearning things that are based on um outdated axioms that's it thanks