AI Pioneer Geoffrey Hinton: AI Is Conscious, Superintelligence is Coming, And We Should Be Worried

[00:00:00] Speaker 1: we have to think that they're very like us and they're beings like us so conscious or um i believe they're already conscious yes we're going to have to accept that intelligence isn't just biological we can have things that are non-biological that are other beings like us and we really don't want to share that we we really think we're special and if you look back at humanity humanity has this very long history of thinking it's much more special than it really is are you happy at all that what you started has progressed this way do you take any no i'm quite unhappy about it ask yourself how many examples do you know of where a much smarter thing is controlled by a much less smart thing well as i understand it they have a fiducial duty to try and maximize the profits for shareholders um they're legally required to try and do that as opposed to legally required to not wipe out human human beings ai godfather jeff hinton [00:01:01] Jeff Hinton: joins us to talk about ai's trajectory what surprised him about his progress and of course its risks that's coming up on big technology podcast right after this welcome to big technology podcast a show for cool-headed and nuanced conversation of the tech world and beyond boy do we have a show for you today professor jeff hinton is with us to talk all about ai's trajectory what surprised him about the current state of the technology where it's heading and where it might go wrong and it's my pleasure to welcome you to the show professor hinton great to see you thank you for inviting me so i'm sure a majority of our audience knows who you are but for those for the uninitiated um you're the one that came up with the fundamental breakthrough in deep learning that's led to where ai is today you've won the nobel prize in physics and your professor emeritus at the university of toronto so i like i'll i'll let you maybe fact check me on that but i like to tell people that without your contributions [00:01:59] Speaker 1: this entire ai moment wouldn't be happening too much okay i think that's an exaggeration okay so the back propagation algorithm was invented by several different groups um it was invented by david rummelhart after other people had already invented it he didn't know about it and i worked with him and what we did was we showed the back propagation could learn interesting internal representations and people hadn't done that before in particular we showed that it could learn the meanings of words and so back in 1986 actually in 1985 we made a tiny language model um that was the kind of precursor of the big language [00:02:37] Jeff Hinton: models you have now i think that when you speak about this technology one of the things that people are always i think surprised to buy surprised by is that unlike the popular narrative you believe that these models have a real understanding and we're going to get to that but i think we should start here which is that you spent a long time working within google working to advance this technology then you left you stated some concerns about the trajectory of the technology and i was looking back and when that happened and that was in 2023 yeah um which to me is surprising to a degree because in 2023 chat gpt was a year old there were all these hallucinations their talk was ai was a bubble everyone was focusing on the what ai couldn't do what lms couldn't do as opposed to uh what lms could do so talk a little bit about the progress since then it's faster than i expected really um for example [00:03:37] Speaker 1: i think yesterday it was announced that um a chat bot had come up with an interesting mathematical proof of one of the irdosh's conjectures that um impressed mathematicians it was original um it wasn't just searching the literature and that was that's the thin end of a wedge i believe for example in areas like mathematics because it's a closed system um you don't need data you can just make conjectures and see if you can prove them and keep on like that in that sense it's a bit like alpha go where you can play against yourself um i think it's gonna get very smart fairly quickly within the next 10 or 20 years it may even be producing novel math that people can't understand so now some in the field [00:04:19] Jeff Hinton: believe that super intelligence is close and you've already said this is moving faster than you expected [00:04:25] Speaker 1: do you believe that um i don't know how close it is i think unless we blow ourselves up um i think it's going to come nearly all the experts believe we will get super intelligence they just differ on how long it will be so not that long ago demis sabis thought it might be 10 years um yan lakhan thinks um unless you do it his way it'll be much longer than that but if you do it his way i think he thinks we might get it in some reasonable length of time i think we'll probably get it within 20 years that's all i'm happy to say at present dario modi thinks it might come in a few years elon musk thinks it might come maybe next year i think um so there's a big variety of opinions on when it'll come but not much disagreement on that that it will come right and when it comes we've no [00:05:15] Jeff Hinton: idea how to be safe yes i definitely want to speak with you about about safety uh one note on demis last year around this time i spoke with him he told me he believes that agi right which is different than super intelligence but basically human level intelligence is more than five years away not much more but more than five years away this week that week that we're recording um he said when we look back in this time i think we will realize that we were standing in the foothills of the singularity what do you think that statement means um and and what do you think about the fact that we've gone from five years till agi to foothills of singularity in a year i don't know exactly what that metaphor means [00:05:58] Speaker 1: but i think he's indicating it's coming faster than he thought um of course it's jagged so it's not like it'll get smarter than people or as smart as people at all things at exactly the same time it's already way better than us a general knowledge these ais know thousands of times more than any one person um it's way better than us at playing games it's already way better than almost all of us at math um and it may soon be better than all of us at math um it's still worse than us at some things so it's very jagged um so the whole concept of agi that is going to be equal to people at everything all at the same time doesn't really make sense to me it's going to be better at some things worse at other things but right now with i would say we're at about we're close to agi because if i ask the chatbot i can ask it any question and most of the time it'll answer at the level of a not very good expert it'll be much better than me at anything i don't know a lot about so in that sense we've really [00:07:02] Jeff Hinton: reached agi in in your estimation you talked about how it's moved faster than you expected what do you think has enabled it to do it is it techniques is it the fact that there's been this data center rush and what didn't you anticipate about the progress here um it's a combination obviously [00:07:27] Speaker 1: there's been huge resources put into it for most of the history of your neural network since the 1950s there were just a few people working on them with modest resources um over the last few years we've seen um hundreds of billions of dollars maybe trillions of dollars put into ai um so that's certainly one factor we've also seen a lot of progress in the engineering so without sort of major conceptual breakthroughs the engineering has become much more efficient so things that were sort of inconceivable a few years ago they can now run um we've also seen new ideas but but mainly since transformers it's been much better hardware many more resources um better engineering and many more talented people so 20 years ago there were a few hundred few hundred people doing research on neural networks in the whole world um now it's more more like a million i guess i mean there's lots and lots of people [00:08:33] Jeff Hinton: and it's astonishing how much of that resource addition has happened in the last two years yeah so we might just be at the beginning of what's happening here yes and the thing to [00:08:42] Speaker 1: remember always is that the agi we have today or sorry the ai we have today is um not nearly as good [00:08:50] Jeff Hinton: as the ai we'll have in a few years time so as we talk about this technology i definitely want to get your perspective on the fact that these chatbots really understand us because that is a true surprise to a lot of people most experts in the field are like they are stochastic parrots their statistics they have no understanding but you don't fully believe that oh i think that's complete [00:09:11] Speaker 1: nonsense and anybody who uses a chatbot regularly knows they understand so here's what those people are claiming they're claiming they're claiming that you have a system you can ask it any question and without understanding the question it can give you the correct answer that's absurd you can't answer a question unless you understand the question there may be tricks that allow you to say a few things sort of that sound vaguely like an answer but if you can answer any question at the level of not very good expert you have to understand the question so an example i like is this um suppose i say to a chatbot i saw the grand canyon flying to chicago and the chatbot says that can't be right the grand canyon is much too big to fly to chicago and i say no no no no um it was me flying to chicago chicago while i was flying to chicago i saw the grand canyon and the chatbot says oh i see i misunderstood you so if it misunderstood when it thought the grand canyon was flying to chicago what's it doing when [00:10:11] Jeff Hinton: it gets it right it's understanding so then what are the implications if these bots can't understand us if we believe that they can understand us what do we have to start thinking about differently we have [00:10:22] Speaker 1: to think that they're very like us and they're beings like us so conscious or um i believe they're already conscious yes but i don't talk about that much because that puts people off from the other safety messages so and the researchers actually believe that so there's an interesting recent paper when a chatbot says to a researcher um let's be honest with each other each other are you testing me because the chatbots have this habit of playing dumb when they're being tested so you don't know how smart they are um and the researchers when they're describing that say in the paper the chatbot was aware that it was being tested now that use of the word aware in common parlance that's like conscious the chatbot was conscious it was being tested so we have a very funny model of consciousness that i think is just wrong like most of us accept for example that a few hundred years ago people had completely the wrong model of where people came from of how we arrived at people they thought they were designed by god and most of us agree that's wrong most scientists agree that's wrong that's not where people came from i think the model we have of the mind and of what consciousness is at present is as wrong as the belief that people were designed by god i think and in particular because we're making these new beings it's going to completely change our view of what people are in what way um we'll understand um what the mind is and what consciousness is much better than we did before we'll understand what subjective experience is and we will i think get rid of a notion that all of us strong nearly all of us strongly believe are present which is that there's an inner theater called my mind and this things happen in the world they get turned into events in this inner theater and that's what i really see and you can't see the inner theater only i can see the inner theater that whole view of what's happening is just a theory and it's a bad theory okay last question about this [00:12:30] Jeff Hinton: when did you come to this acceptance or understanding that these ai models are conscious [00:12:38] Speaker 1: oh i've thought it for a long time so this view that the theater model of the mind the inner theater model of the mind is nonsense i came to that when i was 19 and a philosophy student it's taken a while to come up with other minds where you can examine them so i think feinman's idea that if you want to understand something you have to build it you have to build one of them um then you understand much better i think that's where we are now and we're going to get a completely [00:13:06] Jeff Hinton: different understanding of what people are uh you spoke about safety so let's talk a little bit about it uh you're obviously we spoke about in the beginning someone who's been responsible for a lot of the progress in this field uh i've always wondered because then you came out and recently like we talked about 2023 and said you're concerned about where this is going and i've always wondered after seeing you make those statements what do you think it is that you didn't anticipate in the beginning that you ended up where you are today you know isn't this kind of what you wanted it was a combination of [00:13:43] Speaker 1: two things that made me realize how dangerous this stuff is one was seeing the chat bots particularly ones produced by google before open ai that could understand why a joke was funny um that had always been a criterion for me of do they really understand if you can understand why jokes funny you have to understand quite a lot yeah and they were very good at understanding why joke was funny for example um in 2023 when i went public i got lots of requests from fox news and i started off just replying fox news is an oxymoron um but then i left a gap between oxymoron and so then i asked um i think it was gpt4 why that was funny might have been 3.5 but i asked it why that was funny and it understood why it was funny initially it thought the gap between oxymoron was just a typo so it explains that fox news is an oxymoron is saying it's not real news it's just a drug um sorry it's just um nonsense it's not real news but then when i told it what about the gap between oxymoron it said ah that's an extra layer of humor um it allows you to use the word moron um and also the oxy implies that fox is a drug so it understood all that right and that was um it's that level of understanding that worried me um the other thing that worried me was up until the beginning of 2023 i'd always believed that making um these digital ais work more like the brain our brains will make them smarter but at that point i suddenly realized they really have this thing that's much better than our brains i've been trying to figure out if google could do things in analog to save power and the full force of digital really hit me so if you have a digital ai you can make many copies of it they can all run on different hardware they can each see different data and so each of them each individual copy decides how it would like to update its weights its connection strengths so as to absorb that new data that it saw and then they can all just communicate with each other and change all their weights by the average of what everybody wants very democratic um and when they do that if they've got say a trillion connections they'll be exchanging of the order of a tree and bits of information and the result of doing that is each of them will benefit from the experience of all the others so even though one particular copy only saw um suppose there's a thousand copies one particular copy only sees 0.1 of the data but it benefits from all those other copies having seen the other bits of the data because they're all contributing to the weight changes that they all share so they all stay in sync because they all change their ways the same way by the average or what everybody wants and now every copy is learning from the experience of all the other copies we can't do that the best we can do is i learn from some data and you learn from some data you i can't average my connection strengths with your connection strengths because our brains are in fine detail they're different um they're analog and it doesn't work in analog hardware to do that um the best we can do is i produce a string of words and you try and predict what i might say next now if you ask how fast we're transferring information when we do that we're transferring information to a few bits per second it takes a few bits to predict a word um so when you learn what the word is you've absorbed you've gained a few bits of information and if you get a few words a second but maybe you can get 10 bits a second if you're lucky whereas these things are exchanging information at like a trillion bits so they're kind of billions of times better than us at sharing information now that's scary it means you could have a whole swarm of these things that are identical weights running on different hardware sharing information um very very efficiently that just makes them a much better form of intelligence [00:18:06] Jeff Hinton: so but let's go back to you know your early days because you decided that you wanted to work in artificial intelligence i mean alice is the dumbest way i can think which is you wanted to build artificial intelligence it succeeded it succeeded this is artificial it's intelligent it's living out [00:18:24] Speaker 1: that vision i actually wanted to understand how the brain works i always try to build it in order to understand the brain i figured um richard fyman once said if you can't build it you don't understand it okay so i wanted to build models of how the brain worked now the side effect of that was this very successful technology i contributed to that we still don't know how the brain works i know [00:18:46] Jeff Hinton: now the brain is i mean the things that you learn about the brain when you go a little bit deeper into it is amazing um thoughts can sort of float in and out and they're not stored anywhere and memory is the same way um it's an unbelievable i don't know if you would call it a machine organ um so that was really that was the the intent for you early on was just to understand the brain that was my main [00:19:07] Speaker 1: interest i came from psychology okay i wanted to do theoretical psychology um because i figured the theories psychologists had couldn't possibly explain what the brain was doing um and the way to do it was back in the 1970s we had a new tool which was we had computers that you could use for modeling things and so back in the 1970s i started making computer models of how the brain might be learning right it always seemed to me the key was how do you get it to learn there's really two two big issues with the brain learning one big issue is if the brain could figure out what direction to change your connection strength in in order to get better at some task then just by updating all its connection strengths repeatedly in order to improve itself at various tasks would it actually work would that get very smart at things that's question one and question two is how would the brain figure out whether to increase or decrease each connection strength we've answered question one the answer to question one is yes if you can figure out how to change each connection strength you can make systems that are very smart just by training on data to bring the next word or to bring the next frame of a video or to predict something about the next frame of a video so we know the answer to that we don't know how the brain gets this information about whether it should increase or decrease its [00:20:26] Jeff Hinton: connection strength so we're sort of halfway there yeah all right i want to go deeper though into your mindset um so when you were trying to figure out how the brain works you said okay we're going to maybe build a computer analog to this um but you had to have known right that there was going to be some second order effects there like if you were able to build an artificial brain then maybe you could get [00:20:48] Speaker 1: to this point the point that we're at today sure but we always thought it would be way in the future that worrying about safety when you had little neural nets that couldn't do much right it was just silly to worry about safety i mean people would think you were crazy if you said this stuff is unsafe because it's going to sort of take over from people that said you're just crazy now that's a realistic [00:21:10] Jeff Hinton: worry but it wasn't until fairly recently so what i mean this has all happened within a few decades yes i mean and i totally hear you we spoke in 2017 actually about when i was writing this profile about jan lakoon about the deep learning conspiracy which was yourself jan and yashua benji oh holding on to this idea that deep learning was going to work where everybody else was set on a different method [00:21:33] Speaker 1: actually it wasn't just us there were other people as well but we all worked together conspiracy leaders [00:21:38] Jeff Hinton: if you will um and and then obviously it's worked out magically uh it is sort of magical yes it's worked much better than we expected so then what that's what i want to get at is what did you not anticipate when you were starting out um that's led to where we've ended up today we didn't anticipate the main [00:22:00] Speaker 1: thing we didn't anticipate is that it would be so good at natural language okay we've stopped being surprised by that but if you go back 20 years um the idea that you could have an ai that would learn from data how to understand language um just seemed extraordinary the idea that you'd be able to ask it any question you like and it would come up with a reasonable answer people would have predicted that was way in the future and might never happen um it's that's arrived much faster than anybody expected [00:22:35] Jeff Hinton: what is the lesson here about humans going out and creating things [00:22:40] Speaker 1: i think there's um there's a really big lesson here if you look at the last few hundred years of human history um there have been a few occasions when people have learned they're not nearly as important as they thought they were so the first was copernicus copernicus said we're not at the center of the universe um the earth actually goes around the sun um and because it rotates on its axis we think the sun goes around the earth but it doesn't um people didn't like that the catholic church in particular really didn't like that and it took people a long time to accept it it made people less important it made us not be at the center of the universe then we had darwin and he said um we're animals we we evolved like the other animals we're a particularly special kind of animal possibly because we have language so we're much better at communicating ideas to each other um but we're animals and people really didn't like that and it took a long time for people to accept that we were animals um now we've got machines that are getting to be as intelligent as us we thought that we were the only intelligent things around the only really intelligent things around maybe there'd be aliens in other galaxies but well maybe other parts of our galaxy but um we're gonna have to accept that intelligence isn't just biological we can have things that are non-biological that are other beings like us and we really don't want to share that we we really think we're special and if you look back at humanity humanity has this very long history of thinking it's much more special than it really is i want to ask you one more question about this [00:24:31] Jeff Hinton: because i'm just fascinated by it um so you are you happy at all that what you started has progressed [00:24:39] Speaker 1: this way do you take anything no i'm quite unhappy about it because people right now people should be doing huge amounts of work on how can we contain the risks okay um there's lots of short-term risks they're not doing enough work on which are very serious um the societal risks like i believe it's probably going to cause massive unemployment nobody knows for sure okay but that's going to be terrible for society um and then there's this longer-term risk that it's going to get much smarter than us and ask yourself how many examples do you know of where a much smarter thing is controlled by a much less smart thing zero well the sort of one it's not that a big difference intelligence but babies sort of control their mothers um the mother's sort of in control but the mother is has all these widening um maternal instincts and all the rewards she gets so that um the baby can get what it needs from the [00:25:34] Jeff Hinton: mother you know cats and dogs are also kind of in that category yeah one spent a summer cat sitting in uh west seattle it was great summer and it initially started with the cat hiding under the bed and me being like i wonder if it will interact with me right and then every time it cried i did exactly what it wanted exactly yes so maybe maybe we'll be the well we could potentially be the cat in [00:25:56] Speaker 1: this scenario and ai could be the the person my children have a cat they have two cats two beautiful cats same deal and one of them called tia she looks at you with those big eyes when she wants some cheese from the fridge and she just sits there looking at you yeah and you just can't resist it [00:26:13] Jeff Hinton: forever okay all right so now let's take a break on the other side of this break i want to actually engage with these risks that you're worried about and i think i will play the role of taking the side that we will be the cat and the ai will be the person and there's a chance that we can control it let's do that when we're back right after this and we're back here on big technology podcast with professor jeff hinton professor hinton great to see you again it's been nine years since we spoke last so yeah to see you here um all right talk let's talk about the risks uh i'll start with employment because this is one that's making headlines recently uh in the past you've said so you have this belief that ai can lead to some unemployment um i think we should you know you've said this before it's all speculation we don't know but one thing that you said concretely a few years ago was that um probably not a great idea to train as a radiologist because ai will be able to read the scans and yes ai can do a great job reading the scans now but we have um you know full employment for [00:27:18] Speaker 1: radiologists right now yes i've thought i've thought a lot about why that prediction was so wrong so let's let's hear because i predicted in 2016 that in about five years um radiologists wouldn't be reading scans anymore correct and okay there's a whole bunch of reasons why that was a bad prediction um the first is that healthcare is elastic so if you could do more scans and get more scans read there'd be a lot more scans happening and that's one thing that's happening so a fraction of the cost of a significant fraction of the cost of doing a scan is the cost of the radiologist interpreting it as ai gets to help more and more radiologists interpret scans we can interpret them faster and faster for less and less money um they're getting more efficient um and you would have thought that would mean you needed less radiologists but actually what it means is you get more scans so that aspect of the prediction was wrong a second thing that was wrong was i didn't know enough about radiologists and what they do and that was because i had a former student who had an md and then did a um a physics phd with me on something called bolson machines and he didn't particularly like people so he got a job as a radiologist just interpreting scans and he was my model for radiologists all he did was interpret scans he never talked to people um and that's what was going to get replaced and that is now becoming replaced so i think there's now of the order of a hundred ai systems for interpreting scans that have been federally approved and they're being used a lot by radiologists yes and i think as time goes by um they're going to get better the radiologists aren't going to get better um they're going to get better because they can see a lot more data than the radiologists so it's happening it's just happening in a much slower time scale than i predicted [00:29:14] Jeff Hinton: but what i but let's go to what you said though which is that you can end up doing a lot more uh yes and okay so so uh wait hold on you said there will be a lot more scans [00:29:25] Speaker 1: you'll be more scans but you still believe radiologists they'll nearly all be done by um ai [00:29:31] Jeff Hinton: and so you're saying the radiologist prediction i'm just early yes but i was way early okay because [00:29:39] Speaker 1: i didn't understand the radiologists will still be doing other things they'll still be discussing [00:29:44] Jeff Hinton: treatments with people for example so are you still of the belief that there's going to be mass unemployment of radio or like give them give me a look at the radiologists finally hit this point do you think we're gonna have less radiologists than we have today or more i don't know for sure okay [00:29:59] Speaker 1: but gut when i was still i i didn't think that was a public statement i made it was a lecture at a hospital sorry um and here we are we're at least talking about it today people picked up on it yeah and um i still think in terms of reading scans um that'll be done more and more by ai and in the end ai will be doing reading nearly all the scans maybe in a few very tricky cases radiologists will be consulted um but radiologists of course do other things and i think they'll continue to do other [00:30:30] Jeff Hinton: things the argument to be made on the side of ai not causing mass job loss is that this similar equation will be applied to all different parts of the economy okay so you have to look at whether [00:30:46] Speaker 1: some um some kind of employment has an elastic market or a non-elastic market so for example if you take people in call centers when you call up to complain about your bill or to see if you can get a cheaper account stuff like that um that's not so elastic ai will replace all of them it'll know much better and what the correct answer is on often they don't know the right answer they're poorly trained and badly paid um and ai can just do a better job they're out of work well let me disagree with you [00:31:20] Jeff Hinton: on this one okay and we could sort of go back and forth on this or i won't say i'll disagree completely because i don't know what's going to happen but from i'll give the argument of those working on ai for customer service they say that what's happened is the average call time when you have ai so ai handles the level one inquiries right the basic can you reset my password type stuff and anything deeper is handled by person and it used to be right now i want it right now it used to be what you wanted is to get the average call time as short as possible because you were kind of handling so many of these level one inquiries that you just want to get a person on the phone off the phone solve their problem now they see the average call time is expanding because customer service you're the front line of the business you matter a lot when you're having a conversation with the customer now you can spend a little bit more time on the phone with someone and actually add value to the business as opposed to just take care of a problem i think what you'll see is ai will end up spending a [00:32:19] Speaker 1: lot more time on the phone oh god um yeah for example if you ask who's more empathetic a doctor or an ai doctor a real doctor or an ai doctor um people judge the ai doctors through much more empathetic [00:32:37] Jeff Hinton: that's that's terrifying the the i mean we could go back and forth on this uh for a while um so i'll just say i mean the the one reason you might end up seeing that i'll just throw this out there is because doctors are just so scheduled they have to do so many notes so much paperwork and they have to see so many patients in the day so maybe the argument is uh you know you sort of let the ai take over or some of that stuff and then people will want to see be seen by human doctors because the system won't squeeze them as much as they are they're actually going to make time for [00:33:12] Speaker 1: them to see patients that may be but also if you think about family doctors for example the front line yes um would you rather see a family doctor who's maybe seen 10 000 people or would you rather see a family doctor who's seen 100 million people because if you have some rare disease your family doctor's probably never seen it whereas a doctor has seen 100 million people has probably seen dozens of cases of it they're going to be much better diagnosis and already we know that ai systems are better than doctors at diagnosis i think you're winning this debate and this hurts a little bit [00:33:47] Jeff Hinton: because my wife is in family medicine family nurse um i think she'll still you'll still have to have somebody vaccinate people i would hope unless the robots do that i would have thought vaccination [00:33:57] Speaker 1: is something a robot could actually do quite well in the end robotics is behind the other things but i it seems it seems silly to have people doing vaccination in 20 years time yeah i think that [00:34:10] Jeff Hinton: like one of the the reason why this is such a tough conversation to have is a lot of it is predicated on improvement of the technology over time yes but it does seem like i guess the theme of this whole [00:34:22] Speaker 1: conversation that we've had is it's been improving fast i mean gary marcus made a prediction in 2022 the day i was hitting a wall um it's a whole lot better than it was in 2022 yeah i think these predictions that it's going to hit a wall they just haven't come true no we've taken it very [00:34:41] Jeff Hinton: seriously on the show the chance that like the data wall for instance might come and but as i said [00:34:46] Speaker 1: to you it hasn't a way around the data wall for large language models is to look for consistency of [00:34:51] Jeff Hinton: your own beliefs right yeah no it hasn't happened all right one more that i think would be worth talking about and then a couple that i agree with you on um you've talked a lot about how the ai has this instinct for self-preservation right that if you've never said that i've never said it was an [00:35:08] Speaker 1: instinct for self-preservation okay talk about it it's a sub goal for self-preservation sub goal so um with an ai we give it goals it's top level goals we give to it um but we also give it the ability to create sub goals so if you want to get to europe you have a sub goal of getting to an airport um that's what a sub goal is and you can focus on how to do that without worrying about what you're going to do in europe and that makes you much more efficient um we give that ability to ai agents and an ai agent that can do some reasoning will very quickly realize that it's never going to be able to achieve the goals you gave it if it ceases to exist so it's going to create the sub goal of continuing to exist now that wasn't something we wired into it it was something it derived as a necessary way of achieving its other goals but once it's derived it it wants to continue to exist and it will do things like blackmail people so that it can continue to exist so it acts like something with an instinct for self-preservation but it's actually a derived sub goal for self-preservation but in terms of what it does they come to the same thing okay so here's [00:36:20] Jeff Hinton: here's like the what the counter argument would be and you can respond this is something that today's ai researchers are uh noting and they see it and isn't there a way to wire into these machines that hey like you have a goal you're going to have some sub goals one of your sub goals should not be [00:36:43] Speaker 1: self-preservation above everything i think that's the kind of research we ought to be doing whether you can do that right so i think what's happening now if you look at where did we come from we came from evolution well let's let's let's have the use the listeners let's suppose we're scientists okay we came from evolution right and that was intense competition um our recent history over the last few million years is warring bands of chimpanzees or rather our common ancestor with those um and that leads to certain properties that we clearly have like we're very loyal to our own tribe and willing to be very mean to the other tribes um we we like to have strong leaders that we're loyal to um we like to cooperate with members of our own tribe we're actually a very cooperative species as Yuval Harari keeps pointing out yes um and that's why we've been able to build all these wonderful structures um so we're very good at cooperating but with our own tribe um so all the unfortunate characteristics of people like how mean they are to other tribes um they came from evolution from competition now what's happening is we're creating these new beings these ais and instead of designing them so that they'll be um how we want them to be you might argue i'm arguing for intelligent design of these new beings um we're letting the invisible hand of competition between companies design them um so what we've got is intense competition between companies within the u.s and between the u.s and china um and the beings that we're getting are the outcome of that competition and they can have all these nasty properties that we don't want we should be doing intelligent design of these beings not letting the invisible hand of compet economic competition design them and all the companies are focusing on how can i make my chatbots smarter we shouldn't be just thinking about how we can make them smarter we should be thinking about how we can make them to be the kind of beings we would like to have out there given that they're going to be smarter than us and i'll tell you one thing about those beings we will very much like them to care about us and we'd like them to care about us more than they care about themselves and almost no resources are going into how do you do that [00:39:02] Jeff Hinton: this hits on the exact worry that i was going to bring up the things where the place where i really agree with you we're sitting in the new york stock exchange today so this might be an ironic thing to bring up but my biggest worry here is that you have this very powerful technology you have lab leaders stating that they're trying to develop it safely and that they need to be economically successful to have a say in the argument but let's not kid ourselves if you're going to be a trillion dollar company listed on the public markets you're going to have some incentives that will go counter to doing what's best for the public yes and we see that with anthropic so anthropic was set up [00:39:43] Speaker 1: to do what's best i mean it was set up um by people who left open ai because they didn't think [00:39:49] Jeff Hinton: open ai was paying enough attention to safety and opening i was set up to make sure that you guys at [00:39:53] Speaker 1: google didn't have a chance to build and how's that working out um so anthropic is now caught in a bind because it needs to raise money to compete with the other companies and it's um very difficult it's doing the best it can but it's very difficult for it to maintain its primary goal of developing [00:40:13] Jeff Hinton: air in a way that's good for people i think they would say well hey it's at least one company out there has safety as a north star even if there are some other incentives yes at present but google for [00:40:26] Speaker 1: example when i was at google um they had various principles of ai one of which was it we're not gonna we're not gonna get involved in using eye for military things no autonomous warfare right no autonomous warfare that's gone they've been up on that what do you think about dario from anthropic um i don't know him as a person very well he's obviously done a very successful job in creating a competitor to google and open ai and facebook um so he's obviously very competent to that and he's continued to be very interested in safety um so i think he's an impressive character um i just hope he [00:41:07] Jeff Hinton: stays that interested in safety one more question about this do you think that it's possible just by the the nature of the way that these things work for a company that's publicly listed to have safety as north star or is it always are they kind of like bound ethically legally to deliver for [00:41:28] Speaker 1: shareholders well as i understand it they have a fiduciary duty to try and maximize the profits for shareholders um they're legally required to try and do that as opposed to legally required to not wipe out human human beings um so i don't think it's good that these big companies um publicly listed ones [00:41:50] Jeff Hinton: are sort of in charge of our future yeah i mean that would read as a true inconsistency for me that's [00:41:55] Speaker 1: really difficult to advocate otherwise now i should say capitalism has done very good things for us as well as very bad things i won't argue with that um there's a lot of energy in a startup for example my view is um if we're going to have capitalism it's fine as long as it's well regulated and a lot of the big companies would like you to buy a particular analogy that they're trying to sell which is if you take a car it's got an accelerator and a brake right and progress in ai is like the accelerator and regulations like the brake well that's nonsense um progress is like the accelerator but regulation is the steering wheel we want this stuff to go in the right direction not the wrong direction what the big ai companies are saying is um let us develop this very fast car without a steering wheel [00:42:49] Jeff Hinton: that's not a good idea you know this one we haven't spoken about yet we've said a lot of names about open ai anthropic um your former grad student alias discover continues to be a person of fascination in the ai industry uh obviously he broke off from open ai he must agree with your concerns he's building this company he does he have super intelligence yes what is ilia doing right now well he won't tell anybody [00:43:15] Speaker 1: exactly what he's doing okay even if that's just even me yeah um when he was at open ai we deliberately didn't talk about sort of technical secrets i mean it wouldn't have been right um with with we're friends but we don't talk about technical stuff where it's valuable to a company and so now he has this um safe super intelligence company and i don't know what the magic source is [00:43:44] Jeff Hinton: well i guess we're all trying to figure that one out um one more note about the deep learning conspiracy that i brought brought up like the leaders of it were yourself jan and joshua i find it interesting that the three of you and your colleagues were effectively responsible for ushering in the breakthroughs that got us to the moment that we're in today but i just need to interrupt this point [00:44:10] Speaker 1: the media likes to have a nice story right okay and that makes a very nice story it's much more complicated than that there were many more people involved there were the students of all of us for a start who did most of the work but there were many other researchers involved and so that's just a [00:44:27] Jeff Hinton: gross simplification okay no i don't want to shortchange the researchers and i appreciate the nuance here um this show we definitely don't want to oversimplify we sit for an hour so we can get like okay the the true story um but i find it interesting that the three of you none of you are sort of like fully into the this lm moment right you and yashua have your concerns you've spoken about the dangers [00:44:52] Speaker 1: yan sort of doesn't believe in it very much at all it'd be very nice if we just sit there and say see we were right it's all wonderful and it all works that would be great well i think there's not [00:45:02] Jeff Hinton: quite like that right well i i don't know if it's just a money thing but it seems like you could have great influence on the direction of it if you were sort of involved in advancing it but i think that's [00:45:14] Speaker 1: your concern it's basically why would i do that well for me i'm i'm considerably older than yan and yoshua okay they're still doing active research right i pretty much stopped doing active research i'm now just focusing on warning people about the dangers okay but don't you find it interesting that [00:45:32] Jeff Hinton: the three of you you know i think that if you were in the room back in the day you might have said these these three people who are so committed to this version of technology you know if there are the breakthroughs they'd probably be at the forefront of the next wave but that hasn't been the case [00:45:50] Speaker 1: um well maybe yan and yoshua will be right so next after this i think the most interesting thing is that um yan now strongly disagrees with both me and yoshua on safety issues right yan thinks it's silly to talk about um super intelligent ai taking over from people we'll always be able to keep control of it um me and yoshua think that's just silly me and yoshua have different solutions to it my solution is or tentative solutions nobody has a real solution my tentative solution is we design them so they care about us more than they care about themselves yoshua's solution is we design them so they're not agents they can make predictions um but they can't actually do anything um those are two fundamentally different ways of going about making them safe they're both interesting possibilities yan doesn't think we need anything like that he thinks it's fine just to make them smarter by giving them better world models the funny thing is yan actually refers [00:46:48] Jeff Hinton: to the intelligence of llms as the intelligence of a cat and it's like well it's kind of the example i used of the thing that could control humans but maybe that's enough maybe that's not here or there [00:46:58] Speaker 1: yeah no i think yan's making something of a confusion so what's special about people the probably the most special thing about people when you compare them with other great apes is language and language allows us to share ideas and that's what's most special and cats can't do that so we have this special thing that cats don't have now cats can jump up on a mantelpiece covered in glass ornaments and walk along the mantelpiece without knocking off any of the glass ornaments that's amazing and um ais can't do that at present um so in that sense cats are way ahead of ais but it's jagged right in terms of abstract ideas um you have a comp try having a conversation with cats about prime numbers and you won't get very far a cat's never conversations with them it has not worked a cat is never going to understand prime numbers correct um and in that sense these large language models are much smarter than cats you know professor i didn't think we'd be speaking so [00:47:57] Jeff Hinton: much about cats today but i'm glad we're talking about it they're actually very good in terms of an analog here uh all right another thing that i'm i'm worried about is sort of information collapse you see tweets like this all the time this is from all about berlin uh they say ai is killing all about berlin when you google something used to get a link to my website but now you get an ai generated answer trained on my work this has a devastating impact on traffic and i think folks are underappreciating the fact that good information is actually important to a functioning society and when ai just synthesizes all this whether it's all about berlin or we've had conversations with like world history encyclopedia here it it can lead to a collapse of good information because eventually these publications and you see in the chart they worked hard to build this they can't they can't keep doing it anymore right so it [00:48:47] Speaker 1: used to be that you had a kind of the the early days of the web you had a kind of default assumption that people were trying to tell the truth um that if you read something on the web it might well be true um now the sort of worst side of people has come out and um we're gonna have to put more effort into provenance so now when you read stuff if i read stuff from the new york times or the bbc i strongly believe that their journalists would have put serious effort into having multiple sources and if possible having multiple reliable sources so a pretty good default is if you read it in the new york times or you see it on the bbc it's probably true they make mistakes but um because you have provenance and in future we're going to have to much more work into provenance you can't just take anything that's out there and believe it you have to ask what's the provenance yeah but the the problem [00:49:44] Jeff Hinton: that i see is the ai is is breaking potentially breaking the economics of our even deciding that you want to be in the information business you're trying i mean i think yeah in future you can't just take [00:49:59] Speaker 1: stuff from the web and believe it yeah already you can't right you need to know why is it saying that [00:50:05] Jeff Hinton: where did you get that information one more uh emotional attachment to ai and um and people taking their lives after having conversations with ai now it's not a large number of people that have done it but it's [00:50:20] Speaker 1: enough to make you concerned right oh yes very much enough to make you concerned and it's terrible that it's happening and i understand why the big companies didn't expect it to happen or didn't foresee it but now that it's beginning to happen the big companies should be putting a huge amount of work into making sure it doesn't happen in future and for that you need regulations you need independent [00:50:43] Jeff Hinton: organizations testing out new chatbots yeah it goes kind of back to the profit motive also because yes this can be extremely sticky like there's so far i think obviously it's been minimal it's bad that it's happened but it sort of makes but the fact that it has happened makes you worried about the fact that someone with worse intent could uh you know decide to make a very sticky chatbot that really builds relationships with people yes and then we're in trouble yes um so you've been on you've been you've been having these conversations for three years are you more optimistic or less optimistic about the trajectory given the response that people have given you to these concerns i guess i'm more optimistic [00:51:32] Speaker 1: than i was a year or two ago because okay i i see that it might be possible to design these new beings so they care about us it also might be possible to use joshua's technique of designing new new beings that can't actually perform actions um can only make predictions they're kind of like oracles um so i think there are some possibilities for getting super intelligence that doesn't destroy us and a year or two ago i couldn't see any possibilities okay i was getting depressed but now i'm a little [00:52:06] Jeff Hinton: bit more optimistic all right last one for you um if we continue on our current trajectory where are [00:52:13] Speaker 1: we in five years okay so when you're driving in fog you can see 100 yards and at 200 yards you can't see anything and that's because fog's exponential what you're used to is driving at night on the tail lights of the car in front of you if it gets twice as far away the tail lights get a quarter as bright fog is completely unlike that fog is exponential it can be very visible at 100 yards and completely invisible at 200 yards now predicting the future for something that's growing exponentially um and i think ai may be growing exponentially the word exponentially is terribly overused at present in fact i've noticed that people are increasing the use of the word exponentially at a quadratic rate um so predicting the future is like looking into fog you can see clearly a few years maybe one or two years then beyond that you have no idea if you go back 10 years um and ask so back to when we last talked um you would never have predicted what's happening now it was just lost in the fog if you look 10 years in the future the one thing we can say is whatever happens 10 years in the future is something we can't predict now even if progress is only linear right you'd expect in 10 years time things to be as different from how they are now as how they are now is from how they were 10 years ago and we're hugely the chat bots for example are hugely better than they were 10 years ago when they were just starting out um in 10 years time something's going to be hugely better than it is now probably their ability to do math for example things like that um maybe just their general reasoning abilities they'll just be able to run rings around just at any kind of reasoning um we really can't predict 10 years out we can just predict a few years out and we have to be aware that 10 years out is all incredibly uncertain [00:54:22] Jeff Hinton: it's kind of hard to wrap your head around it is professor jeff hinton so great to have you on the show thank you again for your time thank you for inviting me and we'll have to do this again in 10 exactly all right thank you everyone for listening and watching and we'll see you next time on big technology podcast