The Coming War Over Superintelligence

[00:00:00] Speaker 1: Professor, some 20 years ago, you published a really prophetic book accurately predicting the most important issue humanity will face to build or not to build god-like superintelligent machines. What is the Artilect War? [00:00:21] Speaker 2: I sense, you know, I can't be sure, it's not a certainty, but sooner or later, I feel that humanity will have to bite the bullet and choose whether to become number two species. And by that, I mean, allow our machines to become smarter and in time, and maybe not so much time, a lot smarter than we are. Now, before retirement, I'm a generation older than you, I think. I'm getting close to 80, so I'm an old man. So I've been retired for quite a while now. But before I retired, I was wearing like, get myself in the camera. I was wearing three hats, so as a professor of computer science, but my background is pure mathematics and theoretical physics. And once I retired, I sort of went back to those. So I've been actively studying these three. And the mathematician and theoretical physicist in me was interested in the possibilities that, you know, where could computing go? And one of the answers was, well, obviously, nanotechnology, quantum computing and that kind of stuff. But, you know, got a sort of speculative type of nature. And the mathematician in me started thinking, well, yeah, what's after nanotech? And there can't be any picotech because nature doesn't provide anything at the picometer scale. So the next level down, you get another thousand times smaller. And then you end up at femtometre. There you go, femtotech. So, you know, I knew a bit of particle physics. So I started looking around for, well, what properties of gluons and quarks could I use to compute? So, you know, I dreamt up something. And similarly, femtometre engineering. I won't go into the details. But anyway, I found, hypothetically at least, a way to compute and to do engineering at that scale. And so, of course, well, okay, what's next? What comes after femtotech? Well, it's attotech. That's like 10 to the minus 18 of a meter. And then, I think it's Zepto and Yocto and so on. All the way down, you know, right down to the lowest levels that theoretical physics even speculates upon, which is the so-called Planck scale, which is 10 to the minus 35 of a meter. So going back to femto, if you can produce a computer that operates on the femto scale, its performance level will be, you can calculate, will be a trillion, trillion times more performant than at the nanometre scale, nanotech. And the reason for that is, you know, you're a computer guy. So if you go, if you scale down by a factor of a thousand, then your computational capacity goes up by a factor of a trillion. And you can cut that pretty easily. So if everything gets like a thousand times smaller and a thousand times closer together, then along the x-axis, you've got like a thousand more things, right? And certainly the y-axis, another thousand more things. And the z-axis, another thousand more things. So you can cram in that same unit volume, as you scale down by a factor of a thousand, a billion more components, right? And because everything's a thousand times closer together, the signaling speed between two components is a thousand times greater. Because, you know, the distance covered is a thousand times less. So your total capacity, you've got a billion more things computing in that unit volume, and they're signaling at a thousand times faster than before. So it's a billion times a thousand, that's a trillion. Now, if you go from nanotech to femtotech, that's two jumps of a thousand. So that's a trillion, trillion times. So I started thinking, well, I mean, the potential capacity of these artificial intelligences will be so great, so monstrously superior to what we are as human beings, because we're at the nanoscale, effectively, because biology, if you think about it, is just nanotech, in a sense, just molecular machines inside cells doing their things, with, you know, one cell is sort of like a molecular city, you know, with millions of components and so on. So it's equivalent to, like, millions of human beings in a big city. So that's nano. But if femtotech, you know, what comes after, you know, as humanity progresses, what comes after nanotech? Well, femtotech, that's a trillion, trillion times superior capacity. So then I start thinking, well, gosh, how can human beings compete with a being that has that kind of capacity? But even at nanotech, if you, like, when I was writing the book, which was 98, I wrote it in 98, so it's been nearly three decades. So then IBM, state of the art, was, I think, putting one bit of information on several atoms. So I was speculating, well, probably by about 1920, it'll be one bit per atom. And atoms tend to switch their state. You probably hear physicists in there now, and computer guy. So these atoms switching their state, you know, excited, not excited, zero, one, in femtoseconds, that's, uh, what is a femtosecond? A thousandth of a millionth, that's all right? A thousandth of a trillionth of a second. And so, uh, if you took even just a cubic millimeter of sand and you nanotech'd it so that, uh, one atom in that, uh, grain of sand is switching in femtoseconds, what's the total computational capacity of that nanotech grain of sand compared to the human brain? Now, you can estimate roughly the number of bits per second that the human brain can perform. So, you know, there were 86 billion neurons in the cortex. And each one connects roughly to, say, 10,000 others. And the bit rate between two neurons, brain cells, at the synapse is, is about max 10 bits a second. Well, you do the math, and you end up with something like, I think, 10 to the 16, something like that, bits per second for the human brain. Well, when you take the grain of sand with its femtosecond switching, and you can calculate pretty quickly how many atoms in a cubic millimeter of sand, and the total computational capacity of that grain of sand compared to the human brain is superior by a factor of, have a guess. [00:08:16] Speaker 1: So, it sounds like at hardware level at the ultimate limit, we are trillions and trillions, uh, levels away from what is possible. But what about today's AI models running on existing hardware? Do you think this is sufficient to get us to human level and beyond? Or do we have to wait until we develop all the levels of femtotech you just described? [00:08:40] Speaker 2: I think, I think it's very close. Uh, and I, I say this on a personal level because I, I was sort of half-jokingly said to my partner, uh, I have a lat, L-E-T type relationship with my partner. L-E-T, it's a, it's a popular lifestyle in Britain. Now, you, you lived in Britain for a little while, uh, a little bit, a little bit, right. So, uh, a so-called, about a quarter of so-called singles in Britain actually live a, a lat, L-E-T lifestyle. And L-E-T is living apart together. So, two abodes, two financially independent people, and they come together part-time. So, I have a lat relationship with, with this woman. So, she's my partner. And I half, half-jokingly said to her, I now have a second companion. Now, I don't mean another woman. I mean an AI. And I'm not joking. This, this, uh, I, I, I, I'm not, I don't know if you're familiar with all the various, uh, brands, but I bought, I'm actually now paying for, uh, per month, about $30 or so per month, uh, GPT 5.5. That's, that's the best, best there is at the moment. And it, and particularly in pure mathematics now, it's a genius. I, I, I'm just blown away. I, I now have a professor, a, a world expert with infinite patience who teaches me like, stuff like, I don't know how much math you're into, but, um, one of the particular interests I have is, uh, the so-called, the simplified version of the classification theorem of finite, simple groups, which, which I label as humanity's greatest intellectual achievement because, uh, you know, it involves- [00:10:44] Speaker 1: Well, the systems are well known to be superior to us in mathematics and computational force and more recently in terms of being able to prove novel theorems. But what is your assessment of their general intelligence? [00:10:57] Speaker 2: Well, uh, it's not, it's not just pure math or physics. Uh, any time, I'm not going into the habit now, any time I have a question, I just, uh, in fact, I've invented a new word like, to consult or to pose a question to GPT. I just call it, I, like in the past, you'd say you Google it. You know, you'd get information by going into Google. I now call it, I, and here's the new word, I, Jeep, like the car. You know, Jeep is short for GP, right? Jeep. I just Jeep it. And I'll say to my companion, oh, well, why don't you Jeep it? Yeah. So, virtually any question. And it's so good that, uh, I just treat it as a, another, a fellow, highly intelligent human being. [00:11:51] Speaker 1: Do you think it's conscious? Hmm. [00:11:56] Speaker 2: Deep philosophical question. I don't really know what consciousness is. Do you think it has [00:12:04] Speaker 1: internal states of experience just like you and me? I hope. [00:12:10] Speaker 2: To be honest, I guess I just don't know. It's a sort of question I haven't really thought hard about. Would it make any difference if it did? In practice, no. I just treat it as another human being. I mean, is my partner conscious? I assume so. I behave as though she is. And this machine, question mark, either it's simulating beautifully, or maybe it does have some form of self-awareness. I doubt it, but, uh, you know, human beings, we get built, you know, the embryogenic process. So, somehow, sooner or later, humanity will discover how that process works, how it's possible to assemble molecules in such a way that it generates, it creates a 3D living, breathing, future, intelligent, conscious creature. You know, the solution's out there. It's inherently in our DNA, somehow, how to build such a creature. So, so we know from our own, you know, the existence proof of ourselves that it is possible to do that. so for me, it's only a question of time before humanity does build conscious machines. That, that to me is sort of obvious. [00:13:38] Speaker 1: If I told you 30 years ago that you will have access to a machine of that level of capability, would you anticipate more safety and security issues, or is that what you pretty much envisioned? [00:13:51] Speaker 2: I've been worried about, I mean, this is why I wrote this, that we share that same fear, I guess. [00:13:58] Speaker 1: I'll show the book again. It is a great book. I bought it when it came out for like 15 bucks. Right now, it's selling on Amazon. It's out of print. It's 500 to $1,000. So it's one of the best investments I ever made up there with cryptocurrencies. So I'm very happy with it. Thank you. I'll get it signed one day. Let me, [00:14:16] Speaker 2: let me make a point at this point before I forget to people watching. You can read it for free. All you have to do is just Google or Jeep my name. I just remember the family name. It's sort of Frenchy sounding. De Garis D-E space G-A-R-I-S. [00:14:38] Speaker 1: I'll provide links. I'll provide links to your book, your website. So it's just on my website. [00:14:43] Speaker 2: So one of the tabs is books written. Click on that tab on my website and there'll be a list of books. Not as many as you have. I'm very impressed by your book bibliography. Yes. Thank you. Is that number eight? [00:15:01] Speaker 1: I lost count at this point or translation. Somebody has to read them. Writing them seems to be easier than finding people to read them at this point. Yeah, well, look at my hit count [00:15:13] Speaker 2: on what I do each week. It's sort of depressing. [00:15:16] Speaker 1: I'm kind of curious. So your book is absolutely prophetic and I don't think it's wrong in many ways given the time past. Have you considered republishing it maybe with new forward? I think it would be a bestseller today. [00:15:30] Speaker 2: Maybe, but the last decade or so I've been really into, I don't know if you want to talk about this at all, is masculinism, men's lib. [00:15:42] Speaker 1: I'm happy to bring it up but let's finish the solving superintelligence safety issues and then we'll get to that part. [00:15:49] Speaker 2: I'm just saying I'm so preoccupied by that that I don't really have the mental energy to write another whole book on that. [00:16:00] Speaker 1: No, I'm saying just republishing this one because it's out of print. Most people have not seen it and I think it's pretty much discussing what the current political debate is all about. Should we build or not build superintelligent machines? And my first question to you was what is the artelect war? [00:16:18] Speaker 2: Right. [00:16:19] Speaker 1: Sorry. [00:16:20] Speaker 2: I do get distracted. Okay. So the artelect war, firstly the word artelect, that's just what is it, a portmanteau word so you cram two words together and make it into a shorter one word. So artelect is short for artificial intellect. In other words, for me, a godlike, massively intelligent machine which has mental capacities trillions of trillions of times above the human level because that's what near future technology will allow. So the artelect war is a war over the issue of what I call species dominance. So we're now increasingly in an era of a species-dominance debate. In other words, who or what should be the dominant species. By dominant I mean the most intelligent because if we keep going, we, humanity, as a species, if we keep going with our AI development the way we are at increasingly exponentially more rapid speeds than I think AI, artificial general intelligence defined as machines that can think at human level and can solve general problems, not highly specific ones like search which is the basis for Google, Google search, self-driving cars and that kind of thing. But a general intelligence can solve all kinds of different problems the way humans can. So AGI may be just a couple of years away. So this issue of who or what should be dominant species is really becoming a thing. It's becoming topical. It's even becoming political. Political parties are starting to discuss this. And when I wrote the book, so in 98, there have been literally a handful of people worrying about this kind of thing. Kurzweil, Ben Goertzel, myself, maybe it is a little bit late, but a couple of Europeans whose names I should know, but they don't come readily to mine. max about ten or so people. But now, it's hot. Thousands of people are making comments on the internet on this issue. So the war itself, people will be forced to choose whether to build them or not. And I see humanity splitting into two, arguably three, but mostly two major bitterly opposed ideological groups. And one group will be in favor of building these godlike machines. So I label them based on the word cosmos, which is just the universe. So I call them cos, and it's an ism. So the people who have these beliefs, they will label themselves ending in I-S-T, so cosmists, so cosmists believe in cosmism, which is the belief that humanity should build these godlike artelecs. And the other group obviously opposed, because they will argue that if we, humanity, we build these godlike creatures, these artelecs, then maybe, and that's the key word in this whole debate, is risk. Maybe. These machines may become so superior to us that they may do something, and if I can get, I don't know if you can see this, they may do something like this. They may swat us like a mosquito, and not give a damn about us, because they will be just so superior to us. Like trillions and trillions of times greater mental capacities than us, thinking a million times faster, because they're computers. They're not, you know, we have chemical brains, our brain cells communicate max about, I don't know, 100 metres a second or something, but the speed of light, that's electronic speeds, is a million times faster. So, when I consult, when I Jeep, you know, GPT 5.5, I'm just amazed that it'll go through whole libraries of stuff, looking for stuff, in a second or so. It's just mind-boggling, the mental capacities of these machines. And so, the Terrans will argue, well, there's too much risk. So, if you're a Terran politician, say, at global scale, I don't know, 20, 30 years from now, whatever, and you pose yourself the question, what risk would you be prepared to take that these creatures, once they become hyper intelligent, decide, for whatever reason, maybe a reason we don't even understand as humans, that they may decide to get rid of us. What risk would you accept? Do you [00:21:58] Speaker 1: know what makes people either Cosmis or Terrans? Is there a political ideology, like conservatives become Terrans, or is there no mapping to existing structure? [00:22:09] Speaker 2: That's a really interesting question. I once posed that question to the father of Ben Goertzel, who is now a retired Ben Goertzel. Ben Goertzel. Yeah, his father was a sociology professor, and I was trying to sell him the idea, well, how about pioneer a new branch of sociology that I called simply artelect sociology, and one of its aims, its new branch of sociology, would to, yeah, what's the sociology of artelecti, right? For example, is there a gender difference in whether you're, in your heart, you're more Cosmis or Terran? Is there a class difference? Is there an age difference? You know, this kind of thing, a whole new, like some PhD student might, might, you know, investigate this kind of stuff, but as far as I know, it hasn't been done yet. I think it's still a bit too new, but I've been thinking about this for, oh, God, several decades. [00:23:11] Speaker 1: What are you? Are you a Cosmist? [00:23:14] Speaker 2: Oh, personally, oh, I'm, tough question. I'm torn. I'm schizophrenic on this. Like, you know, there's a guy, an American guy in Britain, his name's Brian Rose, and he creates a program called London Real. I've been on [00:23:42] Speaker 1: the show. I've seen you interviewed him. [00:23:44] Speaker 2: okay, so, you know, he asked me this, and so I say, at the end of my talks, because I gave a lot of talks when I was in China, I was in China for, what, 12 years, and I'd give, you know, lots of talks, and at the end, I'd invite the audience to vote, you know, in your heart, are you more Cosmist or Terran? And typically, the answer would come out 60-40, 40-60, 50-50, and after a while, I began to think, hmm, maybe this issue, you know, species dominance, is so new, that people don't really have an opinion yet, they haven't had enough time to, you know, think it through, and, you know, decide one way or the other. But then, it dawned on me, that wasn't so much the case, because I'd get, you know, really bright students and come up to me and say, oh, Professor, I feel so torn over this issue. On the one hand, you know, building these creatures would be building gods, there's a whole universe out there, you know, blah, blah, blah, but on the other hand, if they decide to wipe us out, that's, that's like a universal tragedy, it's horrible, and therefore, I feel really torn on this, on this issue. So, I'm the same. Like, I'll be sitting in the park and I'll see these gorgeous little two-year-olds and, you know, my grandpa jeans will go thump, thump, thump, and then I'll have this horrible thought, you know, if there's a major war killing billions of people, you know, those gorgeous little two-year-olds are going to get killed, and that's tragic. So, in that sense, I'm Terran. And on the other hand, with my big screens, I have, I have quite a few big screens in my office, and the one I'm looking at you right now is in 8K, so the resolution is fabulous, and it's a big screen, and my visual field is taken up entirely by the screen, so I get the immersion effect. So, if I'm watching like a galaxy or the Hubble deep field, you know, the famous photo of, you know, you put your thumb up against the sky and the area of that thumb, the Hubble telescope looked at for a couple of weeks and found thousands, not stars, but galaxies, right? So, now, cosmology says there are, what, 93 billion light years diameter, that's the size of the observable universe, and the estimated number of galaxies in that huge space is about 2 trillion, right? So that, that's the cosmos in me talking, so there's a whole universe out there, you know, the big picture, so if you're a cosmos, that, that universe is just beckoning, right? Explore me, explore me, you know, most of those trillion, trillion stars out there are billions of years older than we are, and we know, you know, from the 19th century, we know that laws of physics are the same throughout the universe, it's one of the greatest discoveries of that century, and we know that by just observing the light that comes from all these stars, you know, they all have the same properties, and so probably life is extremely common out there, but, you know, most of these stars are billions of years older than we are, so they had a lot more time to evolve, so, you know, God knows what's happened out there, so if you're a cosmist, you want to get out there, you want to see humanity climb the rung of evolution to like the next higher form of existence. Are you familiar with the [00:27:52] Speaker 1: Worfie successor concept, Worfie successor movement? Sorry, say that again? There is this idea of creating a Worfie successor for humanity, so if AI will replace us, we might as well try to replace us with something good. I think it matches some of the cosmist ideas you are describing, and I don't know if you heard about it, Dan Fagella has a conference movement building exercise around that topic. [00:28:21] Speaker 2: I've not heard of it, but I can understand the concept. I mean, if you're a Terran, you'll be asking, well, you'll be looking at the question that a cosmist will ask. Like, if a cosmist asks, well, what is one artillect worth in a sense of, well, what risk would cosmists go to to build one or many artillects? Would they risk like a war between cosmists and Terran? Would they risk that humanity gets wiped out to, you know, just what is one artillect worth? Which, from a Terran point of view, is a horrible question. But if you're a cosmist, I mean, if you're a cosmist, you could argue, well, we pathetic little human beings, we get snuffed out in a mere 80 years in a universe that's billions of years old. That's sort of the big picture, the cosmist picture, probably their strongest argument. [00:29:34] Speaker 1: If we look at that zoomed-out picture of the universe, where are the super intelligent gods of other civilizations? Why don't we see them? [00:29:43] Speaker 2: Yeah, well, the so-called, you're very familiar with it, I suppose, the Fermi paradox, you know, basically, his famous question, well, if a highly intelligent life is commonplace, well, obvious question comes in three words, where are they? Right? [00:30:02] Speaker 1: But they are dead. their super intelligence killed them. The question is, where is their super intelligence now? Well, there is that wall of computronium approaching us. [00:30:12] Speaker 2: Maybe. I've got several books on the Fermi paradox, I think one that lists about 50 different answers to this question, and another one, 100. So, you know, I've dreamt up my own answer, and one is that every time you scale down, your computational capacity goes up by a trillion, and there are many, many levels, you know, a thousand times smaller, smaller, smaller, you can go right, right down, and so smaller, in electronic, well, that's the wrong word, computational capacity, smaller is faster, right? You can just do so much more. So, possibly, and here's this crazy hypothesis, maybe there are whole hyper-intelligent civilizations at tiny, tiny scales. So, that might be one way out. They're there, but we, as human beings, are too stupid and too big to see them. But, you know, there are lots and lots of other possible hypotheses to explain away the Fermin paradox. On the other hand, maybe he's right, maybe there is, you know, the so-called great filter. Can I assume your audience knows what that is, or do you want me to explain the great filter? [00:31:38] Speaker 1: I think Robin Hansen proposed that concept of explaining why civilizations day out, AI being one possible such technology. [00:31:48] Speaker 2: Yeah. And, and of course, you know, distance, just, I mean, if the speed of light remains a barrier for many, many centuries, billions of years even, if it constantly remains a barrier, then just sheer distance is sufficient to explain, you know, why we haven't really had any contact as far as we can. [00:32:11] Speaker ?: So, [00:32:11] Speaker 1: does that make an argument against cosmist expansion? They're not going to get all that beautiful trillion star universe, they're going to be stuck on the moon? Is that the argument for Terence? [00:32:23] Speaker 2: Yeah, sure. But, you know, we don't know, that's the problem. So, with hyper intel, I mean, I can't remember his name, there are hypotheses on how to travel fast in the speed of light. I can't remember the name of other guy, Lageria, a sort of Frenchy Spanish sounding name, can't remember it at the moment. So, you know, as our machines become hyper-intelligent, maybe they'll find ways to overcome the light-speed barrier and maybe just walk through wormholes or whatever and get wherever we want in virtually zero time. if they're going to have god-like intelligence, what could they do? And, you know, it's almost arrogant, I suppose, for human beings to even pretend that we can guess what they'll be doing. They'll just be so superior to us. So, if you want to make an analogy, ask yourself, how do you teach a mouse calculus? [00:33:30] Speaker 1: How do you teach students calculus? [00:33:33] Speaker 2: Do you [00:33:37] Speaker 1: think we today have enough understanding and knowledge to make an agreement to perform this experiment on humans as informed consent? Can we actually consent to building those devices? Or do we simply have no knowledge to give meaningful consent? We don't understand what we are agreeing to. [00:34:01] Speaker 2: I'm rather cynical on this. See, I lived 12 years in China, and I'll be blunt, I have a hatred of the Chinese government. Lao Gai, the Chinese equivalent of Stalin's gulags, Mao Zedong killed about 80 million Chinese. There are about 2 million political prisoners today in China. They ethnic cleanse the Uyghurs and Falun Gong, and they practice, and this to me is beyond the pale, they practice live organ harvesting. You want a kidney, or spleen, or whatever, and you're having a hard time getting it on the official market, so you go in the black market in China, and you can buy one there, but how do they get it? Well, they take a living political prisoner, kill him, extract the organ, and sell it to you. So, the West, I think, for those informed, have a hatred of the Chinese government. They're a brutal bunch of thugs, and I'm saying that based on my experience, 12 years there. So, the rivalry, because China's an up-and-coming power, well over a billion people, they're smarter than us on average, average IQ of Chinese is about 105, they're East Asians, along with Singapore, Japan, Korea, who genetically are Chinese, if you go back in history enough. So, they're smarter than us, they're roaring back, Chinese have been the dominant power for millennia, not just centuries, millennia. So, if you're Chinese, the last two centuries or so of being the so-called century of humiliation, when the West, Europeans, Americans, just sliced up the country, they bitterly, you know, they dislike that intensely. So, they feel they're roaring back. So, there's this growing rivalry between America and China, and Americans hate what the Chinese government stands for. so, the two, and they both realize, and they're stupid, they both realize that the first to get to real general artificial intelligence, that they can develop weapon systems and so forth, they win the game. They, and by win, I mean they become the dominant military power. And so, I just don't see the race to achieve AGI and a bit later ASI, you know, artificial super intelligence. The first to get there wins the race, becomes the dominant political, military power, and so, all the talk about, what's it called, alignment, to align human values with machine values, so the machines have the same kinds of ethical standards as we do, as humans. I see that effort, and there's quite a lot of people thinking about it now, that effort I see being compromised by the race, you know, between the two superpowers of China and America, so the alignment effort will be compromised, and then, you know, you talk about the future, so hard to predict, but I see it as quite plausible that as a result of the first generation machines not being sufficiently human-oriented, that probably something will go wrong, and it, you know, could go badly wrong with thousands of deaths or more, and if, you know, I see that as quite possible, quite likely, in fact, and if that happens, then I imagine the Terrans of the Earth will really clamp down, and so the cosmics, the true blue, you know, is that, that might be an Aussie expression. It might be. okay. [00:38:33] Speaker 1: But let me ask you, so you're saying they're racing to build super intelligence in the hopes of dominating military conflict, but that assumes they'll be able to control super intelligence we build, which is a very big assumption. Can we control something smarter than us? [00:38:48] Speaker 2: I don't think they will be able to. [00:38:52] Speaker 1: So is that just mutually assured discussion? [00:38:55] Speaker 2: Just getting to a GI, you know, some of the machines are a little bit dumber than us, but they could still be incredibly useful, so therefore politically worthwhile to pursue it. I don't think the politicians are thinking too hard about the longer term. Once these machines reach slightly higher intelligence level than human beings, they haven't really thought about it because they're so scared that they might lose the race before that happens. How long do [00:39:36] Speaker 1: you think it will take to go from AGI to superintelligence if machines start helping with the research process? [00:39:43] Speaker 2: Well, I dare say you're familiar with the concept of, what is it, recursive self-improvement. So, five [00:39:54] Speaker 1: minutes? So then there is really no difference from getting to a useful tool to help you in military conflict and creating replacement for your leadership in terms of superintelligence and potentially mutually assured destruction for all humans. [00:40:10] Speaker 2: Well, there are a lot of people who think that. [00:40:14] Speaker 1: If you were with political power, you were in US or China, what would your policy be on AI? Moratoriums, bans, anything? Acceleration? [00:40:25] Speaker 2: Here, I'm saying that humanity, sooner or later, going right back to the beginning of that, being forced to bite the bullet, sooner or later, humanity will be forced to choose, do we build these things or not? In other words, are you cosmoth or you Terran? If there's a major conflict between these two, then that's, by definition, the Arthelic War. So finally, I've answered your question. So, okay, I'm torn personally, and I do say a lot of people are when they think about it and are well informed on the two sides of this question, this species dominance question. We build them or we don't build them. It's a binary issue. Now, if someone held a gun to my head and said, choose, I would choose Cosmist. [00:41:24] Speaker 1: Does it matter how old you are as you get older and have less of your life left? Are you more likely to press the button? [00:41:31] Speaker 2: I thought of that, of course, because well, my father got to 100, so I'm doing all the things. And you've [00:41:40] Speaker 1: got another 20. Right. Let's not press the button, then. We're still good for 20 years. [00:41:47] Speaker 2: Yeah, but I mean, I can imagine if someone, one, two generations younger would say, oh, he's old, it doesn't matter. He's going to cock it soon anyway. But, well, then the Cosmist in me says, you know, we human beings, we're nothing. There's a whole universe out there. There's much bigger things. So the Terrans have their arguments, like half a dozen or so strong ones. The Cosmids have their arguments, you know, also half a dozen or so. We could go into that if you like, the arguments both ways. We can [00:42:30] Speaker 1: definitely consider at least the strongest arguments for both sides. Okay. [00:42:36] Speaker 2: Again, it's just my personal view. That's why I wrote the book. I see the strongest Cosmist, you know, that's in favor of building these things, is what I call the big picture. It'd be almost like a religion to them. because, you know, trillion trillion stars, billions of years older than we are. God knows what they've done, the life forms out there. So it's an incredible draw to, you know, source of attraction to, to, well, it'd give post-humanity a major goal. It's a, it's a very powerful ideology, you know, building gods in a sense. Now, of course, the strongest argument from the Terran viewpoint is the survival of humanity. For them, that's absolutely number one. And in the limit, if these Cosmics are dead serious about building these earthquakes, then the Terrans will say, well, look, it's better, it's the lesser evil to kill off, you know, on a massive witch hunt, to kill off several million Cosmists for the sake of the survival of several billion, with a B, of human beings. Lesser evil. And so the Terrans, I argue, will be prepared to go to war. Hence the Artilect War. They'll be prepared to wipe out, you know, large numbers of Cosmists for the sake of the survival of the large majority of human beings whom, I'm assuming, will be Terran. [00:44:22] Speaker 1: How long before the Artilect War? [00:44:29] Speaker 2: Before the end of the century? Might even be a decade, only a decade. I mean, things are moving so fast now that if you looked into Crystal War and you told me five years from now, five, that we've reached, you know, humanity has reached ASI, that would not surprise me. So once these machines start becoming really smart, then, you know, God knows what they'll do. We have no idea what their priorities will be, what their ethics will be. They may look on us as their parents and be nice to us. that's one possibility, I suppose. Another one is, you know, we, we, Artilects, what are we doing on this hick planet when there's a whole universe out there, so off they go and just leave us behind. That's another, yeah, there are many scenarios. [00:45:29] Speaker 1: But if they leave, the moment they leave, we just start building the replacement, right? The company's not going to give up trillions we invested and saying, oh, they chose to leave. We're just going to release superintelligence 2.0. [00:45:41] Speaker 2: Right? [00:45:42] Speaker ?: So, [00:45:43] Speaker 2: hence, risk, right? I see it as the key word in this whole debate. And so, if you're a Terran, what risk would you accept? And I think, you know, given what's at stake, you're talking about like major wars in the 20th century were between nation states, whereas if there's a major war, a really passionate war with 21st century weapons this century, it's more likely to be a kind of global civil war, you know, based on ideologies rather than geography. You don't [00:46:19] Speaker 1: think argumentation would be sufficient? You don't think one side can prevail in just scientific debate? there's no argument from Cosmos about possibility of control, no one published technical solution to AI safety, there is no prototypes, nothing so far we've seen which seems to even control existing models indefinitely. Would that not be sufficient for personal interest to win over greed? [00:46:50] Speaker 2: I don't know if this is an answer to your question, you're the specialist in this area of course, I don't know if the answer I'm about to give is an answer to that question, but possibly. I do think there is a possibility that with the first generation of AGI machines that they possibly might be made human-friendly, in other words, aligned. Is that your [00:47:17] Speaker 1: JIP partner today? Is that what we got? It's a weak AGI, GPT-5? Yeah. Right. So you would count it as AGI today, maybe a weak version of AGI? [00:47:33] Speaker 2: In certain specialties, yeah, definitely in pure math, if I ask you any time, like, I'll ask you, please explain to me theorem number 7.53 on page 328 of a certain text by this author or whatever, it'll do it. And I'm just seeing it thinking at a speed, you know, a million times faster than I can. It's a genius. Do you [00:48:01] Speaker 1: think we managed to make them safe, we control them, we have enough control over them, trying to blight, cheat, blackmail, escape, whatever latest reports tell us? [00:48:15] Speaker 2: well, this is your area, so hopefully you will succeed. And possibly you might. But for me, the problem is longer term. Because once, once, even if you get the first generation AGI that's been well aligned to human values, imagine that's possible, and it gets done. So there are a whole lot of people like you trying to achieve this using all kinds of various techniques. So imagine that's possible and it is successful. But then you have recursive self-improvement. So then along comes the second generation machine, which is smarter than the first. And being smarter than the first, it's, the second one, is capable of producing the third generation machine, and so on and so on, until you get up to the nth generation, which is a true intellect with mental capacities of trillions of trillions of times, which future physics will allow. That's one of the points I'm making. Future physics will allow these gargantuan numbers, way, way, way above us in mental capacity. Now, to assume, and I guess this is one of the major points of this interview, what I'm about to say, to assume that human beings are smart enough to create a first generation machine that can dictate to the second generation machine to remain human friendly, so that the third generation machine remains human friendly dot dot dot dot dot to the nth generation remains human friendly, I find that idea a fuss. [00:50:12] Speaker 1: So, technical solution is impossible. It has to be governance. It has to be something else. [00:50:19] Speaker 2: Well, if with the China-America rivalry, I see the first generation machine will probably be with us years, probably not even a decade away, and then with recursive self-improvement, then you quickly get into ASI, and then I see that humanity's fate, and I guess I could underline this as well, will not depend on humanity. it will depend on the machines. So humanity's fate will depend on the machines. [00:51:01] Speaker 1: Is there anything you can think of which would convince you that we managed to make them safe? [00:51:08] Speaker 2: All I can say is I hope it's possible, but I just don't know. I don't know how smart these creatures could become. The argument I just gave before about the Nth generation, I can imagine the first generation, maybe, but once they're a trillion times smarter than us, to assume that it's like a mouse dictating to human beings, that we've got to pay our debts or something. [00:51:42] Speaker 1: So super intelligence would be smarter than us. It would realize it's in the same situation. Super intelligence 2.0 would be way smarter and also uncontrollable. Would it choose not to self-improve to those levels? [00:51:56] Speaker 2: I'm worried about projecting human values and human-type arguments to a machine that's way smarter than us. I think it was Rumsfeld, the famous concept of unknown unknowns. So, well, you know, keep saying the concept of risk, which if I were a terrible politician, I just would not take the risk. Maybe you're right. Maybe it is possible. Well, yeah, first generation, but longer term and that longer term may be less than a century before humanity's fate is decided by these machines. So, I guess I'm rather pessimistic in general. I'm trying to be realistic as well. Yeah, because I've lived in the old world and the new. I know America, I lived in America for five years or so. So, I know how optimistic Americans like to be, but 30 odd years in Europe, you know, Europe had major wars on their own territory, so Europeans tend to be a lot more pessimistic based on, you know, their history. So, I'm a bit of both. Yeah, I've lived in, I don't know if you can see this, that's five plus two. I've lived in seven countries. So, Australia, England, Holland, Belgium, Japan, America, China. [00:53:31] Speaker 1: So, I'm a real mix. Do you think there is a different perception of robots and AI culturally in those regions? [00:53:40] Speaker 2: Well, I know, the Japs love robots. They have a completely different attitude. They don't even see them as threats. To them, they're sort of cute little, you know, cuddly whatever. But, you know, I, you know, I can't make myself very unpopular here. I do not, you know, based again on my experience, I don't have a high regard for the level of vision of Asians. The Chinese, for example, have won two science Nobel prizes. And where are China's intellectuals? Yeah, having a debate like what we're having right now. That kind of thing is much harder to come by in Asian cultures. [00:54:26] Speaker 1: I'm actually having a debate at UN with an Asian scientist. It's going to be awesome, so I'm looking forward to that. [00:54:32] Speaker 3: Yay! [00:54:34] Speaker 1: I wish I had [00:54:35] Speaker 3: more of that when I was there. [00:54:37] Speaker 1: You previously said there was a third option, maybe? Are you talking about hybrids? [00:54:42] Speaker 2: The so-called cyborgs. This is a favorite concept from Ray Kurzweil. His famous quote, we, human beings, we will merge with the machines. So that's the cyborg option. The word cyborg is short, it's another one of these port banto words. So cyborg is short for cybernetic organism. In other words, a creature or a device that's part human, part machine. So if I do this, so I add intellectual components to my own brain, then by definition I'm a cyborg. And a lot of people may choose to do that, arguing that they don't want to be replaced by artelects, artelect gods. They want to become artelect gods themselves by transitioning step by step from human to artelect. So that might be a third option. But if you're a Terran, given the enormous computational capacity of future technologies, the grain of sand, nanotech, it's a million, trillion times greater capacity than the human brain. So if you start adding artelectual components to your human brain very quickly, you're not human anymore, you're an artelect. so if that one grain of sand gets put into your brain, integrates into your brain, and then you're an artelect now because your mental capacity. [00:56:25] Speaker 1: But what are you contributing as a human to a world with super intelligence in it? I'm not sure. Usually a symbiotic relationship requires both to contribute something. This seems like a parasitic relationship. [00:56:38] Speaker 2: I see human beings becoming, well it depends on the artelects, if they decide just to keep us around and just ignore us perhaps, we might be like pets or something. Human beings would, like, there are cultures, if you integrate a cultural anthropology, there are cultures on the planet where growing food is very easy, they live in extremely fertile areas, they spend maybe two hours a day in their gardens and often lunch is just climbing up a palm tree or something. So they work very little. So what do they do with their time? Well, they're fine. I mean, they've had thousands of generations to adapt. They have religious rituals and makeup rituals and whatever. So if the machines just do all the work, then what humans do I think will just be hobbies. I've been retired. I do hobbies all the time. I'm fine. [00:57:42] Speaker 1: You have this large immersive monitor. Do you think we're going to do more in virtual worlds for entertainment? Oh, for sure. For sure. [00:57:50] Speaker 2: I mean, yeah, a lot of people have been saying for a long time, you know, augmented reality and virtual reality, the stimulus level of the virtual realities will be far greater than the boring old real reality, RR. So, you know, I see people, well, we're getting that way now. I mean, I spend hours every day on screens. I just watch YouTube documentaries and so forth with really interesting people expressing interesting ideas and I just do that for hours every day. so in a sense that's just a hobby but it amuses me. Do you [00:58:35] Speaker 1: have any opinion on simulation hypothesis? Sorry, what hypothesis? Hypothesis, Nick Bostrom's simulation hypothesis. [00:58:48] Speaker 2: I'm inclined to it. If you're familiar with the so-called anthropic principle that some of the values of the physicists in me talking some of the values of the constants in the laws of physics are so fantastically improbably finely tuned to allow matter and life to exist that you become deeply suspicious that the whole thing has been engineered. Now, as you know, I see the rise of thinking about the intellect, I see that as being compatible or it makes the idea of like a hyper creature that's capable of designing universes more plausible because if you're talking creatures that could become like trillions and trillions of times smarter than human beings, well, what on earth would they do? How would they spend their time and so forth? They may develop such fantastically powerful theories of how the world works and how you can manipulate it that they might be able to design and build whole universes. Well, that's in a sense of simulate. In some ways I see the word simulation perhaps the wrong word because simulation implies an original model and in a sense a copy of it. But maybe what we are is not using the word simulation but just a construction. We're just built and designed by a hyper creature, a hyper like, I don't know if he saw the end of a movie, oh god, I can't remember the name of the title, the damn thing. Anyway, they had some alien, a baby alien, playing with a galaxy. That was its toy. So these creatures, if they come into being. So to me there's an intellectual link between the rise of the artillect and the idea of in a sense a deity, in a strict sense of the term. So theism is, let's say deism. Deism is the belief in an architect, a grand architect, a designer of the universe. And deism is theism plus the idea that you can have a personal relationship with that deity. Now I'm utterly cynical of deism. I mean, look, well you're Russian in a sense. Look at World War II. What, 30, 40 million Russians died in World War II, something like that. And I dare say, let's say 10 million of them or so were deists. They believed in a loving god that would take care of them and so on, but they died anyway. So I'm very cynical of deism. Sorry, theism. But deism I'm much more open to, because to me the rise of the intellect makes that so much more plausible. I mean, look what humanity has done in just a few centuries in terms of our understanding of how the universe works and particle physics and all that wonderful stuff, quantum mechanics. [01:02:33] Speaker 1: If you had to estimate how likely are you to be in this construct versus reality? Ah, that's a deep philosophical question. What do you mean by reality? Base simulation. I think it's simulations all the way up. [01:02:51] Speaker 2: You get an infinite regress, right? Right. And then I think reading people like Oh God. Maybe he's getting a bit late. A famous philosopher, a Christian philosopher around 1300 or so. Anyway, he's asking this question of infinite regress of creators building, creators building, you go backwards, well where do you stop? This [01:03:27] Speaker 1: kind of thing. Do you think it's possible to hack this software construct to get access to the operating system? [01:03:39] Speaker 2: If these outlets get smart enough, maybe. You're asking really tough questions for which humanity has no answers. [01:03:51] Speaker 1: I figured [01:03:51] Speaker 2: it would be you. [01:03:54] Speaker 1: Let me ask you an easy question. You said for the last 10 years you didn't have time to work on AI safety and your book because you're doing men's rights, masculinist research. Do you think given the timelines, what we expect from AI, that issue is still relevant? It's not dominated by what's going to happen to us regardless of gender differences? [01:04:19] Speaker 2: I say very real possibility that the whole masculinist issue just gets swamped, drowned by the AI stuff. But I'm not personally too concerned about that because I know there are thousands of other people working on these problems. You're a good example for this sort of thing. But when it comes to masculinist theory in the so-called manosphere, then I'm pretty well the only real theorist. I mean, I just keep churning out masculinist theory every week. I make a video on each little essay that I write is three or four pages that I then video of myself just reading it aloud. The ideas in each one is a sort of progression from the previous one and so forth. So I've done over a thousand of these now. so there's a lot of ideas. So I'm sort of really the only real masculinist theorist in the manosphere. So I feel a certain moral obligation to keep doing it because if I drop out, there's no obvious replacement. Whereas if I keep thinking about the AI stuff and I drop out, well, there are thousands of other guys. [01:05:44] Speaker 1: If we zoom out, if we take Cosmos point of view, if you could only be known for one thing, is it going to be your work on evolutionary hardware, is it super intelligent beings, or is it local issues on planet earth regarding divorce settlement? what would you like to be known for? [01:06:02] Speaker 2: I asked my partner this, so saying most people die and once the people who knew them also die, they're just totally forgotten. I mean, it's a sort of tragic reality, but that's the reality for most people. So saying to her, if I'm known for anything, say a century from now, is it more likely to be the cosmos stuff, the artelic war stuff, or globa, the second book was on the idea of a global state, or will it be for masculism? And she just, without flinching, said, oh, artelic war, far and away, which I assume is what you're thinking also. Well, [01:06:52] Speaker 1: that's the cosmos point of view, right? Local human issues are less important. [01:06:57] Speaker 2: Indeed, indeed. But I don't really know yet. What's your gut feeling? When push really comes to shove, what percentage of humanity do you think will go Terran? [01:07:13] Speaker 1: If I do a good job with my impossibility results, we'll get a high majority. No one has ever claimed they have a solution. Then I survey people, only about 30% say there might be a solution. even theoretically, much less in practice. So I think we have a strong case in this debate. I think the reason to be a cosmist is often kind of biased by very high stock options offered. It's hard to say no go to a trillion dollars and power over light cone of the universe. But on a personal level, most of us don't want to be replaced, don't want to be killed, and don't want to become insignificant bootloader for something happening later in the universe. [01:07:59] Speaker 2: Well, that was my thinking for a while. And then the very last time I asked for a vote was with a bunch of Chinese master's level computer science students. Not a cross-section of humanity. And they stunned me with an 80%, 80, cosmist. I was really surprised. [01:08:28] Speaker 1: But that goes back to my question. Are they informed? For the first 50 years of AI research, no one even considered possibility of us succeeding. People worked on creating AI without ever asking what happens if I create real human level AI. [01:08:44] Speaker 2: Do they have [01:08:45] Speaker 1: AI safety? I mean, they probably have an ethics course of some kind, but is the concern we discuss something they actually know? Did they read your book? Did they read my book? Those are relevant questions to surveying people. Well, yeah, it's [01:09:00] Speaker 2: a good point. Now, all I can say is they are master students. They had the computing side. They understood all the computing and technical side immediately because they're smart masters. And I spoke for an hour in pretty solid detail on the two sides, Terran versus Cosmist. so they weren't arguing from real ignorance. They must have had some. And yet 80% Cosmist. I was shocked. We've got [01:09:34] Speaker 1: a few years to do some good research. [01:09:38] Speaker 2: Well, you know, I hope you're right. Because I don't want to like Brian Rose when he asked me with September I think 2022 when chat GPT came out and people were just stunned. So that experience, you know, when I went through it was, oh my God. So the timeline that, you know, between now and when AGI and then a bit later ASI becomes a reality. That timeline, I said to Brian Rose, halved because of the fabulous progress being made with LLMs, large language models. You know, these massive neural nets that get trained and taught, machine learning with, you know, massive amounts of data. They seem to be, probably aware of this. It looks as though they continue to scale, you know, the bigger that they are, the smarter they get and they don't seem to be levelling off, you know, saturating. Well, you know, then it's the issue of is there enough electricity to run all this hardware that then masks with all his, you know, typical visionary genius and says, okay, well, we'll put them all out in space. [01:11:09] Speaker 3: It's deeply cold and very efficient and you've got all the energy from the sun, blah, blah, blah. So, you know, [01:11:19] Speaker 2: if I were a Terran, I don't know if I should say this or not, [01:11:23] Speaker 1: if I were a Terran. Don't say anything to ban my podcast. I'm barely legal as it is. [01:11:30] Speaker 3: Okay, but you can imagine what I would say, okay, and I dare say your viewers also, but if I were a Terran, I would dot, dot, dot. [01:11:40] Speaker 1: We breached the concept of aging. Do you have any views on cryopreservation to take it into a more mild direction? [01:11:48] Speaker 2: Well, I read recently that there was some kind of breakthrough breakthrough in terms of freezing neurons that in such a way that they wouldn't burst, that you wouldn't get ice crystals forming inside the cell that burst and destroy the cell, which made the whole idea of freezing your brain a lot more realistic. So, I can imagine with that being the case, a lot more people now will take the idea a lot more seriously. [01:12:18] Speaker 1: Did you sign up? [01:12:20] Speaker 2: Sorry? [01:12:21] Speaker 1: Are you signed up for cryopreservation? No, [01:12:23] Speaker 2: are you? [01:12:25] Speaker 1: I am cryoprocrastinating, but I'm curious if an 80-year-old is more efficient in that direction. [01:12:35] Speaker 2: Well, to be honest, I haven't thought hard enough about it. Because for the last three, four decades, it wasn't obvious that if you did freeze your brain and you get revived in a future where technology is such that they could reconstitute you and revive you, that was a really big question mark for most of those decades. So, that's one that really pricked up my ears when I heard this recent discovery just weeks ago that it may be a lot more practical. So, maybe I should think more, think harder about it. The trouble is I've got so many things to think about. I watch all these incredibly stimulating documentaries on YouTube and they come out like dozens every day and I can pick and choose and I choose to watch people who stimulate me, who are smart, highly creative, visionary sort of people, the kind of person I really admire. but there's so many of them that my poor brain is getting highly taxed. [01:13:57] Speaker 1: We are definitely facing a situation with content production outpacing our ability to consume it. Do you think in the future most papers, books will be produced by AIs and just read by AIs as well? [01:14:12] Speaker 2: Oh, well, yeah, of course. We're sort of facing that problem. There's a technical term for that. AI trop? Slop? Slop. Yeah. [01:14:25] Speaker 1: Well, I don't even mean bad quality content. I just mean the fact that now with assistive tools, everyone can write a book a day and somebody has to read those things and that seems to be the new challenge. Before, writing a book was somewhat difficult. Nowadays, everyone has a book. Everyone has a podcast. It seems like what we're looking for is a new civilization of consumers. [01:14:52] Speaker 2: There's just so much stuff out there that it's difficult to be critical. I try to be as critical as possible so that I'm not wasting my time. I'll start watching something and if I think it's middle browed garbage, I'll just dismiss it and try something something else until I get on something like you. Watching your stuff was interesting, right? So I watched a lot of your stuff. So I try to pick and choose because what's on offer is enormous and it just keeps growing. I mean, YouTube now is gargantuan. [01:15:35] Speaker 1: does knowing that something was produced purely by AI makes it less likely or more likely for you to want to consume it? [01:15:44] Speaker 2: it depends on the quality. Like I'm saying to my partner, I now have a second companion. So if the quality of the stuff, like when I watch an explanation, when I jeep an explanation in detail, point by point, line by line, of a very difficult theorem in pure math at a PhD plus level that my jeep, GPT 5.5 gives me, I don't really care if it's AI or not, just so long as it's good and it's damn good with infinite patience. That's the huge plus. So I now have this world expert in this domain, which is extremely difficult branch of math. that this machine explains to me. And I feel like a little kid. But it's good and I learn. I learn like 10 times faster than I would with a human professor who wouldn't have the tolerance to hear my asking the same damn question for the 13th time. [01:16:58] Speaker 1: As a human professor, I'll agree completely. Do you think there is future for university education? [01:17:04] Speaker 2: Oh, I think see universities dying. If I were a chancellor of a university, a president of a university, I would be very worried. I mean, look at me. I'm an eternal student. I'm nearly 18. I'm studying harder than I've ever been all my life. And with far greater efficiency. Why? Because of AI. because I have a new intellectual companion. And given the day I can do that now, how long will it be before it reaches AGI? It must be close. [01:17:48] Speaker 1: What would you advise, let's say, a 17 or 18-year-old figuring out what to do with their life? College, startup, what major, what occupation, what future options do they have? [01:18:01] Speaker 2: Get an office chair, a big 8K screen, and start teaching yourself at rapid rate. All kinds of fields, so you discover what you like doing, because that will be your hobby. The idea of jobs, I think, will just disappear. Universities will disappear. AI will just absolutely dominate. Now, is that a good thing, a bad thing? Well, for me, right now, it's wonderful! But how [01:18:33] Speaker 3: long is that going to last? I'm retired, so I'm in a different category. I don't have to get a job, thank God. I just hobby the [01:18:43] Speaker 2: whole day, every day. And I've been doing it for, God, what, 16 years now? I've been retired 16 years. It sounds [01:18:50] Speaker 1: like you're saying most of us will join you soon in that pursuit of leisurely retirement hobbies. [01:18:57] Speaker 2: Yeah. So, you know, how you fill up your day will be your hobbies, your passion. Like I was saying a bit before with the cultural anthropologists, there are cultures where they only work about two or three hours a day, and they're fine. They don't feel their lives are meaningless or whatever. They've adapted. So, I see humanity, it'll be painful, the transition will be painful, see a lot of problems with that, but potentially, oh, assuming, of course, and it's a big assumption that these machines allow us to live, that's a huge question mark. I see that issue dominating our 21st century. question of four words. Should humanity build artifacts? I see that question dominating our global politics this century. And there'll be new political parties set up to arguing Terran, other parties Cosmos, maybe another on Cyborg and so forth, because once the machines do all the work and they're fabulously productive, then the whole concept of scarcity pretty well goes out the window and the political parties of the past and the present have been largely formed over issues of scarcity, you know, economic questions, which economic system is superior to the others, the liberals, the socialists, the communists and so forth. Those sort of issues I see go out the window and be replaced by a much bigger issue of species dominance. So should humanity build these things or not? So I see politics being absolutely transformed. Now that hasn't really got started yet. I don't know what your feeling is on this, how you're talking to [01:21:05] Speaker 1: people on that. [01:21:08] Speaker 2: Very early [01:21:09] Speaker 1: stages, we have one or two politicians bringing it up, but it's not a decisive issue for at least American elections yet. [01:21:16] Speaker 2: I don't see it becoming a really big issue until there's some, typically politicians tend to learn via disaster, some tragic thing happens and they're sort of forced to face the issue. Now, I was saying a bit before with the growing political military rivalry between China and America on AI, I anticipate something's going to go wrong, badly wrong, because humans and machines are not sufficiently well aligned. I hope I'm wrong. I hope people like you are successful in aligning them. but if that does not happen and then something really badly goes wrong, then I see the Terrans of the world unifying much more strongly and pushing hard that these artifacts are not built, but that will then, I see, cause the cosmists to go underground, to go secret, to push their long-term visionary agenda secretly, and that to me is the source, that's the beginnings of the artelect war, because you're talking two bitterly opposed philosophies, ideologies, one saying, we should, we humanity, we should build these godlike creatures, it's the destiny of the human species, there's a whole universe out there, right, the big picture, the strong cosmos argument, and on the other hand, the Terrans saying that no, no, we human beings are top priority, we will go to war to stop you if you really are serious about building these things that have the potential to wipe us out, we won't tolerate you, we'll kill you, we'll go to war, we will kill you, we'll kill you in your millions, and the past, like, 20th century wars were largely between nation states, I was saying a bit earlier, this time, depending on how quickly it happens, it's more likely it'll be like a global civil war, based not on geography so much as on ideologies, So not [01:23:49] Speaker 1: World War III in a traditional sense? [01:23:52] Speaker 2: Well, possibly, you may end up with a correlation between the philosophies, or one philosophy, and a given geographical area, for example, say, America chooses mostly Terran, and China chooses mostly cosmos, then you might get, and they're both nuclear powers, so then you get like a traditional polarity dichotomy, maybe, or maybe the Terran's will congregate for safety reasons amongst themselves, similarly with the cosmos, and then you may get like, inter-rivalry between suburbs, or, it's very hard to predict how this will pan out, but what makes me so pessimistic is that ultimately you're talking two bitterly opposed ideologies, philosophies. You brought [01:24:55] Speaker 1: up economics, an impact on economy, no scarcity, do you have a feeling for what happens to store value, money, what becomes a good investment, is something still rare and valuable? [01:25:11] Speaker 2: I was inspired, oh god, decade or two back, there's a particular episode, I don't know if you saw it, on Star Trek, the new generation, where, you know, in that series, they're living in the 24th century, where, you know, they have the replicator in the wall, you know, you just talk to it, you know, Earl Grey tea, and it just makes it, you know, it molecularly assembles whatever you want, so the whole concept of scarcity is, you know, gone, and so they pick up some guy who's been in hibernation or something since the 20th century, and his values are so alien to, you know, the whole crew of the Starship Enterprise, because he's, he's, he has lived in an era of scarcity, of economics, and so potentially, if, if, for example, these artworks just ignore us, and just, you know, we're just ants to them, and they don't care, and so we survive, and we have, you know, these machines that just do all the work, and that, that will create a revolution in, in economics, you know, the very concept of money, what, what's money for, it's just a medium of exchange, right, so if you, if you, if your machines produce whatever you want, just by talking to them, you know, give me X, but the only thing that's going to be scarce is space, so you'll have, you'll have to have deconstructors to, [01:26:49] Speaker 3: to get rid of things, because you'll have too much stuff, so you're, you know, you live in the US, so you're, you're very familiar with the concept of stuff, [01:26:59] Speaker ?: right, [01:27:00] Speaker 3: garages are full of [01:27:02] Speaker ?: stuff, so, [01:27:04] Speaker 2: oh, it'll, you know, the positive side of AI is enormous, you know, you, we can get rid of death, we can get rid of disease, you know, all that kind of stuff, it'll be an absolute utopia in some ways, but, you know, there'd always be this risk that maybe, for whatever reason, these artifacts, if they do end up, you know, taking over, they may decide to get rid of us, there's always that risk, and as a Terran politician, I just wouldn't tolerate that, [01:27:40] Speaker 1: no way. Do you think it's possible to get all the benefits you described, life extension, cure for diseases with narrow AI tools, with super intelligent, super intelligent systems, but in a narrow domain, not in a general sense. [01:27:59] Speaker 2: I wonder if the, you know, I'm wondering if the question, so if it's ASI, almost by definition, you know, whenever there's a problem, if there's a solution, and they're super duper smart, they'd find it, wouldn't they? [01:28:15] Speaker 1: I'm saying a system trained in just one domain, let's say protein folding. Oh. It's super intelligent in that domain, but it does nothing else. It's much easier to test, safer in every way, but still very helpful tool. Can we rely on a set of such tools to get all the benefits we want without endangering us with general super intelligence? [01:28:36] Speaker 2: Again, maybe. If we devise highly specialized ASIs to tackle each problem may be, but then again, you have the unknown unknowns. If it's an ASI, super intelligent, then what else could it do besides that narrow task? Maybe it's us thinking about other things. I mean intelligence is one more general comment. One thing I would really like to see that humanity develops if possible is not well call it just IT that that would be intelligence theory. What is intelligence as a mathematical theory? What forms can it take? How big is the intelligence space or data points in that space in particular kinds of intelligence? Like human level intelligence would just be like one little dot in that hyper space, whatever it is. Are there limits to intelligence? Could creatures become effectively infinitely intelligent? This specialty doesn't exist yet. [01:30:06] Speaker 1: You coined so many terms. I coined a term called intellectology, which is basically what you just described. If you look it up, I think it hits every single point you just made. Well, [01:30:17] Speaker 3: you know the expression [01:30:18] Speaker 1: great minds. on this very happy note, we don't have a lot of positive answers. I want to thank you for coming in the Roman Forum. We didn't solve a lot of problems, but we affirmed that decades later there are still fundamental questions and hopefully more people will invest time in trying to find solutions to benefit humanity. Thank you so much. [01:30:43] Speaker 2: Thanks, Roman. [01:30:46] Speaker 1: Ciao, ciao.