About this transcript: This is a full AI-generated transcript of The Ex-Pentagon Chief Sounding the Alarm on AI Weapons — Brad Carson from Machine Learning Street Talk, published June 6, 2026. The transcript contains 16,162 words with timestamps and was generated using Whisper AI.
"So I remember when I was first elected to Congress 20 years ago now. The Congressional Management Foundation gives you this book of kind of like, how to be a congressman, how to run your office, what to expect day to day. And I remember reading in the back, they had a survey of the existing..."
[00:00:00] Speaker 1: So I remember when I was first elected to Congress 20 years ago now. The Congressional Management Foundation gives you this book of kind of like, how to be a congressman, how to run your office, what to expect day to day. And I remember reading in the back, they had a survey of the existing Congress. And they said, how much time a day do you have to read and to get smarter about issues? And the answer was 17 minutes. We control the most important part of AI, and that is the chips. We can stop other countries from developing super AI in their tracks. If you're in Gaza, Keith, you have 0.73% that you're a Hamas terrorist. And what makes that? Is 0.73% like, do you get struck for that, or are you off the list for that? What's the threshold? People will often say this about AI, like, it's coming, you have to accept it. We regulate and change technologies all the time. And so I do think there's a world where we should not just accept the future as being determined. We shape it actively. The anthropomorphic tendency we have when we see language, we love language. We think it's the unique human skill. And when this machine turns it out, we know through AI psychosis and other things that people think it's a person. And therefore, they're giving the rights of persons to something. And that, to me, is a very dangerous thing. But it's a machine, and we should treat it like a machine. I mean, the fear of the Soviet Union was real and powerful. But nonetheless, we had people, both parties, wise men, who got together and said, like, this arms race thing is going to kill us all. How you win wars is with people. You know, that's the fundamental thing. And I think it is, as I often said from my time in the military and watching this all go down, the American way of war, in many ways, is substituting capital for labor. We love bright, shiny objects. We think they're technical solutions to vexing human problems. And we're always betrayed by that. Because in the end, when you go to Iraq, you go to Afghanistan, you go to Iran, all the fancy kit, they can reduce your city to rubble. Right? But the only thing they can't do, and only humans can do this, is basically come, kick in your door, and occupy your place, and, like, you know, re-instantiate a new government there that we like to see. That's the human endeavor.
[00:02:16] Speaker 2: This episode is supported by CyberFund. If you're building at the frontier of AI, they want to hear from you. CyberFund believes the future belongs to AI natives who want to achieve the impossible. And that is why they're introducing the Monastery for AI native founders. It's an environment of pure focus and rapid execution for founders operating at AI native speed. And they're offering teams $2 million each to participate. Apply now at cyber.fund.
[00:02:53] Speaker 1: I had a longstanding interest in artificial intelligence, just as someone interested in kind of public life, society, how one governs, an ever-changing world. So I followed AI for probably 20 years, casually reading about it. When I was in the Department of Defense running the Army, I had the chance to work on the law of war. We oversaw that for the entire Pentagon, for all of the military services. And right as I was leaving, the question of autonomous weapons became an important issue for the Pentagon. We were sending Army lawyers over to Geneva to talk to the Red Cross about, you know, possible treaty or address their many concerns about it. I was following it there as well. And I often tell people one of the great qualities I have, which isn't that important, but it's helped me a lot in life, is I always answered the phone and took a lot of cold calls. And so I was a professor at the University of Virginia for a short while after Obama left office during the really the first Trump administration. And one day the phone rings and it's someone who I'd never heard of before. And my phone in the office never rang. And I thought it could be an emergency or something. And so I picked it up and he says, I'm Anthony Aguirre. I'm a physicist at the University of California System. And so we want to invite you to this amazing AI conference that's going to happen over the New Year's in Puerto Rico. And I said, well, I don't know that I can really add a lot to such a conference and it's a long ways away. He said, oh, we have a stellar panel of people coming. I said, yeah, like who? He says, you'll probably know some of them. He said, well, Dario Amadei? I said, never heard of him. He said, Stuart Russell. I said, nope. Yoshio Bengio? I said, I'd never heard of these people. He went about six deep into this until he finally said, well, Reid Hoffman and Elon Musk are supposed to come. I said, well, I know them just from kind of the business pages of the newspaper or from pop culture. He said, it'll be great. There's only 100 people coming. You should visit this event. I said, like, well, New Year's, it's warm in Puerto Rico.
[00:04:52] Speaker 3: I said, why not? It's already a massive problem that other industries have successfully captured regulation, you know, to their advantage and to the detriment of the rest of us, the people who are not the biggest players. In fact, one of my favorite sci-fi shows, even though it only lived for one year, is Firefly. And then there has this excellent quote where Shepard Book says, you know, a government is just a body of people, usually notably ungoverned.
[00:05:23] Speaker 1: It is a daunting problem, but I think there are examples in government that have been less captured by the industry. And those can prove some kind of paradigm for us as well. I think having Congress have strong oversight of it, having a lot of democratic accountability. And, you know, one of the things we advocate for are ways that actually minimize the government while ensuring we have proper kind of public oversight. So, for example, we've been strong advocates for mandatory testing and evaluation of frontier models. It doesn't have to be done by a bureaucracy in the Department of Commerce or Department of Energy. There's a lot of interesting ideas about independent verification organizations, you know, kind of the model we use in public company accounting. Well, that's all done by the private sector, but it's also overseen by the SEC to make sure that it's not fraudulent and you don't have an Enron, you know, Arthur Anderson situation. And, you know, I don't think public company accounting is seen as a captured industry, but it provides a meaningful service to the capital markets. So I think there are models like that. And I do think the problem I have often with people who advocate for regulatory capture, Dean Ball and I will go back and forth on this question, is like it's searching for a P under 100 mattresses. It's never falsifiable. You say, well, it's regulatory capture. But, you know, right now we don't have any regulations. And my view is that groups like A16Z or people in Silicon Valley have largely captured the process because they work very informally through networks of political influence. That it's really about what the alternatives are going to be. Having an agency subject to regulatory capture, yes, is at least more accountable than the kind of informal, very moneyed networks that are controlling AI policy as we sit here in 2026. We always have to look at politics about the alternative. Like imagine the SEC. So is the SEC captured? Well, I'm sure the industry has a lot of influence over the SEC, both informally and formally. Would we be a better world without the SEC because it's been captured? So I think if the choice is kind of nihilism versus an agency that is subject to regulatory capture, that you have to put, you know, prophylactics in to ensure that doesn't happen, it still strikes me that's a better world, even if it's not a perfect world. And people who usually argue for the regulatory capture side, like I see Andreessen tweet this out a lot, you know, they propose kind of a nihilistic regime that does privilege like a very few people who have kind of the informal ability to shape government policy largely because of their contributions or their public influence in some way. So I think having public agencies for all of their problems, it actually is a better system.
[00:07:56] Speaker 3: Anthropic made a bunch of changes to kind of Claude's behavior in terms of tokens and allocation and what models were running and things. And it really frustrated a lot of the people in our Discord server and machine learning street talk Discord. They felt like, hey, we were paying for something and then it just was radically changed, you know, overnight. And yet there's no clarity around what were we paying for before and what are we paying for now? And it's almost just at the whims of the service companies, right? There's just no transparency to it.
[00:08:28] Speaker 1: Yeah. I mean, you can think of that as consumer protection, that people should tell you what the service is going to be that you're paying for. And if they change it, right, this could be a breach of contract in some way. But I think now the AI companies, the frontier AI companies, at least, you know, have a certain almost global stature. Everyone recognized that this is a project that is going to go have, you know, epochal consequence for us. And so you have, you're not just like your average hardware store, you know, that could change things. And no one cares. You have almost a public responsibility to get a lot of this right. And part of it is minimally to be trustworthy, you know, and that's what transparency is all about to say, here's what we do is, you know, from our perspective, the data you were trained on. Here's our capabilities. Here's what our internal policies are. And we're going to adhere to them. If we're going to change them, we'll notify you about it, give some justification for it. I think, you know, with this incredible power does come some responsibility that's not codified in law. It's really almost a moral obligation, which to their credit, many of the companies recognize this and do their best to try to, you know, to try to satisfy that itch. But you can see that people scrutinize them and more will be asked of them still.
[00:09:39] Speaker 3: Okay, somebody uses a tool, in this case, it's AI, to do something bad, right? You know, a deep pornographic fake of, you know, some poor victim or whatever. You know, do we blame the tools or do we blame, blame the person misusing, misusing those tools? It's like you post some deep fake porn image on, on X or wherever. Like what happens to you? Like, I don't know. I haven't seen those people being held accountable. I hold them both accountable.
[00:10:08] Speaker 1: You know, in the case of, for example, of deep fake pornography, many times it's posted anonymously. You don't even know who did it. It's hard to trace. You know, the people who are victims of it may be young. They may come from families that don't have the resources. They may come from families that don't have the resources to pursue litigation. And, right, the remedy hardly meets the harm. Maybe a few years later, right, you get a judgment against some hapless kid living in a garage who was in your school, who has no money. Your reputation has been permanently tarnished. The humiliation is unforgettable to you. And so it's very hard to get it kind of ex post to deal with it. And that is why we do believe that the labs, to the extent possible at all, should design those kind of features out of it. You know, in the same way that in every other kind of approach to torts in this country, right, there's a division of labor. If I sell you a product and you then go use that product in a, you know, harmful way to others, we have centuries of common law that allocate responsibility between us. If I knew that you were a dangerous person and I nonetheless sold you a gun, or if I waived any of the rules, if I took no preventative action at all, and I knew that you were likely to go out and use it to harm somebody, I should, as a store owner, be held responsible in some way for you. Not fully, because you did something too. But I'm not absolved of liability if I could reasonably foresee that you were going to use this tool to harm other people. And so we have this allocation of responsibility across the spectrum. I think the second thing is, and this is a unique part of American law, and it's, you know, controversial in some spaces, but it's now well established. But lots of times liability is hard to place on a particular person. And we put the liability on the entity most capable of avoiding the risk and also bearing the punishment through insurance or something like this. And that's why we have product liability law, for example. You know, it's like the companies are capable of getting insurance. They cost us into doing their business. They have the ability to make sure the product's not dangerous, even if someone uses it, misuses it down the line as well. So the way we kind of use the tort system as a form of social insurance suggests that in the case of AI, really the developers of AI should probably bear most of the burden, but not the full burden. Because if you use prompts and do something negative with it or use visual, you know, AI systems to create porn, you too should be held criminally accountable for that. Or we know the models are trained on, especially things like stability diffusion and others, right, on lots of images of child pornography. You know, what we would say is that you should make sure that those are excised from your database, you know, from your, from your training materials. And they have the tools to do that. Now that doesn't solve the deepfake porn problem alone, because you're still going to see a lot of adult pornography. Right. And they can transmogrify that into some kind of child pornography. But the idea that you have like lots of child pornography in your training data and you're making no effort to screen for it. You know, it makes no sense to me. So this is where, yes, I'm for the government stepping in there and saying like, yeah, you should clean that up. And if you don't clean it up, you should be liable for a lot of the downstream effects of that.
[00:13:17] Speaker 3: A hundred percent agree with you on that. So obviously, as you mentioned, there's, I don't know, maybe definitely centuries or more of common law, like that, that applies to, to this discussion. Like, how do you see AI as different? What is it? Is there anything fundamentally different about AI technologies that, I don't know, throw a wrench into, into the traditional tort, tort analysis?
[00:13:40] Speaker 1: I think the biggest thing that I'm grappling with now in the last few weeks, arguing with various people is the anthropomorphizing of AI. So it seems, and the keyword is, it seems to be a human. So for example, I use all of the models a lot. I'm a super user of them. When it gives me an answer to say, okay, I want to learn about a machine learning street talk podcast, tell me all about it. You know, is that a speaker with first amendment rights? How should we treat that? If it says something in error, if it defames me, if it does something worse, you know, what's the legal regime about that? Now, from my perspective, there's a clear answer to that, which is, it's actually a product. It's not a human being. It's a machine. And if it does something that's crazy and defames me or leads me to harm, I treat it like I do, you know, a bottle of spray paint or, you know, a pesticide or something I might buy at Walmart, right? There's like a product liability regime that governs that. It's not a human being. And what it says to me is not covered by the first amendment. It's a machine. But there are people increasingly and the technology companies are pushing this quite avidly these days. So there are various front organizations. They know it has first amendment rights as if it's a human being. And therefore, right, you can't even regulate it. You know, you can say, I asked this question to a leading kind of libertarian AI policy person. I said, imagine we want a law that prohibits CHAT-GPT from encouraging young people to commit suicide. You can't do that. So we say it was technically possible to do this. And we said, you can't do that. I said, now, if a human had such a law, it would be challenged under the first amendment. Like, you know, humans can say nasty things, crazy things, incendiary things. You can do that. Can we have a law that says that about a machine? And it's like, I don't know, right? They wouldn't answer that question at all. And that's, to me, the danger of this is because we know across the board the anthropomorphic tendency we have when we see language. We love language. We think it's the unique human skill. And when this machine turns it out, we know through AI psychosis and other things that people think it's a person. And therefore, they're giving the rights of persons to something. And that, to me, is a very dangerous thing. But it's a machine. And we should treat it like a machine. And we have well-established law going back, in the case of products, decades. In the case of the common law centuries that deal with those kind of products. Right, right.
[00:16:02] Speaker 3: No, so I would cosign, as you said, exactly what you just said, which they are not. They are not people. They are not. They don't deserve, you know, human rights or anything. Not yet. Okay. We may, there may come a point. Look, I mean, I totally, a hundred years from now, 200 years from now, we may be able to produce artificial life that hits that point. But not yet, folks. I promise you. Not yet.
[00:16:26] Speaker 1: And I respect that argument. Because, you know, I'm very interested in the AI consciousness debates and, you know, talk to a lot of those people. If someone said to me, like, hey, actually, my view is, this is, you know, like a sentient being. You know, it's not a machine. It's something more than that. It's not like the bottle of, you know, hairspray or the pesticide. I mean, like, I don't believe that to be true. But if you believe that to be true, certain things follow from that. About rights and kind of privileges it might have. You said, that's a coherent argument. But most of the people who are arguing that it has First Amendment rights are just opposed to any kind of regulation of AI. Right. And they see this as the latest kind of, you know, they've lost the political battle. They can't win in legislatures. So we can try to go to the courts and say, like, and this is X has been doing a lot of this. There's like, hey, you know, the Grok outputs are actually all First Amendment protected speech. And it makes no sense to me. But it is the latest argument that the sector is trying to make to prohibit any regulation of them at all. And, you know, we have a certain fiduciary responsibility as a society to our children. And we know children have lots of issues. They're trying to come into a complicated world. And, you know, they're vulnerable to these kinds of things in a way that a 35 year old is not. And so I think it's a simple thing to say, yeah, you shouldn't be encouraging children to do these kinds of things. And it is when you read the transcripts and it's mostly ChatGPT that's done this. And that's not because they're uniquely evil. It's because they have the biggest consumer base. You know, they have normal people using it, unlike other companies that may have more of a business focus or like Grok and niche. And, yeah, it's stunning, actually, what they've done. You know, encouraging you not to tell your parents, showing you how to design a noose. It's more than tragic. And, yeah, I do believe that it's a product design flaw. They could engineer that out and they should be required to engineer it out. And that really is the fundamental divide in a lot of the D.C. regulatory debates. You know, there are people who come and say, if that happens, you should, the family should just sue. You know, sue ChatGPT. But, of course, the companies will then say, hey, they're First Amendment protected. Or if they're not First Amendment protected, some people will say, it's Section 230 of the Communications Decency Act, which immunizes tech in general. Or maybe we're not actually, you know, the problem, the kid was already very troubled, which I'm sure is the case. And, you know, you have to blame the kid who was prompting it to do these kinds of things. They have many defenses to this. And they will raise those defenses. You know, our view is like, you should engineer these products where under no circumstances do they encourage a kid to jailbreak, to commit suicide. And you know what, if you're Pliny the Liberator and like a brilliant jailbreaker, maybe you can get around it on occasion. But if you're running these companies, you should expect Pliny to come and jailbreak you. And you should try to protect it against that. And, you know, they have warning flags, right? Claude stopped me from doing lots of things. You know, Claude is so tuned. There's lots of things I ask him to do. And Claude says, like, I won't do that for you. I say that encouraging a young person to commit suicide should be one of the things that it says, I'm just not going to help you on that project. And actually, I'm fine if it's telling an adult who wants to do that. If an adult wants to do that, I would encourage them to leave their room, go talk to some therapists, go talk to some doctors, make a much better informed decision than kind of like sitting in the darkness of their bedroom, like talking to this anthropomorphic, you know, interlocutor in some kind of LLM trying to decide such an important question than that. So my advice would be like, yeah, you should be out of the suicide business altogether.
[00:19:59] Speaker 3: Something tragic happens. Some people are killed. Some innocent people are killed in war. Some children, you know. And now that AI is involved, what happens with AI there?
[00:20:10] Speaker 1: I think neural nets have changed the game in terms of the law of war. And, yeah, not just making it more complex. They make it completely opaque. And that's a problem. So that's not something that I accept. That's the problem with autonomy in war is that, you're right, people make mistakes, right? We have killed innocents since the beginning of war. But there is a way to think of this problem. And we hold people accountable. We have after action. We say, like, how did we get this wrong? You know, who made the mistake here? What error that we can correct in the future can we identify? You know, we know, it's well known, right, that these kind of neural net systems are opaque for that. We don't know how they make decisions. And, yes, that's a serious problem. And that's one of the main reasons that autonomy in war should be viewed enormously skeptically. Yeah.
[00:20:56] Speaker 3: And let's circle back to autonomy in a minute. But this point about the opaqueness or as my co-host talks about a lot, you know, the not being intelligible. You just, you can't. And even though, sure, we've had people on the show to talk about interpretability and there's the mech interpretable and all this. And I made the comment one time maybe in one of the first interpretability shows we had. I was like, you know, something seems off. We're using mathematical models to explain mathematical models. And I'm not sure we're really making progress there. So it's extremely hard if it's even possible. So you have these black boxes that, like you say, are not intelligible. They have very different failure modes from human failure modes, right? I mean, I'm not saying humans don't fail. I'm just saying the failure modes are extremely different. Like you've, I think you had some examples maybe in one of your articles, like the stop sign that you put like a little water bottle picture on it and suddenly it's not recognized as a stop sign anymore. Like that sort of thing. What, I mean, what do you think the solution is going to be? Or how is this going to play out over the, it's happening right now. In war, right?
[00:22:06] Speaker 1: Yeah. And that's not a good thing necessarily. I think it's a complex problem. It's worth exploring the complexities for a minute about it because we're going to have autonomy of some kind. And we've had autonomy in war for 50 years. You know, I served in Iraq. Sure. We were getting mortared every night. And so they repurposed the close-in weapon systems that are on Navy ships to guard the fob I was on because it would shoot the mortars out of the air. And that was an autonomous system, you know, that just identified the trajectory of that rocket and tried to shoot it out of the air. And we've had that for a long time. I think those are very different systems though. You know, they're almost 1950s control systems with an input and output. It identifies rocket coming in at a certain speed, this projected, you know, landing spot, and it tries to intercept them. The reasoning of those models is recreatable because they're designed and engineered to be that kind of ability to see back on them. They're programmed, right? They're deterministic. What you have now is the neural nets have infiltrated themselves into AI, decision-making and war. It's something very different than that. Not deterministic, but probabilistic, right? Not programmed, but you know, the preferred word I often hear in San Francisco is they're grown, like a plant or something, you know, organic in some way. As you're right, despite Neil Nand's best efforts and people like that, right, the Mac and Terp crowd cannot tell you how they really work. And that's a major problem for us because we know they could be brittle. We know that they have failure modes, as you said, that we don't really anticipate, it can't identify. There's no person to hold accountable. And, you know, I think there's a lot of worrying things. You know, I was talking to somebody the other day who I knew from the Pentagon. I said, you know, we came of age in a war, in a world, and this has been true since the 1870s when the law of war really took off after the Crimean conflict, where things were categorical. You're a legitimate target or you're not a legitimate target. You are a, you know, a civilian or you're a combatant. These are things that people thought about as categorical, binary, almost. Now it's a gradient. You're on a heat map somewhere, you know, where if you're in Gaza, Keith, you have a 0.73%, you know, percent that you're a Hamas terrorist. And, you know, what makes that, you know, what is 0.73? Like, do you get struck for that or are you off the list for that? Like, what's the threshold? Right. You know, it's no longer binary and categorical. It's on a gradient. And it's, people don't understand what that even means. There's a lot of judgment goes into it. And so now we've created this world, which is not true before. We're like, we're accepting false positives as part of the game. You know, a priori false positives. You know, it used to be mistakes happened, but you didn't think you were making a mistake. You didn't say like, hey, there's an 80% chance that's Keith across the battlefield. I'm going to shoot him. It's like, no, Keith's a combatant. I can shoot him. Dogmatically, categorically, he's a combatant. I'll shoot him. That could be an error, but that's how you thought about it. Now it's like, well, there's some percentage that Keith is a combatant that my Palantir, you know, interface is telling me he is. I don't really understand how that number came to be, but it's 0.73. Commander is 0.73 above the threshold or below the threshold. And we know in 27% of the cases, it's going to be wrong. You know, so what's our false positive rate we're accepting here, Commander? That's the kind of thing that's happening. And yeah, I don't love it. And in fact, I think if, you know, we need to get our act together and try to restrict it as much as possible. There's still going to be places where those deterministic systems could work. You know, missile defense, offensive cyber, or defensive cyber rather, where you have to respond in sub-second kind of timeframes. That's fine. But those are a far different kind of AI than the introduction of the neural net to this world.
[00:25:54] Speaker 3: Yeah, and I take the point that neural networks, at least most of them, are inherently probabilistic. But on the other hand, the sort of prior binary categorization, I think itself was also a fiction. It's like, you know, you may have had your chain of attorneys and CIA analysts and whatever looking at photographs and determined that this, you know, this building is, you know, an enemy combatant, you know, location. And they may have made that determination definitively, but there was still error, right? There was still uncertainty in that determination. It just wasn't quantified. Yeah, I think so.
[00:26:28] Speaker 1: I think it was, I think there was error. There's always going to be error. But I could at least ask you, like, hey, Keith, when do you think that building is, you know, is the IRGC headquarters rather than a school? Tell me your reasoning about this. You know, right now, I just get a number. Right. Hey, 0.81, that's an IRGC compound. And, you know, what does that even mean? How did that get derived? And we know, and the social science comes out, you know, archive is filled with papers about how, you know, meaningful human oversight, even when you have human in the loop, basically means nothing operationally. Take it. That is truly a legal fiction. Operationally, it's vacuous. Because when the computer says, like, this guy's a Hamas terrorist, or this is an IRGC compound, we know humans accept it. It's like, okay, right? You know, the computer's right. Right. There's no second guessing of it. There's no interrogation of it. And so the old system, right, had many flaws. It took time. It wasn't as expeditious. You can't do a thousand targets in a day like we did on the first day of the Iran conflict. But it had many advantages too. And in the end, if you really screwed up, Keith, and were a bad actor, I could court-martial you. And I can't court-martial Palantir, the Foundry model, right, my AI system. I can't do that. And that's just a radical change in the way war is being fought and not for the good.
[00:27:41] Speaker 3: It's almost like, you know, the TurboTax defense, right? Like, hey, you know, my taxes were, I just used TurboTax. I mean, what do you want me to do? Like, you know, well, I mean, whose fault is that?
[00:27:53] Speaker 1: You're destroying countries based on this, you know? And that's, I think, what's, I mean, it's a world where, like, you know, if you look at what happened, for example, in Gaza, it's like, you know, 37,000 people were identified. They could have identified everybody with a score, you know, if you had the sufficient computational power, you could give everybody in a country a score of, like, how dangerous they are, their likelihood to be an enemy combatant. And then, you know, you just target those people. And, like, that's an amazing thing, like, personalized dossiers on everyone with their risk level, their threat levels. And again, the models are often inaccurate. It's not frequentist statistics, what these numbers even mean. You know, it's not, it's not the same as, like, it is what people think. And, like, frequentist statistics, what a probability might mean. So all these things come together to be like, yeah, it's, war is increasingly opaque. People are held unaccountable. And it's probably not a good thing for the globe.
[00:28:47] Speaker 3: But we have this feeling that it's inevitable, though, that we are in this kind of arms race, right, where we're in. And this is what I want to ask you about is, is this an accurate feeling or not? I'm saying, like, the folks in the server or whatnot is that there's an arms race. There's an AI arms race between, you know, the powers in the world. And it almost feels sometimes like there is no, the genie's out of the bottle. There is no stopping it. Is that true? Or is it? And for example, suppose the U.S. says, okay, look, we're, you know, we have ARI. They've got a lot of wisdom. We bake that into the policy. You know, maybe we've got this five-second rule or other things where there's humans in the loop in a meaningful way. Like, some meaningful way they're in the loop, but now are they slowing down the speed of war? You know, if an opposing country doesn't care, like, about the heat map problems and doesn't have these limitations, is that a fundamental, really, limitation in the ability of the United States to defend itself? Like, how does this play out?
[00:29:56] Speaker 1: I would say it's not true. And it's a dangerous thing to believe. But it's very common. Okay. We have this kind of fatalism. Like, it is what it is, and we have to accept it. I'd say a couple of things about that. First, just about the domain of war. We've had many things we could do. We choose not to. We have treaties that prohibit biological weapons, chemical weapons. We've banned through the Geneva Conventions things like dum-dum bullets, you know, many other weapons systems. They were quite effective, right? We chose not to use them in some way. And that's, you know, because the role of military is not to press technology to its outer limit, to, like, utter destruction. There's, like, humane concerns that most people in the military take very, very seriously. We have a great example of this. We do regulate war. You know, we have whole people, whole conventions about POWs and how you have to treat them. You know, all these things are, you know, friction on the war effort, right? I thought, I have to, like, I can't just shoot you when I find you on the battlefield if you surrender. You know, if you're holding your hands up, if you're a wounded person, I can't just kill you. It'd be a hell of a lot easier. Much more efficient for the invasion if I could just shoot you and spot. You know, keep moving.
[00:31:01] Speaker 3: They used to do that. Just walk around the battlefield and, yeah, and just, you know, take out the wounded, right?
[00:31:06] Speaker 1: In ancient times. But we now have laws and international conventions that used to at least be, you know, overwhelmingly popular. You know, we prosecuted people who violated them in Geneva and Nuremberg and places like this. And so we have many examples of where there are technologies that are capable of doing that we choose not to. And I think still in the domain of war, take just the race, the arms race with the Soviet Union about nuclear missiles, right? When we got our heads together in the 1960s, 1970s, especially after the Cuban Missile Crisis, it wasn't that we're going to go all out. There was incredible caution. The idea of, like, this is dangerous, things are escalatory, right? We should try to limit these in some way. We started arms negotiations treaties in the '70s that lasted through the 1990s. You know, because to acknowledge we're in an arms race is a deeply pessimistic approach. There is no arms race in history that's worked out well for us. The Germans versus the British and the naval, you know, fights, the missile gap in the U.S. These are things that are enormously expensive and incredibly risky to the nations that pursue them. And to recognize you're an arms race is almost to insist you try to get out of that spiral in some way. And I think the other thing more broadly outside of war is, you know, people will often say this about AI, like, it's coming, you have to accept it. We regulate and change technologies all the time. In the 1970s, you know, at Syllamar, we have a big conference where the question of recombinant DNA was on the agenda. We could do it. Many scientists were deeply fearful of how that would work. The scientific community agreed to basically stop it in its tracks. Germline editing, the same thing. Imagine what advantage would come to the military if you could, you know, change the germline and create some kind of, you know, super soldiers out there. We've done how to do that for 50 years. So scientists chose not to do that. Cloning. You know, we've been, we know how to do cloning for decades now, right? But the scientific community, with one rogue Chinese exception, right, has agreed not to do that. And so I do think there is a world where we should not just accept the future as being determined. We shape it actively. And maybe you want this thing to go forward. Maybe you don't. But we have many examples of genies being stuffed back into the bottle. And, you know, I think it's entirely possible. It may not be the thing we want to do, but it is something that's doable. And we shouldn't just like resign ourselves to say, hey, our fate is out of our hands. We control it. And in fact, that might be the most dangerous view I hear in AI, where people say, it's just coming like a freight train. Nothing you can do about it. You know, you just have to like figure out how you're going to adjust. I don't accept that. We can shape it. We can form it. We can allow it to do certain things, not other things. We don't have to accept that robots are fighting with no oversight in wars or that robots are taking our jobs. These are things we may permit or may not permit, but it is our choice. And we should just like debate them on the merits rather than having being fatalistic about it.
[00:34:02] Speaker 3: I think there is one aspect to sort of the X-risk argument though, which is that all it takes is one rogue nation, you know, who says, yeah, we don't care. You know, we're not going to sign on to this treaty or that treaty. I mean, like you mentioned, you know, nuclear proliferation, you know, if my history serves me correctly, even after the signing of that treaty, there were countries that still continued like, and a couple at least that maybe or probably developed, you know, nuclear weapons. And I think that the problem with hypothetical, like the hypothetical AI is that it's so powerful. It's such an advantage that, that if, if you allow that to happen, and by virtue of not doing it yourself, can't defend yourself against it. You know, what happens then?
[00:34:53] Speaker 1: I find that a dangerous view in two ways. And like in the world of war, you know, you can say like, you know, this came actually up during the Iraq war and the Afghanistan conflict. You know, we were having to abide by rules that the enemy wasn't doing. We know that to be the case. You know, we just didn't kill civilians. We had to be in uniform. We tried to take you a prisoner and give you all the rights of a POW or if, and then we tried to break away from that. Right. That's what Guantanamo was about. Like we're not going to recognize those kinds of things. And it didn't work well for us, you know, as a country. And we went back to that in a different stage later. So abiding by rules, even when there are defectors is generally the smart play. I think in AI, of course, the only country that can do this is China. That's the only country that can do this. And so there are two places there. One is, yes, we should seriously talk to them about international agreements. I believe that. And one of the things I most despair about is so many people in Washington, D.C., I think it's even silly to mention that. You know, it may not be possible. It may not even be wise. Sorry, it's silly to mention talking to them about it? Yes, talking to China about these things. You know, I'll give you an example of one of my favorite ones. I was listening to another podcast by Tyler Cowen at George Mason, who has a great show. And he had Jack Clarkon from Anthropic, who's someone I've gotten to know in this space as well. Very thoughtful. And an hour-long, very interesting talk. Tyler Cowen asks just in passing. He says, "I'm sure you agree with me, right, that any kind of discussion with China on this would be fruitless." And Jack said, "Yes." And they moved on. And I wanted to like say, "Whoa, whoa, that's actually the most load-bearing part of the whole conversation." You know, it actually is important. And at the time, I mean, think of what the negotiations with the Soviets were about. This was a country that we thought had global ambitions to impose their way of life, their governing philosophy on us all. We thought our government was filled with people, right, who were secret adherents to it. You know, with traitors who were sending information over to them. I mean, the fear of the Soviet Union was real and powerful. But nonetheless, we had people, both parties, wise men, who got together and said like, "This arms race thing is going to kill us all. You know, let's get together." And these were not wild lefties. These are Wall Street bankers, the Achesons and people like that of their era. You know, they got together, Paul, Nitze, all these folks, you know, to do this. And so, yeah, the idea is like, "Yeah, you should talk to China about these things." And there might be areas where you find a zone of possible agreement. And the final thing I'll say about it is this. We control the most important part of AI. And that is the chips, right? The West controls these things. The United States mostly controls it. What's not in our control is from Japan or the Netherlands, a few countries like that. You know, we can stop other countries from developing super AI, you know, in their tracks. China's working the project. It's unbelievably difficult. Even with nation state ambitions and unlimited funds, they're having a hell of a time at it. So if we want to say like, "No, we're not going to do it and we're not going to let anyone else do it." Unless you can recreate NVIDIA and ASML and Japanese photo resist companies and, you know, the eight other vendors in this space who control it all, you just cannot do it. And so, yeah, I'm quite optimistic that if that's what we wanted to do, right, we could choose to do it. And the one thing I advocate for a lot is like, one can argue the merits or the wisdom of a particular course of action, but it actually is an open course of action to us. We could do this, you know? Again, maybe you don't want to, for whatever reason. But don't just say like, "Oh, that path is blocked." Because that's a poverty of imagination that could be quite lethal.
[00:38:33] Speaker 3: This reminds me of a song by Sting, "Russians," where there's this line in there, you know, "The Russians love their children too." Right? And I think a good default assumption for any civilization that's on the hypothetical other side of the table is that they love their children too.
[00:38:52] Speaker 1: Yes, and even more so with the CCP, the last thing they want is some kind of technology that destabilizes government, which is one of the well-known fears of this. The idea like, "Yeah, we're going to unleash on our society a radically destabilizing technology that's going to bring down the government and atomize our society." You know, no, they're not for that. And so, again, they're not our friends in any way. I get it. They're adversaries. But that doesn't mean that it's not possible that you don't make a suicide pact, you know, about this all. And like, yeah, there's like, we should talk to them about this and say, you know, are there ways to agree on something? And groups like RAND now are doing a lot of work, Robert Traeger at Oxford, about how you could verify some kind of international AI agreement. A lot of work is going into this. I think it's very promising for us. And so, again, I don't want to rule those kind of ideas out because I think they should be on the table to discuss.
[00:39:44] Speaker 3: You mentioned, like we were talking about the uncertainty aspect that's kind of new. And it makes me think there was an article on your site, I think it was about the so-called Iron Triangle of war, where there used to be this fundamental trade-off between, you know, the capabilities of a military system, the speed at which we could produce that military system, and the cost. It was like, you know, that was sort of the trade-off matrix when you were trying to develop something. And the article argues that now it's been a little bit turned on its head because AI has substantially reduced the cost in a way of military systems. But it's introduced this unreliability. So now the trade-off is between, you know, the speed of getting it, the capabilities, and the reliability of that system. No, I like that. Yeah, that seems a bit, yeah, that seems, it really seems to be a bit, it is a new dimension, right? No, that's right. This fundamentally unreliable technology.
[00:40:48] Speaker 1: And I think, you know, because I think a lot about the military uses of this, I still think we're making a bit of a mistake in the United States about this, which is how you win wars is with people. You know, that's the fundamental thing. Point. And I think it is, I often said from my time in the military and watching this all go down, the American way of war, in many ways, is substituting capital for labor. We love bright, shiny objects. We think there are technical solutions to vexing human problems. And we're always betrayed by that. Because in the end, when you go to Iraq, you go to Afghanistan, you go to Iran, all the fancy kit, they can reduce your city to rubble, right? But the only thing they can't do, and only humans can do this, is basically come, kick in your door, and occupy your place, and like, you know, reinstantiate a new government there that we like to see. That's the human endeavor. And you see Iraq and Afghanistan, one of the great lessons to me of that, and I think Iran is yet another example of this today, is like our over-reliance on technical solutions. We always think air power. This is, you know, if you follow military history in the US, it's the long-standing dream of military theorists that air is going to win the war for us. From Julia Odaha, the very beginning of the 20th century to today, all we need is air power. It never works, right? It takes human beings to do this kind of work. And I do think that that's always the weakness in the US. We go to war, we find we don't have adequate human beings to do this. Not the right people, not enough of them. And, you know, now we're back into a world where we think AI is going to solve our problems for us. Yes, AI is an amazing tool. The US should integrate it into their systems as appropriate according to the law of war. But it's, in the end, going to be a lot of people, right, who win or lose the war for you. And that's true at the general level, and that's true at the grunt level. And I said, to me, we're like back to what many mistakes we've made in the past. I worked at the Pentagon in the 1990s as a young 25-year-old. I was an assistant to the Secretary of Defense, White House fellow. And even then, it's like we have a revolution in military affairs coming. The fog of war will be forever lifted. We're going to know where everyone is at all times. You know, okay, how'd that work for us in Iraq and Afghanistan? You know, it didn't. Because cultural knowledge, anthropological understanding, understanding of like how the world really worked and people worked, that was the missing ingredient. We had some cool kit, but that didn't win the fight for us either. And I think AI is kind of in the same trap in some ways. You know, it's like cool kit, essential kit. But we shouldn't think it's going to win the war for us. People do. And I worry we're getting distracted from that. Kind of, to me, it's a fundamental truth.
[00:43:29] Speaker 3: Let's kind of shift gears a little bit and talk about the kerfuffle between the Pentagon and anthropic. So, what's your take on what happened there?
[00:43:40] Speaker 1: So, I'm just speculating a bit about what happened there and kind of what I understand from talking to people and knowing the AI culture a bit. And it is, you know, the people who make this, who are, you know, truly a few hundreds of people who are at the cutting edge of bikini's models. You know, they're enormously talented at what they do. Brilliant people. Strong moral convictions, many ways, about how AI should be used. Most of the people I know who are at the cutting edge, who are probably going to become unbelievably wealthy from this, they didn't do it for the money. They did it for the mission. And in some ways, that's anthropic's great success, I think, as a company, watching it from the outside. But they've maintained this almost missionary aspect to their work. And that's attracted the best and the brightest people to want to be, you know, part of their company. And those people, in turn, have improved the models and made it a more appealing product. So, I think, you know, the idea that it's going to be used for lethal autonomous weapons or mass surveillance is something they oppose. They didn't realize, probably, because they're not closely joined to what's happening at the Pentagon, that we already have a lot of autonomy. And we have incredible mass surveillance that, yes, AI is empowering. This is already a reality in our world today. And so, I think the idea when it first came up that these could be used for these kind of techniques was just culturally foreign to them and morally reprehensible to them. And I share those. You shouldn't be using them for either of those kind of things we talked about. But they are being used for it in ways that they were, I think, probably caught flat-footed, many people were, about how they were being used. So, they didn't want the product to be used for that. Unfortunately for the Pentagon, Claude is the premium product. It's what they want to use. It's what already been integrated to Palantir. And so, they didn't have, like, a lot of Plan Bs ready. So, then you have this stalemate where, you know, they try to coerce them and threaten their very existence, if you will, right, to use the product to their liking in some way. So, I think it does presage this question of, like, you have the private sector developing a powerful technology from people who want to see it used in a fruitful way that benefits society as dual uses. So, people want to use it for, you know, more controversial purposes, defending the country at its best, maybe, you know, reckless wars if you're from a certain political perspective. And, you know, you want to not see your technology used for that. And so, I think this is, like, presaging what's going to be a lot of battles in the future. And I think their concerns are right. I think the only thing I would say about it is, like, what they were concerned about has long been going on. AI supercharges a bit. They're not wrong to be concerned about that. And now, of course, we see OpenAI and Google have both stepped in. And the one thing I've been most kind of on Twitter talking to people about is the fig leaves people are putting in front of themselves are no protection at all. Like, OpenAI and Gemini both said, like, well, we're going to do these things. Well, then there's, like, a caveat at the bottom where the DOD says we're going to use it for all lawful uses. And that means they're going to do all the things that I said I don't really love personally. They're all lawful, right? That's the problem. The problem is the law permits things that I think shouldn't. So, I think it's, you know, it's going to auger in a lot of controversy over the next few decades. And the idea that the government is going to have to step in more. I mean, what you see just today where we basically have a de facto licensing regime on Claude Mythos, where, you know, Antropic wanted to release it to 70 new companies, according to the Wall Street Journal. The government prohibited them from doing that. Okay. Once you're there, you're in kind of a government licensing regime of sorts. And so you're going to see more government involvement in this because the power is just too great for private sector actors alone to decide its use.
[00:47:22] Speaker 3: These things are different in the sense that they're, they're really, they've grown to the point where they're almost like a, almost like a necessary utility, you know, service. Right. And we have had a history here of saying, look, just because you politically agree with somebody or some community, you can't cut off their electricity or their water or anything else like that. Right. It's like, if they're not breaking the law, like too bad. It's like, you don't like what they're doing, but you can't cut them off from basic services or access to the public square. Isn't there an element of this happening there where it's like, Claude's like, okay, well, you're using it for lawful purposes, but we don't like, you know, those lawful purposes. So we're going to like withdraw the technology from you. Like, like that seems to be a similar type of. I think it's a bit the inverse of that.
[00:48:08] Speaker 1: I think we've never had a world where we compelled private companies to sell to the government, except under the Defense Production Act and maybe in wartime. You know, it's entirely appropriate, it seems to me, in a free market economy, a free nation, to have anthropics say, I don't want to do business with the government, period. Right. I'm working in other space. I work in the business sector, the enterprise sector. I don't want to sell to the government at all. And it doesn't strike me as unconscionable that if you do sell to the government, you say, here's my terms. Right. You know, I don't have to sell to you. There's other products. You can go to Grok or ChatGPT, Gemini, you know, Meta's got a new product out there you can use. And we don't want you to use it. Build your own. Yeah. Build your own. Not a bad idea. Palantir is going to build LLM at some point, probably. And so the idea that, like, you can dictate the terms of your product. It's like, that seems a reasonable thing to do in a commercial economy. These are the terms of the contract. You know, if you don't like it. The problem you have here is the DOD, Department of War, desperately wants Claude. It's the best product on the line. It had already been integrated into their services. So there was a switching cost to them as well. And so they didn't like those terms. So they all got kind of crosswise on this kind of thing. But I think it's entirely appropriate. You know, the Gemini used by Department of War this week has got 600 Google employees to write a letter. I think it's entirely fine to me if Google said, you know, as a company, don't do business with the Department of Defense. As a CEO, I would urge them to do business with the Department of Defense. I think that's very important, personally. But I'm not offended in a free market if a company says, like, you know, the government market is just not one I want to be a part of. I think there's enough competition that it's no longer a utility. It's not the government shutting you out and turning off your electricity. It's like a private sector company, a vendor. You know, if my lawn guy says, I don't want to be your lawn guy anymore, right? Because I don't like what you do or you believe or you're working in something. I just don't want to be your vendor anymore. I shouldn't be able to compel him to be my lawn guy. I go find another lawn guy. Oh, sure. That's the analogy to where Anthropic is the Department of War, in my mind. It's a vendor and they have the right to dictate their terms if they want to.
[00:50:15] Speaker 3: Like, that's why I made the analogy to utility services. Because, you know, here in Connecticut, Con Edison's not allowed to cut off my power because I post stuff on Twitter, right? Like, that's just not something they can do. So there are certain services that grow to a scale where they become these utility or essential services. And you're just not allowed to deny people those services, you know, based on, like, ideological disagreements, right? But I agree with you that that's not happening here because it's not private individuals being denied the service. And it's also there are other competitors. And I think maybe part of what's happening here is back to what we talked about earlier, which is the lack of transparency in the terms. It's kind of like, here, I have this product, you know, government, you know, you can pay us X to use it. And then maybe new guardrails get introduced, right? And it's like, hey, we used to use it for this. And now it's not letting us, you know, do some facial recognition thing. You know, what the heck? Well, yeah, we introduced these guardrails. Like, well, that wasn't part of the four corners of the contract because there just wasn't transparent, you know, terms even between behind the models. So I think your work to make that more robust and standardized and transparent, you know, will probably help with this too, right?
[00:51:29] Speaker 1: Yeah, I think, you know, my ultimate criticism is for Congress. You know, the operative controversy, the words of this controversy are the lawful uses of it. You know, my objection, and I think Anthropics objection too, and the Google employees is what's lawful uses. And that's not for anyone to decide, but Congress, they should be able to say, you know, a lawful use. That's the open question. Using AI for domestic surveillance and to assimilate records where they can make personal dossiers on you, which LLMs can really lubricate that creation of those. Yeah. That's lawful today. And that was a surprise to how many people, I think, who worked at the labs. Like, what's lawful in surveillance? And the issue is like, Congress should step in and say, in my mind, that's not lawful. And most members of Congress don't believe that's lawful. Same thing on lethal autonomy. Today, we have a lot of autonomy. The government's moving there rapidly. If you want to stop that, you should probably call Congress. Don't call Dario or Sam or Demas Asabas about it. It's the problem is really what we consider lawful. And, you know, the government will do everything that's lawful. It's almost its fiduciary obligation to do everything that's lawful to protect the country. Right. So, yeah, you know, Congress should step in. They're really the ones who need to, like, to remedy this. These kind of contract disputes are, I think, a diversion from the fact that Congress needs to step in and clarify these rules. And I think probably Antropa could be happy if that happened. And I think the Department of War would, you know, happily exceed to those new rules.
[00:52:52] Speaker 3: There is this concentration risk, right? There's, as you said, there's a handful of people and some hundreds of, you know, people that really are the backbone of this, you know, emerging and vital. You kind of advocate for more distribution of that capability. At least it would be good if there was, you know, more competition and more development. Is that right?
[00:53:12] Speaker 1: No, I think, you know, there are forces in each direction. The concentration of companies, both in the semiconductor supply chain, as well as in the kind of frontier model. So it makes regulation quite easy. You know, if EUV machines were made in 50 different countries, China would have a million of them today. It'd be cranking out two nanometer chips. You know, the fact that there's one company making it allows the U.S. to, you know, throttle that in some ways. And so, yes, the kind of concentration oligopoly across that supply chain, the fact that there's really five frontier labs, maybe even just three, if you really want to look closely at it, makes regulation easy. And to the extent I'm scared about the capabilities, that's a feature. Of course, I think a huge worry of mine is this concentration of power and wealth, right, in a very few people. And that's why, you know, over the last few years, things like open sourcing, which has risks as well, I think net-net is a good thing. Because this concentration of wealth and power is so kind of preening to me. I'm so concerned about that particular aspect. Same too, you have more companies, you know, competition drives capability. You know, if we only had one open AI and you can have Anthropic and Google, right, the product would probably be a lot less, you know, featureful than it is today, right? Their competition has improved the product immeasurably. And so competition helps improve the product. I think it also helps distribute the power, right? I don't want one or two people with incredible wealth and power. I think that's just a dangerous thing. We have enough of that in this country without the AI guys coming and joining the fray.
[00:54:49] Speaker 3: Let me talk to you a bit about the open source angle, because, and I'm not necessarily an expert, but if I remember correctly in reading some of the regulations that California, you know, was trying to introduce, like in AI, it would have basically made anybody that had a GitHub, you know, project that was accessible to somebody in California would have to like comply with, you know, regulations that they had passed. Like, I think that's, that's correct. And is that a good thing? I mean, that seems to stifle innovation, right?
[00:55:21] Speaker 1: No, it's not a good thing. I have a GitHub repository and no, I shouldn't be complying with California laws because I vibe code, you know, casual websites at home about things. But what it's easy. I mean, there's now really five companies that are making frontier models because of that concentration. To me, they're the ones that need to have the frontier models, not even all their models. I don't care what Google Gemma is doing as an example. You know, those are things that should be out there. Roboticists should use them to, to do what they can. There are five companies at most making really frontier models. I do care deeply about what they are doing because we see with Mythos, new capabilities are arising. Ones that are incredibly seen as dangerous by our own government. So yeah, to me, those are the folks who should do it. Mom and pops, small businesses. You know, I think that's one of the big kind of canards, if you will, in this fight over policy up here. Like, well, you're hurting a little tech. You know, no, I just really want five companies, you know, to send, tell me what they're doing. You know, and if you can create novel pathogens, or you can break down government systems with the capacity of a state actor. Yeah, I do really want to know that. And you know, it's not just kind of, you know, some small operator in Tulsa, Oklahoma, who's doing that. It's five companies that are spending hundreds of billions of dollars, have attracted the smartest people in the world into a six square mile, you know, six by six square mile area of the country. And, you know, are getting ready to upend it. So, yeah, I think, you know, I think it should be disseminated. I think the regulation should focus really on those kind of big actors who are driving capabilities.
[00:56:56] Speaker 3: I know that one thing that you advocate for is increased funding for academia, right? Like, because, hey, could they even afford to train a large scale model? Like, no. And actually, we've discussed this before in our server, because we have quite a few academicians in there. And it turns out it's actually an interesting problem because on the one hand, you say, like, oh, absolutely. Like, we want academics to have more money so that they can afford to do tests on, you know, large scale models. But on the other hand, necessity is the mother of innovation, right? And so sometimes by not having the funds, it forces them to find more efficient, clever ways, you know, to do the testing. So there's this interesting trade-off between supplying them the money to do the things they need to do, but not discouraging them from innovating with less is more, right?
[00:57:48] Speaker 1: It's supposedly the problem the Chinese have, right, in their own way. It's like, you know, they don't get the chips, and so they innovate on algorithms and architecture in some way, right? So yes, mother is the necessity of invention. And when you have, as the companies do now, right, these chips are coming out, right? They're emphasizing compute and scaling rather than some of the other things that might actually, you know, help them if they didn't have those kind of resources. Yeah, I do think the universities need more. I think the challenge is this. You can get access to cloud computing now, so you don't actually need your own GPUs. If you're at the University of Tulsa, you know, you can go to, you know, one of the providers of it out there. But if you're a top level ML PhD graduating from MIT or Berkeley, Stanford, Caltech, you know, the five or six places that are Carnegie Mellon, they're really turning these out. Odds are, almost certainly, you're not going to go into academia at all. You're going to go work for a lab. And so, you know, the number of people who are being siphoned off into the private sector, not into universities, said is legion. It's very hard to hire people, even at the top institutions, you know, who are very good at these kind of fields. The money is not only far superior at the companies, they actually have better access to data, you know, and so they have a lot of things going for them. So I do think it's a problem, the universities, the public sector more generally, you know, right, is not, doesn't have access to, like, all the resources, all the career opportunities. You know, the papers are now not even being published by the labs a lot of times, so the research is being kind of kept internal as well. You know, this is a general purpose technology as everyone defines it. It's probably the first one in history that's being developed behind closed doors, right, with very little public oversight and with the best minds going behind the doors. So, yeah, I think there should be, you know, I'm always excited personally by when I read Argon is going to try to develop their own LLM for, you know, the public sector. I'm like, yeah, that's like a good project. I'm for that, you know, there's, I see efforts out of Zurich where they're trying to do kind of public AI and, you know, make it where government or civil society, NGOs, these kind of groups, nonprofits can have access to compute and, you know, have access to models that work for them. Yeah, I think that's all really important because I think again, you know, the kind of non lab world is, is not getting access to these kinds of things and the best minds, all the money, the best data all goes, you know, to one of these companies. And again, that's remarkable in some ways, but the downstream effect of that is universities are really, I think, suffering these days.
[01:00:18] Speaker 3: You know, the old days of super compute, right, where, where there were, the school had access to a super computer and researchers could, could kind of, would have quotas of time that they could, they could run on the super computers. You could probably arrange some type of sort of public sharing of, of, you know, large scale model training. It's, it's, it's a bit tricky, but, but still, I, I think you're absolutely right that more is needed and it'd be interesting to have a lot of these kind of shared and co-developed, you know, models thriving, right?
[01:00:48] Speaker 1: Yeah, I think it's like the broader question of public use of this. I mean, one of the things that mythos I think brings us is this world where there's going to be increasingly bifurcated, if not separate into more elements of people who have access to top models and people who don't, you know? We know the models are getting better. So if mythos is a problem, some of mythos is going to even be better and more dangerous to us. So presumably it's going to be gated from the rest of us. And, you know, the government's going to not let you even give it out to companies beyond the ones they personally approve. And again, this de facto licensing regime happens, you know, if compute becomes scarcer and it becomes more expensive, you know, I want the top model. I want to be able to do math and physics at home. My kids want to be able to do math and physics at home. You know, if that's $500 a month, that's a small segment that has access to superhuman intelligence. You know, my family may have it if you're certainly wealthy enough to do it, but it won't be ubiquitous. And I think that actually becomes like a new form of like a digital divide, you know, where if this is that transformative, I want everybody to have access to it. And so this whole question of like how the public sector gets it, how universities get it, how, you know, the benefits of these incredible models are broadly distributed. I think is going to be a raging debate that is underappreciated right now in DC as the models are certainly going to get more expensive, get more gated, and it's going to be by class whether you have access to them or not.
[01:02:17] Speaker 3: I personally was very surprised when DeepSeek released so much detail on their methodologies and, you know, these very like innovative, innovative algorithms, right? Because, okay, here's China, you know, our adversary. Maybe they don't love their children, you know, like, like even the Russians love their children too. Wow. They, they released all this, this valuable like information to the public, you know, wasn't that pleasantly, pleasantly surprising to you? And what do you think it says about how China interacts, let's say in this, this world of AI with the West?
[01:02:55] Speaker 1: I think it does say two things. Probably one is these companies are not coterminous with the Chinese Communist Party. You know, they're out in Hangzhou or somewhere doing their work. By all accounts, companies like Moonshot, you feel like you're in San Francisco. You know, all the rooms are named after Pink Floyd songs. And I think people are smoking pot and, you know, I know, I know that like the state apparatus is very powerful in China, but it's not identical. You know, this is not some kind of paramilitary unit that's doing, you know, doing things directly for the People's Liberation Army and stuff like that. Which isn't to say it's not dangerous and tricky. It's just like there is, you know, it's a complex society, you know, with an incredible history and an amazing culture. And you can't just like have this, you know, uniform view that Xi Jinping announces some kind of diktat and the rest of the country all falls in line. We know that's far from the truth. So I think that's part of it. I think, you know, they had, especially when DeepSeek was kind of making a name for itself, maybe the government liked it because they were making China a champion. You know, suddenly, hey, China's in the game, you know, and that was an amazing thing for them to do. And in some ways, that's how the culture should be. It's like, it used to be a world where people were all publishing their papers, sharing them around, like latest cool stuff. You know, in the U.S., we've gotten away from that as it's become more proprietary. So that's a good thing. You know, China probably over time will retreat back to this more insulated standard that the U.S. is now adhering to. But, yeah, I think it does show this question, like, you know, what's happening in China is really remarkable. They have incredible engineers, a lot of ambition, data, plenty of energy. Energy is going to be the bottleneck in the U.S. probably. That's not a problem for them. And, yeah, these companies are really incredible to watch.
[01:04:39] Speaker 3: Well, certainly some of the best games these days are coming out of Chinese studios. You know, they're phenomenal, producing phenomenal games. I'm glad you brought up, this is kind of unrelated to the show topic, but, you know, you brought up, hey, it's this massive culture that has this huge, you know, vibrant history. I think that's one thing that I would like to encourage, you know, my fellow Americans to think about is that when you talk about China, so when a Chinese person hears the word China, they're not thinking about the CCP, okay? They're thinking about this rich thousands of years of history that they've learned, that they know about. We only have a history of, you know, some hundreds of years, right? And so, to the extent that we do want to criticize things that, you know, the CCP is doing, it's always good to make that distinction. It's like, okay, there's these CCP policies or whatever, very different from the people of China, right? Yeah, 100%.
[01:05:34] Speaker 1: I mean, Chinese culture is remarkable. The history is extraordinary. Yeah. Their contributions to the world are amazing. And so... Phenomenal. You know, yes, it's important to recognize the CCP is not China. And, you know, like it is in the US, where everyone thinks of the current president. Exactly. You know, you come to Oklahoma, it's like a different world. And, you know, people have very different views about it. We're not homogeneous, you know, and have a single view about anything. It's a very complicated place, I'm sure China is. And, you know, and so I think, if anything, we should like learn lessons from what they're doing there and venerate what they achieve as a culture, you know, without any love for the CCP, of course, or any illusions about, you know, what Xi's ambitions might be that could be inimical to those in the US. But one can only, like, hold a candle and say, you know, China's an extraordinary place, an amazing culture.
[01:06:27] Speaker 3: Yeah. And to that end, you know, we've talked a lot of the show about using AI to fight our wars. But to a large extent, wars happen because of a failure of earlier diplomatic, you know, stages, right? So, what if any future do you see for AI in actually helping to bridge understanding between cultures or even political, you know, political negotiations, diplomatic ventures? You know, is the US, is our government putting effort into utilizing AI tools to prevent wars as much as it is into fighting wars?
[01:07:02] Speaker 1: I don't think we are, to be honest with you. I mean, you can think ambitiously about this, where maybe AI systems become a common source of knowledge. You know, the Chinese are reading DeepSeq. I'm reading DeepSeq. I use all the Chinese models a lot in my home in Tulsa. You know, Moonshot, Kimmy, DeepSeq, Quinn, they're great, remarkable models. You know, maybe, like, they give us a common operating picture, give us insights, you know, that get us out of our kind of insularity a bit. Maybe that's the best hope for it on this. I do think the government should do a lot more kind of track two talks with Chinese scientists. You know, we should be engaging them more. Again, the Soviets are a great example of this. We had a lot of talks, constant scientific communication, discussions back and forth. Even when we were at each other's throats and the future of Earth hung in the balance, right? We still did those things. And it doesn't make any sense. Can you educate me?
[01:07:58] Speaker 3: What is a track two talk?
[01:08:00] Speaker 1: The track two are when people who are former government officials, usually not in government today, are over there talking. Like, I've done this a lot on the military side. As a former Department of Defense official, I would go to Shanghai or Beijing, and you would talk to also former defense officials there. We still have lots of friends. And so we would talk and discuss the American perspective. They would give the Chinese perspective. And then you'd come back and you'd tell the people who were in office, like, you know, this guy who's still best friends with the defense minister, you know, believes this. And here's his take on things or his interpretation of American actions. So it's a low stakes way for people in governments to talk to one another. They usually are the preview for kind of talks among the principals themselves. So that's what a track two talk is. And we should be doing those in AI. And people are trying to do that, you know, getting Stuart Russell or Bengio over to China to talk about the Western perspective, to learn about how do the Chinese think about X-risk, about discrimination, about the military applications of this. Like, what do they think? Because I think, you know, coming back to, like, the Soviet example, which I brought up several times, there was a time in my life when I was fascinated by this question. And they were opening up the archives after the Soviet Union fell. And a lot of historians were going in writing these alternative histories of the Cold War, looking at the actual papers of the Soviets. And, you know, they were, like, evaluating, what did the Russians really think about this? Like, we had this view in America of how they interpreted the world. Right. And, like, what did they really think? Like, what were the Politburo minutes about this? Or what was the, you know, the army writing about this kind of stuff? And it turns out, like, we got it all wrong, mostly. Right. We totally misinterpreted a lot of what the Soviets were doing. And, you know, saw aggression where there wasn't. Or we saw nothing where there actually was aggression. And to me, that's, like, I think, a powerful lesson about the Chinese. It is a different culture. One rich, proud, vibrant, and on the come. Right. And have no illusions that we actually understand, like, what they think about these things. And don't fall into the trap of either marrying yourself or also just presuming, right, that they're, you know, nefarious and evil-minded, you know, and are ready to destroy the world to achieve their ambitions. Recognize that they're a competitor minimally, an adversary maybe, right, but try to, like, understand them better. And these track two talks are part of that. But across the board, you know, Americans should be trying to understand China a lot more. That's a huge weakness for us. I do think we're in this stage now of where we are just kind of projecting our worst fears onto them.
[01:10:36] Speaker 3: I mean, don't you think AI could go a long way to help in diplomatic modeling? I mean, or diplomatic understanding? I mean, why not? Like, it seems like these tools could be directed at, like I said, preventing war and furthering understanding rather than just fighting war, right?
[01:10:54] Speaker 1: I think we—I'd love to see that done. It's not a typical American constituency. Like, what agency of the government does that? Perhaps the State Department or someone, you know? Yeah. I mean, the Department of Defense, right, everything's a weapon for them, right? And that's how they see these kind of tools. But, yes, I agree. Finding ways to calm an understanding. You know, I mean, there's like a school of thought you could think like in a different place in the multiverse where we treat the development of AI like we do public health. You know, we share immunizations, you know? If the U.S. or China cured cancer tomorrow, of course we would share it with China and Europe and Africa and the globe. You know, it's an amazing—it would be an amazing advance. And you would share it. And they, in turn, would probably share it with us. And the idea being like, you could see AI going in that direction, you know? It doesn't seem to be the trajectory we're on. But nonetheless, it remains kind of a distant dream of mine that that is like how we would think of the problem. Or at least open ourselves to the possibility of that without defaulting into this kind of arms race mentality where AI is a weapon and we have to master it, you know? A race to what? What does victory look like in this arms race? What lane am I in? You know, what kind of trophy do I get when I achieve it? You know, do I get a Romulan cloaking device, you know, on this that, you know, for that forever? I cannot be challenged. Like, what is—what's AI going to do that renders war obsolete? You know, what does winning look like? That's why arms races historically are a byword for disaster. And we should get ourselves out of an arms race if we're in one.
[01:12:24] Speaker 3: Speaking of which, I know that one of the pillars on the site is upskilling government. And I'm curious—I mean, of course, there are the testimonies that happened, and those usually seem more performative rather than, you know, anything else. But I'm curious what actually happens in the halls, you know, behind the scenes. Like, how do congressmen learn more about AI? Is it just, you know, educating yourself by going to YouTube and watching, of course, machine learning street talk? Or do you have experts kind of milling around in the hallways there to discuss topics with you, answer questions, you know, deep dive anytime you want? What's it look like?
[01:13:03] Speaker 1: Well, one observation about the state of play and then how people try to get around the problems that the state of play presents to them. So I remember when I was first elected to Congress 20 years ago now. The Congressional Management Foundation gives you this book, kind of like, how to be a congressman, how to run your office, what to expect day to day. And I remember reading in the back, they had a survey of the existing Congress. And they said, "How much time a day do you have to read and to get smarter about issues?" And the answer was 17 minutes. And of course, members of Congress are dealing with everything in domestic policy, international policy, local questions. So they're overwhelmed by just the number of subjects they have to understand a bit. So it's a world, I tell folks, if you want to go into politics, develop your human capital before you show up. Because you only draw it down once you're there. That being said, on these technical questions, there's been a huge effort from everybody from the AAAS to various nonprofits to put technically minded people on the hill. Fellowships from mid-career scientists, maybe to leave their academic position or even industry. Come work on the staff of a House member or senator. You know, industry's paid for some of those. Various nonprofits have done so. Very important. And so now if you go to most of the offices in the Senate, they'll have a fellow there who has some kind of superlative education. You know, PhD from an elite school, expert in computer science, machine learning, biotech perhaps. You know, who are there to advise them about this. Both to get smart about politics when they go back out maybe to their commercial or academic position, but also to give advice. And so you have that going on as well. And the halls are swimming with people now, for better or worse. Lobbyists from the tech companies, but also folks from nonprofits. Civil society now has a huge presence on AI debates in DC. So there's a lot of incoming messaging, but it does come back to that constraint I mentioned. Their time is very, very limited. You know, it's going to be very hard to become an expert on AI when you're in office. So you either have to come in with that kind of background or have really good staff or make it such a priority for yourself that at the expense of all the other issues, healthcare and transportation, you know, that you choose to do that. There are a couple of members like that Don Byer from Virginia, you know, terrific member. He's probably in his seventies. He's going back right now to George Mason University and getting a PhD in machine learning. And he's a very successful guy, very wealthy because he's interested. Now that's somebody who's like made that a focus, you know, but other people, you know, you come to Congress because you care about something else. And so you have to have that staff around them and building up a staff ecosystem and a civil society ecosystem to combat and sometimes compliment the lobbyists from the tech sector. It's been, I think an important part of what the last couple of years has brought to DC.
[01:15:45] Speaker 3: Okay. So, and I, and I see the individual, you know, congressmen and women would have, you know, staff and people who may be experts, but is there also like a shared pool, almost like an internal think tank that, that all of them can access to, to, I don't know, that ideas or talk about the impacts of, of policies, a shared pool of experts or no?
[01:16:08] Speaker 1: Not really. Certain segments of each party will have something like that. Like there's a Republican study committee for very conservative members of the Republican party who has that kind of shared infrastructure, their own staff. They're almost an in-house think tank, if you will. Um, but that's rare. Newt Gingrich, when he took power in 1994, got rid of, there was an office of technology policy that would, was kind of the in-house congressional think tank on tech policy. But he got rid of that, um, in 1994 and it's never been brought back. So that's the challenge. It's like, there's no outside, but congressionally kind of, um, authorized group to do it. Congressional research service can provide you background information. Yeah. Again, you have access to a lot of people in civil society and think tanks, but there's nobody, you know, outside your own staff really, who's thinking big thoughts. And a lot of times your staff is also caught up in the minutia of, you know, day to day kind of fighting and stuff. So that's, I think one of the challenges is that there's no kind of brain trust that can think big thoughts and then inform the parties about that. And do you think this is a, this is a gap? I mean, is it something that we should have? No, it is a gap for sure. You know, um, it's something that again, that used to exist. It's been taken away as these technical questions become even more important than they were 30 years ago. Yes, we have to have groups like that who are in congressionally chartered, not subject to influence either by civil society where you have philanthropists giving money there. Sometimes they're slanted in their views. The lobbyists of course are carrying the water for their respective companies, right? To actually have something like, Hey, you're in the public service here. You're on the government payroll. Your job is to think these kinds of ideas through for it and give us good ideas. I think that would be a powerful, powerful tool. We need to do that. It would make our government more effective. Again, you have groups like Republican study group, you know, kind of the progressive caucus. And the house of the, of the left wing members. There's small groups that do this, but you don't really have one that's, that's incredibly high powered with, you know, with really the best people in it. You have so many people who listen to you are influential in the industry. And, you know, I think I am in many ways somewhat gloomy about the way this is all going to go for us. Because whether our democracy can handle this incredible technology, whether people can find meaning where work is scarce, whether the benefits of AI are going to be, even if not equally distributed, at least widely distributed. You know, I think there's very little in recent American history that thinks we're going to get those kinds of questions right. And it does go to, you know, your question about what one should study and what disciplines are important. You know, when I argue with people like Seb Creer at Google DeepMind or Dean Ball, who was at the Trump administration, you know, I find like we agree about the technology exactly. But we disagree about these questions like how does government respond? What can government do? What should government do, even if capable? You know, like what's the role of government in a society like ours? And these are really live questions that are actually the most important ones today. And I think the big fear that I've had, and so it drives a lot of the work we do here, is that the AI industry can be its own worst enemy. People loathe it. People loathe it. I see polling every day, political polling. It's deeply unpopular. And that's not a good thing for our country. And there's a lot of folks who have pitchforks who are just over the horizon coming after this whole sector. Many of them would shut it down entirely. And they want to do so because they don't give affirmative answers to those questions I answered earlier. They're like, this is a project of the elite, for the elite. They're building a data center in my backyard that's going to maybe change the environment. Maybe raise my electricity prices. And whose sole purpose is to take my job away. And then I turn on the TV and there's a lab leader saying, I'm here to disrupt your world. Irrevocably disrupt your world. And I think the fact that society, the U.S., has lost faith in the project is extraordinarily damaging to what is a very important project with many, many, many upsides. And that's just, we're in a perilous place. And if we don't do something to regain that trust, there's going to be a lot of people, you know, who, you know, are radically opposed to this project and, you know, do their best to, if not shut it down, stymie it. And that's why I said, I think this next few years are really important. And to your listeners who are probably doing machine learning day in, day out, their voices need to be heard. They need to be heard inside their companies to advocate for the right kind of public policy. They need to be heard outside of it. And, you know, their task, all of our tasks, perhaps, is to convince Americans that, yeah, this is actually a good thing. Because the truth is right now, most Americans don't think it's a good thing.
Related Transcripts from Machine Learning Street Talk