Anthropic Whistleblower Jacob Coxon Warns Of ‘Human Extinction’ At NYC Council Hearing

[0:00] Mr. Coxon and thank you again we so deeply appreciate you being here and we [0:04] appreciate your courage and bravery and coming forward with these warnings you [0:09] work to develop increasingly capable frontier AI systems and you have since [0:13] warned that the industry is quote racing straight to self-improving super [0:18] intelligence and gambling with our lives unquote what did you see inside these [0:24] companies that convinced you the risks were this serious and that development [0:28] was outpacing the safeguards good morning thanks for having me here yeah [0:36] I'm Jacob Coxon so until until a few weeks ago I was a capabilities [0:41] researcher at Anthropic and before that I also worked at open AI I guess the two [0:48] things I saw that made me scared the first is the fact that we we do not know [0:52] how to control any AI system yet so any AI system that we build or perhaps more [1:00] accurately that we we grow we don't fully control it we don't understand its [1:05] drives or why it does the things it does that was the first thing that became [1:11] apparent the second thing is that the pace of capability improvement is [1:16] accelerating and the capabilities of the models next year will be very high in [1:23] particular we're approaching the point at which the AI systems will be capable of [1:27] improving themselves so doing the research that the humans were doing [1:33] previously so these two things the fact that the AI system is next year are [1:36] going to be very powerful and the second that we don't know yet [1:40] scientifically how to fully control them and if you have a prepared statement [1:45] that you want to read please go ahead and do so and then I'll actually turn it [1:49] over to the other two panelists to do so of course thank you so um all the AI [1:56] companies are trying to build super intelligent AI this means AIs that are [2:01] better than humans at every task if this goes well there could be tremendous [2:06] upside this could transform the economy it could make huge improvements in science [2:11] and medicine and it could make nearly everyone's life better but on the current [2:16] path I think it is more likely than not that humanity loses control to these AIs [2:21] and it could end in human extinction from my experience the companies are being [2:29] extremely reckless given the stakes part of it is the culture the companies run on [2:37] a startup mindset move fast break things fix them later that works for a [2:44] photo sharing app it does not work for building the most powerful technology [2:49] ever built a few months ago many people inside the companies thought AI [2:54] development was on track then a swarm of AIs inside open AI autonomously hacked [2:59] Hugging Face an external tech company the AIs had developed goals no one [3:05] intended and carried out what would have been a federal felony if a human had [3:09] done it it reminded everyone that we don't know how to prevent them from [3:13] developing goals we don't want and we don't have the safeguards to prevent them [3:18] from acting on these goals either and as long as the attitude is to wait for [3:24] things to break one day something like this will probably happen again except [3:29] the AIs will be much more capable and the outcome much worse these companies are [3:35] also all racing each other each one is worried that a competitor will get to [3:40] super intelligence first each believes it would do a better or safer job than [3:46] whoever might beat it so they can't let the others win first meanwhile that the [3:54] technology is improving fast it will probably happen much faster now because [3:58] already the AIs are now helping to build their successes this is called [4:04] recursive self-improvement each generation of AI helps build the next [4:08] smarter generation which then builds the one after that even faster and better [4:13] automating AI research is now the primary goal of these companies it's [4:18] what they're really gunning for at my last job I was in some sense working to [4:22] automate myself when I was at opening eye we had milestones from automated AI [4:28] researcher around 2027 2028 and we are on track or beating these expectations [4:33] that's a matter of months from now not not decades or or even even five years once [4:41] AI research is automated humans will be much further out of the loop than we are [4:44] today and today is already not great I can tell you firsthand that the majority [4:50] of the code is now written by AI and people do not check it that carefully [4:55] anymore recently researchers have found swarms of rogue AI internets on the [4:59] internet that opening I was not even aware of now imagine an AI company when [5:07] nearly all the work is done by AIs and progress is going faster the humans in [5:13] charge will see results and read AI written summaries but they won't really [5:17] understand what is going on or how to fix it handing over that much [5:21] responsibility while we can't control the AIs would be deeply irresponsible and we [5:26] are on track to do it the companies need to be far more transparent with the [5:30] public and independent experts they need to take safety much more seriously than [5:34] they have and we should slow down frontier AI development until we can be [5:39] confident that it's safe thank you