About this transcript: This is a full AI-generated transcript of Intelligence is collective, not artificial — Prof. Michael I. Jordan (UC Berkeley / Inria) from Machine Learning Street Talk, published June 3, 2026. The transcript contains 16,030 words with timestamps and was generated using Whisper AI.
"Nature said that you are the most influential computer scientist. It exists in the real world. This is an abstraction, but it's a real thing. It's like F equals MA. It'll make predictions. So if I write down a game, just like I wrote down F equals MA in some coordinate system, I can now predict..."
[00:00:00] Speaker 1: Nature said that you are the most influential computer scientist. It exists in the real world. This is an abstraction, but it's a real thing. It's like F equals MA. It'll make predictions. So if I write down a game, just like I wrote down F equals MA in some coordinate system, I can now predict what'll happen. I don't think we need to. See, I think this anthropomorphizing of intelligence and understanding and all that is not necessary, not appropriate, and is a distraction for many, many problems. Why say it understands? I think it's science fiction, and I think science fiction is important for society, but it's also at the level it's being promoted and those kind of voices, it's really hurting 25 and 20-year-olds. You know, these young folks of whom there are huge numbers are excited about technology, and they want to build things that help their family and help their country, actually more their family than their country, honestly. And they see real opportunities in doing that, and they're kind of being told by the leaders, well, we had our fun, we developed a bunch of algorithms, we did it, and we were just interested in the pure, you know, understanding intelligence, even though they didn't understand intelligence. They built, you know, gradient descent algorithms. And now you guys, you can't do this, because it's dangerous. It's going to wipe out humanity with a high probability, or it's super-intelligence will arrive soon, so there's nothing left to do. That's in your lifetime. That is so demoralizing. So demoralizing. And that thing, I think that bothers me the most. I mean, the second part that bothers me is there's no economic thinking going on there. So the current generation is just way too, you know, there's not much thought going on. Not much intellectual stuff. It's just, yeah, it's possible to build it. It's possible to steal the data from wherever you want to, because that's what the internet allowed to happen, and not return any value to the person who originated the data. It's possible to run greedy descent on that, but you need huge amounts of money, but it's now possible to get it from people who aren't thinking very deeply. I don't think it's bad to build systems you don't understand. But I think this level of detachment from reality is unusual for human history.
[00:02:03] Speaker 2: This episode is supported by Cyberfund. If you're building at the frontier of AI, they want to hear from you. Cyberfund believes the future belongs to AI natives who want to achieve the impossible. And that is why they're introducing the monastery for AI native founders. It's an environment of pure focus and rapid execution for founders operating at AI native speed. And they're offering teams $2 million each to participate. Apply now at cyber.fund. And what do you think about the term AGI, by the way?
[00:02:44] Speaker 1: AGI to me is just a PR term. And some people think it's fun because you have to have these great aspirations. I think it's just distortionary. I think it confuses young people. And as we will talk about today a little bit, I think that one of the things I find most alarming about the so-called thought leaders that one will see often on podcasts and other venues is the alarmist tone or the exuberant tone. And I think 20 and 25 year olds are watching that and saying, hmm, I'm not going to be exuberant or I'm going to be alarmist. Those are the two choices. And I hope that this conversation we're about to have is one that makes it clear to young people that there is other ways to approach life and technology. I've never actually thought of myself as an AI researcher. I didn't read an AI book. The term was coined in the 50s and John McCarthy and others had particular goals in mind for coining it. And they had particular methods in mind like logical inference and so on that didn't really quite pan out. In the meantime, in the 60s and 70s, you know, 80s, something arose called machine learning. The actual methods like decision trees and nearest neighbor and logistic regression and hidden markup models were developed in other literatures, mostly statistics, operations research and so on. And that led to industrial success stories. So supply chains and commerce and transportation systems all used and still to this day use vast amounts of machine learning. They used gradient based methods and, you know, the cloud was developed to handle machine learning workloads at Amazon, in fact. And so that's the tradition I came up in. I was trying to think about systems building at scale that would also serve multiple people. The AI buzzword returned, I think, you know, maybe five or so years ago because the data that got to be started to be used was language data. And so the box now is not just making predictions about supply chains or commerce or prices or whatever. It spits out human fluent language. And people said, oh, my God, we've solved the old AI problem. In fact, in some ways, if you define the AI problem narrowly like the Turing test. Yeah. But there was this ongoing tradition of machine learning. And by that time had incorporated people from all different kinds of fields. And it was really having an impact. The industry still is. But the AI buzzword returned because of LLMs. And now, to my view, it's been a distortionary effect on the path of research, on how we think about where research should go. But also on the path of how do we think about business models and how do we think about where technology is going. And AI wasn't enough. They had to create this big hyped up buzzword, AGI, which we will talk a lot about economics, you know, as a source of intelligence, social intelligence. And when it's put together with machine learning style intelligence, you can now talk about, at scale, not just numbers of computers and amount of data, but numbers of humans. And that's critically important to me, that the role of humans as producers and consumers in these emerging systems should be respected, amplified, and thought about.
[00:05:42] Speaker 2: Professor Michael Jordan, it's such an honor to have you on MLST, especially given that Nature said that you were the most influential computer scientist a little while back.
[00:05:52] Speaker 1: It's funny, because I was trained as a statistician and a cognitive scientist, but I'll take it. Amazing stuff.
[00:05:58] Speaker 2: Well, Michael, you've just published a paper called A Collectivist Economic Perspective on AI. Give us the elevator pitch. I was never an AI person.
[00:06:07] Speaker 1: So in some ways, it's easy for me to come in and look at people who are self-professed AI researchers and sort of say, "What are you doing? What's your point? What's your goal?" I think, sadly, they often don't have a very clear goal. It's that humans are intelligent. Humans are a computer. The brain is a computer. And if we mimic that and take aspects of it and parallelize it and make it more powerful, it'll just do great things. And it kind of stops there. It's not that there's a goal in society that we're going to try to do this or that. It'll just solve problems for us. And then we'll be happy. And, you know, I got away from Silicon Valley partly because that's just the way that people talk and I got tired of it. And there's not a lot of intellectual, you know, let's call it deeper, long-term thought going on. And now it became a rat race and a money race and all that. So, yeah, my perspective, maybe it comes from a long tradition of other people having sort of social science perspectives on intelligence. We are social animals and a lot of our intelligence comes by the fact that we aggregate. We aggregate opinions and thoughts and, you know, we have cultures and so on that retain them. And moreover, the society provides a context for our intelligence. A smart action in one context is not in another context and it's all very fleeting and contextual in the moment. And so social science ideas are needed to appreciate what that means. When I say social science, I include economics. So game theoretic. The context is somebody else out there is trying to take advantage of me or maybe to collaborate with me and I don't really know. And so I've got to put out feelers and do signals and create mechanisms where we can interact effectively and economics studies that in a mathematical way. That attracts me because I am a mathematically inclined person. I'm not a critiquer of AI. I want to make it right and I want to make it better and understand what it means to be intelligent in this world and safe and interesting and think about long-term issues. And so to me, you have to do that, you know, formally or mathematically at some level. It's not enough just to build things and put them out there. So when I say a collectivist, I just mean that most of this technology is based on inputs from billions of people. So there's already a collective putting input in and it's meant to serve billions. So there's a collective that's serving. So there's really a big network that's kind of latent there. And then economics criticals. I don't want to just sort of, you know, say words. I want to say, I want to write down actionable mathematical ideas.
[00:08:49] Speaker 2: This is interesting, isn't it? Because I think in the 1970s, Dreyfus came up with this idea of the first step fallacy. And, you know, so we create something that is related to, you know, the McCorduck effect as well. We create something so amazing and we just think we're only one step away from being able to do anything. So these systems, they're incredible, right? They produce beautiful text. They can solve problems. They can do programming. And isn't it weird that they don't actually help us that much? We thought it was going to revolutionize.
[00:09:19] Speaker 1: It's not weird at all. Because the model there is the old AI model that just builds something intelligent. And it's only got upgraded a little bit. It's going to be a better search engine. That's fine. I do think that search engine was major progress for humanity. But now it became more than search engine. It's like a secretary sitting on your shoulder, helping you, whispering things to you. And it's just a dumb business model. I don't think many people really will want that. They'll turn the damn thing off. They want to think for themselves. They want, you know, maybe at the end of the day, a summary or something, but they don't want this all the time. You know, they're interacting with this entity thing. It's not a very good business model. And in the meantime, we have huge healthcare systems and transportation systems and finance systems that are all based on data flows among many billions of agents. And, you know, they're ripe. They already have a lot of machine learning in them and they're ripe for thinking in a more economic way. What are the agents and what are they trying to get out of it? And what kind of cooperation and competition is latent there that you could, you know, make improve, you know, markets arose thousands of years ago. And, you know, we learned about some of the principles, but we can improve them. And thinking arose, you know, billions of years ago or whatever. But we're not perfect, not just in terms of thinking, but we're also not perfect in terms of narrowly following our own agenda and hurting other people, even though we don't want to. Humans are wonderful. We want to prize human life and creativity and, you know, emotion and love and so on and so forth. It's fundamental. But humans are also bad or do bad things. And that's where technology should be able to aid you. And so you need to think about the system. You said these systems, they're not really systems. They're, you know, big statistical boxes that do inputs and outputs. That's not a systems way of thinking. There's a lower level system, of course, the computer system. But I want to be above that. I want to say what ecosystem does this belong to? Who's it interacting with? At what rate and what kind of quality and what kind of values are being created? And when I say value, I mean, often mean money. I want jobs out of this thing. I don't want just it to answer and do things for us. I want it to create opportunities for work and creativity and so on.
[00:11:33] Speaker 2: Rich Sutton is quoted quite a lot in respect of design versus evolve. And I watched a wonderful talk by David Deutsch the other day. And he was kind of talking about explanations. And he said that, you know, physicists obviously go for these principled low-level explanations. But sometimes you get these high-level core screenings that are just really good. And maybe economics is one of those. But what do you say to folks from Silicon Valley like Ilya Sutskevar? And they're just talking about human value functions. And they're saying, okay, well, you've got these LLMs and we just turn them into multi-agent systems. And we get all of the economic stuff that you're talking about for free. What would you say to those people?
[00:12:10] Speaker 1: I mean, it's just not a good way to think about engineering. I mean, if you were a chemical engineer back in the '40s and '50s, saying we're just going to throw a lot of stuff together and make it work. Well, you could do it, but you'd get a lot of explosions and a lot of economically non-viable things. You'd hurt a lot of people. And I think a lot of these people are not thinking about all the people that are being hurt already. You know, Facebook and so on. You know, it's damaged a lot of young people. A lot of teenagers are having mental health problems. And this is just not something that wasn't talked about by a computer scientist at all. And now we're talking about yet another level of displacement of, you know, jobs may go away. But, you know, that's tough. It'll create new ones, of course, like always. You know, I just don't like to talk that way. And, you know, so you've got to say, well, step back a moment. What is your point? Are you trying to create a new kind of market where people could come in and have their talents valued and appreciated or bids could be put out for things that people might need. And, you know, and collaborations can emerge. And there could be producer-consumer relationships being explored and understood and developed. And this could all be a mix of computation and humans. I think eventually we'll all kind of merge. But, you know, along the way, just doing something so disruptive with all of these metaphors. That's not good social science or not good mathematics. It's just metaphors. And, yes, you can build it because the previous generation of people created these amazing things that collect data. And we can do gradient descent on it and ad hoc architectures. And, yes, that works. It's amazing. But let's not give so much credit to the people that did that. It's the people 20, 30 years ago who did that. So the current generation is just way too, you know, there's not much thought going on. Not much intellectual stuff. It's just, yeah, it's possible to build it. It's possible to steal the data from wherever you want to. Because that's what the internet allowed to happen. And not return any value to the person who originated the data. It's possible to run greedy descent on that. But you need huge amounts of money. But it's now possible to get it from people who aren't thinking very deeply. And so, you know, it may seem more dark than I want to. I mean, there's a lot of good in, you know, builders. But we also, every previous era of engineering development, electrical engineering, chemical, mechanical, and all, had some builders. But they had a lot of concepts. And they had a lot of thinkers. In fact, all of those engineering disciplines had something like Maxwell's equations or Newton's equations to help them kind of, here, no. It's just people that are very smart and who can code and then have lots of intuitions. And I don't ever see anything that feels deeply intellectual to me. It feels like science fiction.
[00:14:50] Speaker 2: Well, I suppose another thing that doesn't help is that these systems are like soup. And there's even a field called mechanistic interpretability that tries to kind of dig into the soup. And it's almost like they're searching for UFOs. They're trying to find these principled circuits that do reasoning or do whatever the thing is. And I guess you could say cynically that it's not like when engineers build a bridge.
[00:15:12] Speaker 1: Well, I'm a little less negative than that. I don't think it's bad to build systems you don't understand. But then you've got to kind of put things around it. And the things that are beeping around are like buzzwords, like AI safety. It's a buzzword. Okay. What you really need, I mean, you're human. You can't explain to me why you picked this Airbnb over another one or whatever. All the choices you've made today are inexplicable to me. They come out of your brain. And I don't need to know all the whys and wherefores of your choices. And what I need to know is that you're somewhat predictable. And that if I make certain options available to you, you know, you're likely to take this one versus this one. And therefore I can make my own plans and we can start to interact and so on. So that's part of economics is the economic style of thinking says, I don't understand all these other entities out there. But there's certain, you know, rules of thumb that I can use or, you know, quantitative predictions I can put in place that allow me to interact and not get hurt and even get value out of it. You know, so no, I don't think it's necessary to understand all the details. Now, the input-output behavior you often have to understand better than we can now. For example, if I'm denied a loan in a bank and the bank used this big AI program based on past data, I want to know why. And why doesn't mean that you look in the internals and show me some circuit. No one's going to want that. They're going to want, well, there was like 50, here's like 50 people that are pretty much like you, according to the, you know, embedding we're using in this big network. And of those 50 people that are like you, some of them got the loan, some of them didn't. And then here, here, let me just show you what those people are like. And you start to say, oh, I see that they differ from me in this way. That's actionable to me. I could now change things. So you have to build systems around this predictive system. That's a nearest neighbor system, for example. And that system will supply what people might consider more like an explanation. And so it's not just trying to go in the internals of something. Again, chemical engineering is, you know, there's, there's, there's certainly thermodynamics and lots of things are understood, but lots of phenomena were not understood for a long, long time. You mix up a bunch of stuff and, you know, certain waves are created and certain things happen and you exploit that and go and move on. But you understand something about input, about behavior and, and constraints and so on. I think the current generation of neural nets will continue to, you know, they, they have very nice scaling behavior. They'll continue to be there. But they really have to be thought of as a product, part of a bigger ecosystem. And then you kind of ask, well, what can the neural net do in this context? And what's it missing? And, uh, what if I have multiple of them and, uh, you know, how do, how do they engage with each other and with us and how, what, what is needed? What's what transparency is needed for the overall interaction to be an effective one, whether or not I understand all the details or not.
[00:17:56] Speaker 2: For some reason, and correct me if I'm wrong, I have an intuition that behaviorism is bad, that just by not having any mechanistic understanding and only looking at the outputs. There's the famous example, isn't there, of, of, of, of the, um, the hen didn't know his neck was going to be broken. And one example of this actually is, um, alpha fold. So I interviewed John jumper last week at Google and you did some analysis on those 200 million predicted proteins. And, and, and you found they were very good, but there was something missing, but you could robustify them. You could robustify them.
[00:18:22] Speaker 1: That's correct. And I think that's a good example. So, uh, you know, I'm a big admirer of alpha fold. I don't think it's like an LLM. I think it's, uh, you know, targeted. It was for a particular set of problems and it does it very well. Uh, the, the issue that we found empirically, uh, was that when you ask certain kinds of questions, um, you know, in particular, we did one where we were looking at whether quantum fluctuations in a protein, um, were associated with phosphorylation. Meaning the phone, the protein was active or not in the cell. And you might think that these fluctuations, which lead to strands hanging off or kind of like, you know, bad proteins, uh, the evolution wouldn't use them. But it turned out that a lot of them seem to be phosphorylated. I mean, they're reactive in the cell. Hmm. That suggests a hypothesis test. Is there an association between yes, no phosphorylated and yes, no quantum fluctuation. So that's a little two by two table. And you do a statistical test on that. And, uh, the problem is that if you just use known protein data, uh, that's been crystal, you know, there's a crystal structure known. Um, you don't have enough data to, to test that hypothesis with high power. And so you can't reject the null hypothesis. There's no association, even though there looks like there is. If on the other hand, you use 200 million, you know, proteins out of alpha fold, you can test hypothesis with high power and you reject the null hypothesis. But what we found is that the, um, confidence interval on that statistic of that two by two table was extremely narrow and way far from the truth. The true value of the, uh, the, the gold standard value. And we found this in domain after domain. You know, so why is that? Well, what's happening there is that there's probably not many, um, examples in the training set of proteins with quantum fluctuation. Cause it's not been that studied in the past and it's hard to crystallize. And so not many examples means that's quite possible. Alpha fold won't give out a great answer, but it won't tell you that it doesn't give you out error bars. And it doesn't know specifically on the question you're asking. That's where I want the error bars. And it didn't know about that question when it was, you know, built and designed. Okay. All right. So now I have a good statistical question. What if I add a little bit of ground truth data to the 200 million? Can I shift the error bar? So it stays somewhat narrow. So I have high power, but it covers the truth. And the answer is, yeah, there's a methodology. We've developed something called prediction powered inference that does exactly that. And so it'll cover the truth just like in a classical statistical setting, but it's using this rather highly biased architecture. And it's now, it's not biased overall. In fact, its accuracy is high overall, but for the question I'm asking, it might be very biased. And that's going to happen a lot in science because scientists are rarely interested in just studying the past over again. They're interested in brand new things on the edge of knowledge. And that's where specifically these foundation models will be most poor and most highly biased. So there needs to be around any foundation model, the ability to maybe collect a bit of ground truth data to merge it in with some procedure like this. And then they give out a more trustable answer. That's all not science fiction. That's what can be done and what really needs to be done. And I'm sure the alpha fold people are on board with that, that they would not find that weird or surprising. But a lot of other people out there talk about bias and all that. And they either don't worry about it. I say it'll go away. We have enough data. Or they just critique the architectures and critique the outputs. But they have no scientific method in mind that'll help us go forward. So that's kind of the state we're in.
[00:21:48] Speaker 2: I challenged John a little bit about the extent to which alpha fold understands. And he was basically allergic to the word understands.
[00:21:54] Speaker 3: We are not trying to tell you everything. We are not a model of the entire cell. These machines let us predict. They let us control. We have to derive our own understanding at this moment, right? We can experiment now on the artifact. We can look at the 200 million predicted structures, not just the 200,000 experimental structures in order to help us understand. But it doesn't do the act of understanding for us. It does the act of predict and maybe control. Why should alpha fold understand?
[00:22:28] Speaker 2: Well, what would it mean to us? He was sketching it out to me. He kind of said that this is a weird alien artifact. And it's not like it's kind of created. It's refined. There's this recycled pathway. You can put the thing through multiple times. You can kind of corrupt it halfway through. And the network is just iteratively kind of, you know, it solves the complex bit first. And then it's refining, refining, refining. And like, could we interpret that as an understanding process? I don't think we need to.
[00:22:52] Speaker 1: See, I think this anthropomorphizing of intelligence and understanding and all that is not necessary, not appropriate and is a distraction for many, many problems. Why say it understands? You know, some of my heritage comes from seeing in real life, in industrial settings, machine learning algorithms being rolled out 20, 30 years ago. So, when I first went to the West Coast, I visited Amazon. And around 2000, they were using huge amounts of data to do supply chain modeling using the neural networks of the day. It was random forests. And it was really working. They could make really fantastic predictions of, you know, whether certain ships would be delayed in the Indian Ocean or whatever. So, certain parts wouldn't arrive in time. And the overall supply chain takes billions of products and sends it to 100 millions of people per day. And so, you cannot, there's no way that any human can understand what's happening in that big, big box. But it's not necessary. And in fact, you can ask, does that overall system understand, you know, transport and logistics? And the answer is, who cares? It does a very important optimization and prediction process that allows an engineering system to be built around it. It takes down uncertainty. It makes you possible to do kind of stockpiling and, you know, planning. And that's what you ask for. You don't care whether it has to have a word like understand or intelligence applied to it. That's for the media. The media, that's kind of my problem with a lot of these people rolling out AGI and AI terminology. The media laps it up and they know that even though we don't have a clue what understanding or intelligence means. And we, in our research, realize we don't care or need it. We want to build good systems.
[00:24:29] Speaker 2: Yeah, it's interesting because I agree that we live in this complex, adaptive, irreducible system. We can't essentialize it. And folks like Francois Cholet or even David Krakauer, they talk about intelligence as the, you know, adaptation, synthesis of coarse-grained representations. But what if there is a bit of a step? So let's not anthropomorphize it. Let's say that understanding is about, like, not the end point. It's about the path which led us there. And we know that in the real world we're a collective intelligence and there's the blind men and the elephant. And we all take our own paths and lives. And we have different perspectives on the same whole. So what if, like, a better form of understanding is just being able to reconstruct the thing from your perspective using building blocks rather than trying to essentialize it?
[00:25:12] Speaker 1: You know, that all sounds great. It's just not the language that those of us who research would use. I mean, we would think in those terms a little bit, of course. But we would try to turn it into some kind of an equilibrium or optimization problem. And here's the information that's available. And here's the data. And here's the power and the, you know, the error rates. And we try to put a little bit of structure around it of that form. And, you know, there's always this creative moment. Like, I remember when in high jumping, I used to be high jumping, interested in high jumping when I was a kid. And, you know, you would go up to the bar and you'd jump over it in various ways. And there was the barrel roll, rolling across, you know, was the technique the Olympians were using. And then there's this guy, Dick Fosbury came along and he says, no, if I go backwards, I can do better. And no one had thought about doing that. As soon as he did it, everybody did it. And that's, you know, it went up like by a half a meter or something. I don't know. And so what process led to that? Was it an understanding process? You know, it was just a little bit, let's try something different, mixed in with the ability to try it out and to do tests. So a huge amount of industrial planning is try it out and see what works. Those are called A/B tests. And those are done all of the time. And I've got nothing against that. It's not based on understanding, but it's led to optimized systems that can do things that, you know, people hadn't thought about before. So a blend of that with understanding, but just understanding, you know, I was a cognitive scientist and I, you know, I'm interested in neuroscience. One should be interested in those things. They're fascinating, but they aren't, they aren't the leading edge of thinking how to build systems that, you know, work in the world. And they're not the leading edge of trying to believe in the next generation systems. If you put, a lot of people keep saying, well, we've got to put logic back in our symbols, because that came from the, that came from our previous kind of view of what humans are doing. So hopefully humans are capable of doing some logical reasoning and probably have some symbols, whether they're kind of built in some complicated network or they're reified somehow. I don't know. My intuition is as good as yours. But really the goal is to, you know, in a, I tend to be an engineer at heart, you know, a mathematically inclined engineer. I want to say, what are you trying to achieve? Are you trying to displace teachers? Are you trying to make doctors better? Are you, what are you trying to do? And what, what would be the abstractions and the, and the points of entry into that problem? And then how can you kind of pull back from that and do it in some general way? That's, that's elegant and will inspire others.
[00:27:43] Speaker 2: It's so interesting seeing different scientists, you know, from a multidisciplinary perspective attack this problem. The physicists, for example, they, they work very, very low level and they talk about, you know, the, the dynamics of particle systems and whatnot. And what I'm really fascinated in, I mean, you come at it from an economics perspective, which is traditionally dominated by this agential lens. And you talk about equilibria and incentives and so on. How, how does that come into it? How would you take a very complex system and almost kind of decompose it into this new frame of thinking?
[00:28:12] Speaker 1: It's not been done really enough for me to have, you know, tons and tons of, of great examples. But, you know, we've been looking at kind of modestly scaled examples where there's a, so for example, we looked at a little bit of drug discovery and kind of the, the regulation, you know, the, so I'm a pharmaceutical company. I test out all kinds of proteins and I throw them in animals and maybe a few humans to sort of see what's working. And I have some understanding, quote unquote, in other words, I know somebody have the evolutionary biology behind it and so on. And that guides me. But at some point, someone's got to really test this out in the real world and decide, a regulatory agency has got to come in and say, yeah, that goes to market or it doesn't.
[00:28:48] Speaker ?: Okay.
[00:28:49] Speaker 1: So now you've got a kind of tangled web of scientists and pharmaceutical companies, not just one, but many, many of them and proteins. And, and now you've got to think about how that system is behaving. So the, hopefully the regulatory agency is trying to overall over the entire system, have the number of false positives below and false negatives below. That's what the goal is of the problem. So it's a statistical problem. Oh, but wait, a classical statistical problem. You would just go gather IID data, independent, identically distributed data from some source. Here, no, the data is coming from the self-interested pharmaceutical companies. What's their motivation? Money and whatever, maybe to help, they want to help people and money. And all that's kind of hidden from you as a regulatory agency. Okay. So now the economic mindset kind of comes into play. He says, well, it's hidden from me, but it's not arbitrary. You know, I can kind of probe in various ways. And so that becomes very economic. So, you know, economics, economists think about how you set prices, you know. So if I got a lot of people coming on my airline, there's a thousand people that just arrived to want to go from here to London. Every one of them has a different price point. And that price point will shift in the moment. How eager are they to get? It's not just because they have a lot of money. It's because they have needs. And I don't know what those are. So what I do is I set up various services and various prices that kind of bracket the possibilities so that overall, it's likely I'll make enough money and they will, everybody will be kind of happy and the service will go forward in life. So it's a blend of knowing a few things and admitting that you don't know other things, but putting it in a system that actually can work with that kind of mix of asymmetries and incentives. So, you know, the incentives there are that, you know, there's a certain service and price. If you pick that one, you're likely to be able to get on the airplane. You're likely to get, you know, to have the goodies you need or whatever. And that then doesn't make you do something. It incentivizes you. And in the pharmaceutical world, if I could get them to be incentivized to mostly send in drugs they've done some testing on or they have some belief, it's a pretty good one and not just throw arbitrary ones at me, then maybe the overall system will actually have the error rate you want it to. Okay, because if you don't do that, then if it's a drug that'll make a ton of, you know, a billion people will use, then you're going to make money whether it really works or not. Okay, so you need it to get to market. How do you get it to market? Well, you just throw it at the regulatory agency and there may be a false positive. They just got a false positive and they put it on the market. You make a ton of money. And if there's enough of that, the incentives are all wrong and the overall system will not control type one and type two errors. Those are examples we've actually worked on, but I just hope you can appreciate when all this stuff starts to really roll out in society. It's not going to be there's a few big LLMs and everyone consults them like a search engine. That's just not the model. It's going to be there's local data, like I told you about with prediction-powered inference, everyone's got to vet what's coming at them. There's going to also be local data because I collected it with some expense and I want to just give it away. Okay, thank God finally Amthropic is paying people for money and that has got to be the future. So I'm going to have some competitive value in my data and I can just give it out. All right. And so now if you start interacting with lots and lots of people that want to get some value out of the interactions, you have to talk about the incentives. What's the incentive for them to send the data, but not only send the data, send correct data, send truthful data. Don't just add or not be adversarial. And so I cannot imagine, you know, a fully full fledged version of all this rolling out in society and all of our decision making throughout our lives without a deeply microeconomic perspective, a company in the gradient descent on data.
[00:32:30] Speaker 2: You spoke about this three layer model. So there was an example where, you know, you might have, you know, consumers and they might have their data and you've got Google and then, you know, Google is using the data. The consumers are getting a service and then Google might sell the data over here. That's kind of like a traditional model. Let's start with that. Okay.
[00:32:46] Speaker 1: So those are really kind of like bore Adam kind of things. We're, we're being scientists there. We're trying to say what's a minimal model that exhibits some of the behavior that we want to study here. So let's think about a data market because data is not just now something you have analyzed to build a big LLM. It's also something you would sell and buy and has value. And also there's privacy concerns about data. Okay. So let's put a little minimal model together where we could study that. All right. And so one we've done is called, we call it a three layer data market. And it has, it is, it exists in the real world. This is a abstraction, but it's a, it's a real thing. You've got a user or multiple users coming into some platforms. The platforms provide a service like, you know, imagine, you know, payment service. And as I use that, they get data from me. They learn about what kind of purchase I've made and so on. And they use that data to make their service better. That's a good little, nice little loop there. Okay. The problem is that rarely do they make enough money off of that service. They, you know, take a small cut that the merchants don't like to give them. So they have to do other things to, to, to stay in business. So typically now for a long time now, probably 20 years, they've been selling their data to third party data buyers. And these are not evil people just trying to, you know, ruin people's privacy. They're, they're trying to do market research, learning what, what would work in a, what, what are people really doing? This is behavioral studies. And so the data is valued to them. They pay for it.
[00:34:11] Speaker ?: All right.
[00:34:11] Speaker 1: So, you know, Google doesn't need this because they create this artificial advertising market, which we could talk more about that kind of super powered all this nonsense. But other companies like MasterCard would do have to have to sell their data. So, so now it's a three layer thing. And as soon as that third layer was introduced, the, the equilibrium has to shift because the, the user who's sending their data engine, just lost something. They lost a little bit of privacy. Some third party that I don't know anything about is getting data about me. And I can't just accept that, you know, but I can't walk away. And also there was a stress on the system now. So in an effective economic system, what would happen is that the, you wouldn't just wait for the regulator to come in, the government say, you know, no, this can't be done. And what you would do is that the platforms would say, well, we'll offer you a, a, a tunable level of differential privacy. For some cost. Or we'll just say that this, our company, I'm Google, I'll offer you level, you know, 0.3. And some other company says, well, I'll, I'll offer you level 0.7. Okay. So the user looks at that and says, ah, 0.7. That's, that's better. That's better. Um, I, I really care about my privacy. So I'll go there. That company, then we'll get start to get more data and their service will get even better. And whoop, you've got a little nice little feedback loop there. But now the data buyers will look at the data from that person. 0.7 means more noise has been added to the data. It's less valuable to the data buyer. Data buyer will say, I'll, I'll spend less. I'll give you less money for that. I'll get more money to Google. And so now you can see there's conflicting tendencies here. The, the incentives are aligned, but they're not, you're not optimal for everybody. Um, and so now the mathematics is not just an optimization problem. Mathematics is an equilibrium problem, but it's an equilibrium problem that involves statistical, uh, assertions, data, and how much you can predict with this data and so on that. So you quantify that with error bars and statistical predictions. So you put that all together in a big mathematical system and you can find the equilibria as a function of various system parameters. So for example, is there a minimal level of privacy the regulators could require or not? Or, you know, is there some heterogeneous privacy budget, you know, et cetera, et cetera. You can put in various, uh, those sort of sectors. And now you, you do a little plot of how the equilibria moved and the equilibria have overall utilities for all the three players summed up. That's the social welfare. You can ask how high is the social welfare or that equilibrium versus this one versus this one. And then another regular could look at that and say, well, I prefer this one because it's overall higher social welfare and laws could be made at that level. Okay. So even though this is a toy little, you know, little toy model, uh, it has the ingredients that I'm very interested in. Predictive models, data markets, but, uh, money incentives and a real system that really is already kind of working, but people aren't thinking about it very well. And just like in the drug discovery domain. But if you take an economics point of view, uh, you can make a lot prog, but you can make the system better.
[00:37:12] Speaker 2: Of course, I'm modeling it as a dynamical system, which we can simulate. And then we get these modes of. No, we don't have to simulate it.
[00:37:19] Speaker 1: In that case, it's, you can actually write equations and calculate equilibria. It's, it's a Stackelberg game and you can actually find the equilibria. You know, in other cases you would simulate, but the point is a lot of my machine learning colleagues don't know much about fixed point algorithms and, um, finding equilibria and how they shift as you shift various parameters and all that. That's economics stuff. Um, machine learning people are really good at optimization, but this is not an optimization problem. It's a, and there's all these algorithms and, and, um, other branches of mathematics that find Pareto frontiers and, and do it statistically and, and, uh, do it as a function of size of various markets and size of populations and all that. And it's kind of amazing in this era that the two have almost never met. The economists never had a lot of data to inform their design of their market. So they just wrote down a bunch of equations and made rational assumptions and all that, and then found equilibria mathematically or otherwise. And the machine learning people never thought about the equilibria. They just had a lot of data and they used it to do the obvious thing, predict the next word in a string of words. Uh, but the, the, you know, the future has gotta be that those came, those branches come together. The economics equilibria perspective is critical, but the, oh, it's gotta be adaptive perspective is critical. And also you alluded to earlier, some of the machine learning, uh, Silicon Valley types just saying, well, we've got all this data. Therefore all the behavioral stuff's already built in. Um, and that's, that's too naive, obviously. Uh, but it is a useful point of view. Um, in a certain sense, the economists, uh, do make rational assumptions they shouldn't have to make. And if you put in data instead of that assumption, you'll probably do better. You'll have some of the behavioral economics already built in. But if you do it outside of any economic thinking whatsoever, you'll just make a mess of things. And of course, that's what Silicon Valley seems to be pretty good at.
[00:39:06] Speaker 2: What, what, what is the difference between like data and the kind of knowledge that you're talking about?
[00:39:11] Speaker 1: Um, social knowledge is very ephemeral and it's very in the moment. Um, you know, I can walk down the streets of Copenhagen here and there's all kinds of little mods. markets out there and what's available and at what price and you know, what I might like and all that. It's all super ephemeral. And that, that's kind of, I think a better way to think about all this. You can't just gather enough data to know that that person walking down the street there, they're going to come buy this product. And it just, and all of our decisions that are our choices, uh, you know, cannot be, uh, that you have enough data to cover all of that. You know, everything about what's going to happen even in the next, you know, 10 seconds. Um, so you have to be a little more humble about, you know, that I have a lot of ignorance, but that doesn't mean I can't build a safe system, like a market of some kind that could, people could come in and they can not get cheated. They can get value and, and it could evolve over time. It could shift in ways. And I don't have to be, I'm not the, the, the, the God figure at the top, you know, designing the human value function or whatever to put it in so that it responds particularly well to what humans really want in my God vision view of the world. Uh, no, it's gotta be a system that permits, uh, bottom up preferences to be expressed in the way the human wants to in the moment. And that's not built in and the system respects those things and wants to maybe learn more about them and then use them in the moment and then maybe keep some of it. Maybe it's all ephemeral and goes away, but there seems to be a huge naivete about what data, uh, even if you have, you know, whatever, but exabytes or whatever data, you're going to miss all the details. They're probably the main thing that matters for a particular kinds of class of decisions. Even alpha fold based on huge amounts of data doesn't do for well on certain queries. And yes, they'll patch those and it'll do better, better and better, but always the, the, the new questions that people will ask, I will always be on the edge of knowledge or often be on the edge of knowledge. And people aren't thinking about that. They're thinking about, well, I just going to replace the teacher because the teacher is working not on the edge of knowledge. They're working on back in the, you know, all the stuff that's already known. That's fine. Uh, you can aid teachers, but good teachers also kind of know how to migrate to the edge of knowledge. Yeah.
[00:41:15] Speaker 2: And when we do abstraction and idealization, it's always a little bit lossy. And I'm, I'm fascinated by this observation that market, you said markets were around before capitalism. So it's this bottom up thing. It kind of, it's enough. This is not capitalism.
[00:41:27] Speaker 1: That's one method, one methodology for make markets work, but it's not the only one.
[00:41:31] Speaker 2: Exactly. So it's a, it's a, it's a natural phenomenon that the, the, you know, the emergence of something we might call markets and it's constructive and it's divergent and diverse. And then you're saying somewhere the rubber can meet the road so we can create abstractions. We can do some kind of modeling and we try and do it in such a way that we don't, it's not too lossy. Absolutely.
[00:41:52] Speaker 1: Human culture creates abstractions. Individual humans create abstractions too, that work for them. Yeah. And, and when those abstractions are kind of useful enough and they can be, they kind of get promoted into the culture and that flows up and down all the time. Uh, and indeed that's something that systems could perhaps help with. I'm not going to just trust systems to take on that burden, but that could, it could be helpful. And so indeed, uh, it's not just the individual cognitive entity that creates abstractions. And we should just reify that. Yes, it comes up, but cultures create abstractions. Uh, and you can study the micro economy of that or whatever, or not. You can just sort of say those abstractions came or part of the culture and they're useful. They've stayed around and, and, and they're useful. And as going forward, it's not that we're going to just keep the old ones, but we're going to build systems that allow new ones to emerge. And that is not the God figure figured it all out and putting them in there. Um, I, I, you know, so Silicon Valley says, well, you just got it. We got so much data and we'll have so much that we can do all of it top down. And they're forgetting somehow that first of all, the data came bottom up and the data was all contextual and the data was supplied by people and so on. But they're also forgetting that it all has got to continue to, um, be a micro level that is kind of going to be beyond their ability to sense. And we're not going to want them in so much in our lives. You know, the, after the search engine, which I thought was a fantastic piece of technology allowed access and all that sort of, then a lot of it was very prying. We're going to put glasses on you. We're going to put, you know, things around cameras in your home and all that. And we're going to, we're going to know all the details of your life. And then we're going to, you know, make your life better somehow. And that just, that equation did not calculate for me. Yeah.
[00:43:29] Speaker 2: So this idea that the culture is the abstraction hard drive in the sky and the culture is very adaptable. So we can delete strategy. I mean, knowledge decays very quickly and organizations maintain knowledge and really good bits of knowledge stay around for a long time. And, you know, down at the bottom up, we're creating new bits of knowledge. So how does that whole ecosystem work? How, how do we kind of designate good things that stick around and how do we find a new bit?
[00:43:51] Speaker 1: We don't, you and me, the answer is you and me don't, but, um, that's something that, uh, good intellectuals do. Um, that's what, you know, economics, like there's a whole field of behavior organization, or, you know, how does, how do organizations effectively emerge? And those are, that's really interesting. It's not everything, but, you know, there's a lot known there. And some of it's the mathematical and some of it's not some of its best practices. Uh, but those are the kind of, uh, ways that these, these AI ecosystems should be talked about and not just in terms of neuroscience and, you know, metaphors of the neurons. And, and physics metaphors and all the stuff that it was part of my heritage too. But it just felt, it felt so lacking when we actually see these things, the rubber hitting the road, as you, as you say. Uh, and so yes, behavioral organization, how people are organized into things that promote, you know, not only, you know, good revenue for companies, but also promote democracy and so on. So there are people that talk about all these things. And I just don't think they seem to have much presence in Silicon Valley. Um, and maybe that's for good, you know, Silicon Valley, let them go with, you know, they'll just burn a lot of money and, you know, cause some headaches. And then they'll also creates things like search engines. And, um, and I think a lot of the companies are also really focused on creating value. I do think Amazon is different from meta. Amazon has got a business model, bring packages to doors. And behind that they create some technology to support that. And it's mostly to the good in my view. Um, you have to worry about labor markets and so on, so on. But those are all things, good things to worry about, but just, um, creating, uh, computational artifacts that make predictions. And that if you were to wear goggles, you would be able to, you know, live in their world. Um, it's not a business model. It's a, it's a science fiction dream that it may or not be, may not be helpful for humanity.
[00:45:39] Speaker 2: Well, can we explore that because you gave the example of, um, Spotify. So, you know, we were talking about this three layer thing before, and, and now we've got this weird incentive structure where Spotify are actually incentivized to generate the songs with, with AI. Right.
[00:45:52] Speaker 1: Yeah, they are. And I, I'm a, you know, I have a, a project. I'm a, a scientific advisor to something called United Masters, which is an alternative, uh, which has, uh, musicians keep their, their, their, their, their work. And, um, and United Masters, uh, connects them to brands and to other kinds of opportunities. So that they are kind of more like a real artist. Um, not just like that. They, their, their song got streamed and they got a little bit of money. Uh, cause Spotify indeed is not. It's a, it's a close to perhaps a monopoly, but, uh, it's not incentivized to pay, pay, pay. There's not a pricing. There's monopoly prices, if you will. Um, and so one would hope that somehow the market will fix that, that enough young artists will say, I'm getting screwed here. I'm not making any money and another service will emerge. Um, but we are in an era where some of these services, uh, do become monopolies pretty quick. And that's, I leave to my economist friends to think that through and to look back at historical examples and think about, you know, is regulation needed or is, um, or is there other market making mechanisms that'll make this more healthy for human beings? Um, I'm not against Spotify and say, but you know, it should be part of an ecosystem that actually rewards the artists more right now. An artist is getting paid very, very little. I'm, I think I'm not, don't believe the prices are being set under competitive mechanisms, but I, I think there's this broader, you know, macroeconomic view of what are these systems doing and what are they, what's their role in society going to be? And with the search engine, I was, many of us were puzzled about how they would make money. It just didn't seem like, you know, there was a money making. And then the whole advertising thing was a bit of a surprise, at least to me that it would become so huge. Of course, the underlying thing is that people expect things for free. All right. And so Google couldn't kind of make payments, but I think they made a mistake at some point. I think like with YouTube, when they acquired YouTube, YouTube is more than just pointing people to a website. You know, YouTube is, uh, incentivizing creators to create things that people will watch. At that point, I think a socially responsible Google to critique them a little bit would have. Said, oh, we've created a market here. We've created a producer consumer relationship. We've got to make that market a little bit more valid. And we can actually have that. When someone's watching things, they can have a, some sort of a economic connection to the person who made it. And then it can be incentives flowing that this person now incentivized to make more because here's my audience connected directly. Instead, it was all going through Google and then Google was putting advertisers next to make a ton of money for themselves. And then there's a modest incentive to give back a little bit of money. That to me was a huge mistake. And then Facebook made it even worse.
[00:48:26] Speaker 2: So you've butted up against folks like Jeffrey Hinton and Stuart Russell at Berkeley. And, um, these guys are painting a picture that this technology is recursively self-improving, that it is a gential, that it's not a cultural technology. It's a thing in of itself. This seems a little bit science fiction on the first read. Very science fiction. What do you think?
[00:48:48] Speaker 1: So I think it's science fiction and I think science fiction is important for society, but it's also at the level it's being promoted and those kinds of voices. It's really hurting 25 and 20 year olds. You know, these, these young folks of whom there are huge numbers are excited about technology and they want to build things that help their family and help their country or actually more of their family than their country, honestly. And they, they, they see real opportunities in doing that. And they're kind of being told by the leaders. Well, we had our fun. We developed a bunch of algorithms. We, we did it. And we were just interested in the pure and, you know, understand intelligence, even though they didn't understand intelligence, they built, you know, gradient descent algorithms. Um, and now you guys, you can't do this because it's dangerous. It's gonna, it's gonna wipe out humanity without, with a high probability or it's super intelligent right soon. So there's nothing left to do. That's in your lifetime. That is so demoralizing. So demoralizing. And that thing, I think that bothers me the most. I mean, the second part that bothers me is there's no economic thinking going on there. It's zero. It's really about, uh, cognitive science mentality or neuroscience. And we figured out how the brain works. It's gradient descent with a lot of distributed neurons. And the fact that these LLMs are working so well shows that we figured it out. It wouldn't work so well otherwise. Well, I think that's dubious. We don't, the brain is way beyond. I mean, you ask a neuroscience, if this has anything to do with the brain, basically they'll say no. It's a nice metaphor. It's cartoon. Uh, does gradient descent work at massive scale? Yeah. More than we would have ever imagined. Um, but is it showing its, uh, weaknesses? Yeah. Can it be fixed? Certain areas. Yeah. You know, you build certain verticals, they'll do good things and, uh, it'll make mathematicians go faster, but it won't put them out of business and so on. Um, it's having a big effect on society. I worry more about labor and capital relationships than I worry about it decided to take over. Um, so the rest of it, that, that to me is more on the ground, sort of, you know, how does, uh, uh, the next generation take, uh, technology and work with it. And I don't think that voices like that are actually helping that, that, that, uh, generation to actually perceive what they should work on and why. Um, super intelligence versus extinction. Those are your two options. Um, and then God damn it. Those aren't the only two options. There's a huge, a number of very positive things that can be done at human scale. Um, and let's hope that enough of the young people mentality is kind of get behind that. Um, um, but they don't have enough examples of people out there who made money by making, did Sam Maltin make a life better? You know, not clear. And, and, and, uh, uh, you know, I think in previous generations, there was a little bit more, you know, here's people that are out there making things that, uh, you know, vaccines or whatever. Oh, I want to be like that. And right now, not so good.
[00:51:44] Speaker 2: I don't know if I could, um, get you to be a psychologist for a minute and try and understand why these, I don't know whether it's the search for purpose, but have you noticed as well that some folks think that there's going to be a utopian future. And when you, when you speak with them, they, there's quite similar DNA. Yeah. So they, they also think that it's recursively self-improving. It's going to be super intelligence and so on. But if, if, if I was to press you to say, why do you, I can understand, right? These things are so clever, but why is it? Why do they believe that?
[00:52:14] Speaker 1: I mean, they're, they're clever in the way, in a recognizable way, at some level, they're, they're not, they're taking all this human cleverness and packaging it in a new way. And, um, again, I think it'll kind of always be missing a little bit of the point, cause it's not in the moment. It's not the ephemeral stuff. Um, um, but that doesn't mean it can't be even more clever. And I kind of think that that's okay. I think that the, for me, the goal here is not to build a super intelligence and have it dictate or tell or anything like that. It never was. And I'm kind of shocked that some people seem to think that was always the goal. To me, it just never was. Um, rather again, I think I said this in the very beginning, humans are wonderful. Um, you know, I'd hate to have robots taking over from us. And I don't think that's going to happen. Um, and there's just too much good about human nature and about what humans are, you know, able to produce that are shockingly beautiful, um, and creative and inspiring. Um, we need to support all that. Now the, the issue though, is that the flip side is that, um, we are not, we're far from perfect. You know, our people really hurt a lot of people and they, they are being empowered to do yet more of it. And, uh, we are very narrow-minded. Uh, we also don't understand, like, you know, people hurt other people often because they don't understand their motivations. They got a misunderstanding of how many wars are created because someone didn't understand the intentions of the other side. And they said, well, let's just proactively, let's just bomb them. That's just all the time. That's how humans act and think. What's missing there is an appreciation of uncertainty and information signaling and sort of eventually game theory arose to help people think it through a little bit, but it's still extremely rough. And if you look at our political system, you know, an, an aged, you know, charlatan leading a country, um, you know, this is our optimized human system for making decisions that, you know, the, of the highest kind. There's so much room for improvement of the human being and democracy has got to be the way, but democracies right now are a few aged people sitting in various rotundas, uh, in, in various capitals, you know, not knowing what they're talking about. Mostly. Uh, we have a very broken human system in many, many domains. We have a few that are mapped. I think the universities are pretty good. And I think a lot of companies are pretty good. And a lot of, uh, human associations of various kinds of various skills are pretty good. Uh, but we have so many broken ones. And so to me, that's what AI is about. AI is about, uh, helping the things that were too hard for humans and aiding the information flow. So the humans could actually make the good decision in the moment that most of them really wanted to make. And, and, and, and, and not making the bad decision that they were afraid they had to make because they didn't know enough. So there's so much to me opportunity. If you think about it at that level, that's what AI is about to me. AI is not about this. Replace, uh, you know, the human with the computer, the recursive self-improvement stuff. I mean, it just feels like a, a metaphor. You know, we work with recursive algorithms. We work with improving algorithms. I don't see that getting out of control. Like, you know, a virus that somehow, you know, we're, we're going to work with these systems. And we're going to, uh, like I say, hopefully mostly focus on getting right. Some of the things that evolution didn't quite get right for the human being, especially at scale of 7 billion. Evolution perhaps didn't prepare for that. Um, and focusing on that to me is what AI can be about. So I'm positive. I'm bullish about AI in that sense. I, I, I, I'm appalled by the dialogue has become between the people that have all the money and want to just build something for a build it sake. And the people that are just anti-intellectually saying it's terrible. It's going to, it's going to destroy all of humanity. That's the dialogue in, in, in the public eye right now. And that's just said, so I, I find that so harmful. And, um, and it, it does bother me that people that worked on it for all these years think that we've reached the end, that somehow the gradient descent is like the brain. And therefore you could take multiple brains and you confuse them together. And oh my God, it's going to just do uncalculable things. That's just such science fiction. It's, it's, it, it, whether it's, you know, even true or not, it's worrying, not even thinking about it. It's what, what's the path? How do we engage younger people to do things that are actually positive? And what mechanisms are you going to talk about? What kind of education are you going to talk about? What, what goals are you going to set? And the thought leaders are not talking any of that kind of language. And, you know, I think, I think it's unusual for human history. The thought leaders are, are, are, are heading off in these two directions.
[00:56:42] Speaker 2: It's also complex because there are very real security and safety risks of having any autonomous software, you know, just doing things without direct human supervision. So we, we should say.
[00:56:52] Speaker 1: Well, yes and no. Think about airplanes. You know, that's the classic example, but there's very, very few, uh, airplane crashes at massive scale these days. There used to be a lot when I was a kid and it's because of the autopilots. Yeah. And mostly now planes, planes are flown by autopilots and, um, and the human can come in as need be, but it's, it's because of that. And so there's this blend of automation, uh, with human is actually the most effective way to go is again, it's improving. Humans didn't evolve to be flying this big thing up in the air. And, uh, so you can improve upon human ability there. You put the two together, you can do something that's helpful for everybody.
[00:57:29] Speaker 2: I suppose in that case, it's quite a well-specified problem. So we want to go from A to B and here are the parameters.
[00:57:35] Speaker ?: Yes and no.
[00:57:35] Speaker 1: I mean, you have multiple planes in the air, you have clouds, you have, you know, change in weather patterns. You've got, uh, some person who did something stupid, you know, it's easier. Cause yeah, up in the air, there's a lot of room, you know, in 3d is a lot more room than in 2d. But in, in 2d, you got all these cars floating, flying around and you've got tens of thousands of people dying each year in each country. Um, it's a mess at some level, even though it's very important and effective for many of us. So we do it. Um, but a hybrid system that had a lot of autonomy with some human and so on. But you gotta think about it at the system level, just putting a super intelligence behind the wheel of a car. Dumb, dumb way to think about technology.
[00:58:13] Speaker 2: Is there any hope? I mean, um, I dunno what you think would be the thing that would make these folks update.
[00:58:19] Speaker 1: I think that Ilya and others have done some great things. I mean, they built some systems that all of us are, um, not only using, but kind of, it's changing our thinking and all. And I think that's kind of what I get out of what they're saying is that I'm a builder. I'm not a, you know, you think I'm a guru and a thinker and maybe, maybe I think I am too, but maybe I'm really not. Maybe I'm really better as a builder and I can build things. And with, uh, the resources that are now available to you. And again, it's not just the money. It's, uh, it's the whole internet and the whole, you know, all the things that previous generations of people did that. One thing that bothers me a lot about these people, not the Elon Musk's or the Sam Altman's, they're just coming in and taking the cream off the top, you know, from all this effort that people put in. And a lot of these people are, you know, rightly not just that they wanted the credit. They're just, uh, annoyed that this is the direction it's that these people are now taking it, uh, without the appreciation of what, why were these people building these things? Not for you, but, uh, had other goals in mind. Um, so, you know, I, I think that, um, yes, these are some builders and, um, uh, there's some very, you know, impressive builders, but, um, it, it is, I don't, and I think there's this system that we call it Silicon Valley, whatever that these people live in. And, and we're thinking the more outrageous, the more far flung, the more physics, biology inflected neuroscience inflected that your language is the more you sound like a guru. Um, and, and people enjoy that, uh, that posture and that activity. And, uh, it creates great amount of money. They don't care about the wealth perhaps, but it creates them, allows them to yet be more prominent because they can now have another company that tries some other crazy thing. And, um, and, and so if it doesn't work, that's a sign of you had a great idea. Um, so it's a, it's a, I wouldn't want to be in that world. And I am trying to become a bit of a historian. I mentioned chemical engineering, electrical engineering, but you know, you look back at the history, there were, there was some glimmers of some of these kinds of things. But I think this level of detachment from reality is unusual for human history. Um, this level of my crazy science fiction, 25 year old dreams are all, that's what I'm going to pursue for the rest of my life. Whatever with the, you know, come hell or high water. Um, and then at some point I'll flip because I realized, oops, I didn't really have a great goal in mind at all. And what have I got here? Oh, I've just spent a lot of money and I got this thing and I don't really know what to do with it. And I'm worried about it. You know, that to me is a sign of a certain level of immaturity, frankly.
[01:00:44] Speaker 2: Circling back to, you know, you, you were talking about this statistical contract theory, which is when, you know, we, we, we have, uh, things with an information asymmetry and we model, um, incentives. Um, a lot of folks in the audience would have heard of game theory, right? You know, um, what's the difference? Oh, well, game theory is a discipline, a mathematical discipline.
[01:00:59] Speaker 1: You know, it started with von Neumann in the twenties and, uh, it's got many, many branches to it. Um, and it's a mathematical way of thinking really. Uh, um, and one way I like to think about it is that, um, it's like F equals MA. It's a set of, um, it'll make predictions. Okay. So if I write down a game, just like I wrote down F equals MA in some coordinate system, I can now predict what'll happen. And in the case of F equals MA, I integrate a differential equation. In the case of game theory, I write down the game and I calculate the Nash equilibria or the correlated equilibria or some other equilibrium concept. And I say, here's what'll happen in nature. Because my little mathematical model has the kind of appropriate captures the appropriate ingredients. And for F equals MA, yeah, the, the thing follows a parabolic, you know, curve. It means the theory is right. And then Einstein said, it's not quite right. And he makes a better one. And in game theory, same thing. You look at, okay, do those, those equilibria actually characterize how systems and organizations and people behave? Sometimes yes. Sometimes no. But those aren't, that's not the end all. So there's all kinds of other equilibria, Stackelberg equilibria and sequential equilibria and various kinds of figures of merit, you know, various social welfare constructs, various regret constructs and all sorts of things. It's a whole huge field of its own. And let's think about it eventually kind of being as big as physics, because it's all about strategic interactions and so on. And, you know, not, not molecular interactions, but, but now you can also ask the, the, the inverse question. In physics, the inverse question would be, I want to build a bridge. So my goal is not just to see if something follows a parabolic path or something. I want that bridge to stand up. So I invert F equals ma. Okay. I go from the goal back to the design that would ensure that that thing stood up. All right. And so most engineering fields are inverse problems. They go from the goal back to the design. Whereas the, the forward direction is science. You say, here's the, here's the setup. Here's the prediction. And is the prediction realized or not? So, okay. Yes, it is. That means the model must be good. Uh, so what's the inverse of game theory? Okay. Well, it's outside of economics, not talked about perhaps that much. Uh, game theory sounds like it's sort of everything. Well, the inverse of game theory is what's called mechanism design. And mechanism design says, oh, I want a certain outcome in the world that this person gets paid, that the, the, the, the wealth is divided equally, that, you know, there's some fairness or some market that's created. What game do I design so that that outcome is realized? So I'm the designer of the game. I'm not sticking the game as given and then looking at what it predicts. Mechanism design has got many pieces too. I work in contract theory. That's a part of mechanism design. It says, what if I have two entities interacting? They're not symmetric. They, one knows more than the other and they have to interact with each other. That's, that's contract theory. Auction theory is another part of mechanism design where I've got a bunch of people coming in and I think of them as symmetric. I don't know who's got more money than who wants to bid more than others, but I have this mechanism called an auction that reveals their value. And the outcome is that the person who wanted the painting the most got it. That's that desire. That's one desired outcome. Anyway, long story short, game theory is a super rich, not so old discipline, you know, a hundred years now. That's, that's continuing to evolve and to continue to supply all kinds of algorithmic ideas for those of us who are in the business. So I'm, I've been mostly a statistician in my career, kind of worried about uncertainty and probabilities and decision-making and uncertainty. But when I go to equilibria and games and, or, or economic ideas that I get there, he's got a part and parcel of the thinking.
[01:04:40] Speaker 2: You've said that we need to be thinking about, I mean, we, we've spoken about incentives. We've spoken about collectives. The other big one is uncertainty quantification. Now that there's this wonderful field in machine learning called conformal prediction. Yeah. And it was invented by my professor at university, Volodoy Wolf. Oh. And you know, so we, we learned about the transductive confidence machine. Oh, nice. Yeah. Yeah. These measures of strangeness. So that would be like the distance from a hyperplane on an SVM. And you can basically kind of, you know, I, I suppose, calculate something like a P value, right? And have a confidence region. The E value actually. Oh, an E value. Go on. Tell me more.
[01:05:15] Speaker 1: Oh, well, I don't want to get into technical talk about E values, but just now we're kind of, you know, Vladimir is, is fantastic. And I don't know what he thinks of himself as, but I think it was a statistician, you know, with game theory background too. And, you know, he's in the school of like the Phil Davids in the world and the David Blackwells who spilled out of statistics to do all these other things. And so, yeah, classically P values were just kind of a one shot quantity that statisticians would talk about that. It was like Fisher that said, I've got a model of what's going to happen in the world. It gives a probability distribution on the outcomes. Some outcome arrives. It looks very improbable under that model. The model must be wrong. That's kind of the P value. And so the P value is the tail probability. The problem is if you do that repeatedly and you look at maybe the smallest P value along the way, that's called P hacking. And that gives you wrong answers mathematically. And then in practice. So E values are different. It's an expectation of some non-negative random variable or a non-negative super martingale in more generality. So you're watching this evidence kind of accruing and you make sure the expectation of that evidence is less than or equal to one at each step. And then you can think about a multiplicative kind of evidence gathering that if it's always an expectation less or equal to one, then it'll kind of stay below one. And if it's non-negative, it'll just kind of decay, decay away. So under the null hypothesis, I've got this stochastic process, which is kind of decaying away. Well, I can look at that at any time and sort of assert that it's decaying away and I can look at it repeatedly and keep asserting that. And I can have control. There's something called Ville's Inequality that Vladimir and others have exploited that says that can be controlled over the entire path of this thing. So now we can do statistics in a new way. It's called any time inference. We can peak. We can change. We can gather new data. We can do this in an update every day. Very liberating. And Vladimir is one of the leaders of that. And E value is one of those martingales stopped at a particular time. By the optional stopping theorem, you can stop it whenever you want. So that has opened up a lot of connections. In fact, our statistical contract theory, what is a contract? Remember, it was like services and prices. Well, the services are like evidence gathering. And then the price also is part of the, it's a random variable. And it turns out that we can have an incentive compatibility in contract land if and only if E value in statistics land. So there's a nice tight connection between game theoretic probability and the theory of incentives. So I, to me, uncertainty quantification is rarely just here's an error bar. That's kind of classical statistics. And it's more what the context is here. The context might be a contract or it might be some other evidence gathering mechanism. And this way of thinking opens you up to a broader class of evidence gathering.
[01:08:13] Speaker 2: Very cool. And I should say in your paper, you have this figure of a triangle, which we put on the screen now. But you're kind of saying there's, you know, there's economics and there's computer science and there's statistics.
[01:08:22] Speaker 1: Well, these are thinking styles. I don't even call them by those disciplines. So there was a paper by Jeanette Wing, you know, a few decades ago talking about computational thinking. So it says, oh, computer science has developed these thinking styles that are more abstract than just computers. It's like modularity and abstractions and APIs and all that. And why don't we teach everybody in all the sciences and all the disciplines to do computational thinking? And I think that's totally right on. That's great. But lots of algorithms don't come about from those kind of computer science principles. They come about from thinking about inferential uncertainty. And how do I gather data to, you know, make predictions about things that don't yet exist and think about incentives? How do I make sure that, you know, incentives are in place? And I called those two kinds of thinking. One of them inferential thinking. So not just statistics. A lot of fields have inference in them. And then economic thinking. It's not just economics. It's social scientists of all kinds and legal scholars and so on. When you put those three together, you get a pretty good platform for training of the next generation and a pretty good platform for problem solving of the kinds that we've been talking about this entire time. Just one of the fields. Just computational algorithms and optimization. That kind of gives us LOMs. Fine. Great. But it doesn't give us any of the context around the LM. The incentives kind of gives you the whole thing we've been talking about. And then statistics to me is critical. It thinks about, you know, about what kind of errors I'm going to make, how to make sure the data is, you know, controlled so I don't make the errors. And we put the three together. Yeah. They also bring kind of some partners. You know, the economists talk to the behavioral, you know, psychologists, the computer scientists, you know, talk to the physics people or whatever. The statisticians talk to the legal people, whatever. There's a whole sub-communities that come together. So to me, if you put on this triangle there and you think it's around it, it starts to become a new way to think about academia. This is the liberal arts of the era. This is the core. And now my colleagues in the humanities might disagree. The core is, you know, still the humanities, but I just don't think it's touching the core intellectual issues of the era, which is, you know, about data and about compute and all. But I want to put the ingredients in place that those things are thought about in a societally responsible way. But could you bring this to life?
[01:10:41] Speaker 2: So you famously spoke about, here's a language model. And I'm going to ask it, how confident are you about the answer? And it tends to be quite modal. So it'll either be like, you know, one, zero or nought. And like, what's the difference? Why does the language model not really have any idea about its confidence?
[01:10:58] Speaker 1: You should ask the language model builders, because all they're doing is predicting the next word. And there's not any thinking about uncertainty quantification in doing that. And you can graft in ideas, you know, but they're dubious. They're not, they're dubious. They're often putting up dubious prior in or, or, and, and so you can, you go to the statistician and, and that's what, you know, people have done. And they've said, okay, I can just treat it as a black box and I can put conformal prediction around it. It's a nice method, doesn't require a lot of assumptions. So yes, that's true, but it makes a lot of, you know, it's not, there's an exchangeability assumption. The data, you know, if you scramble it, you get, it's the same. And, and so while I think all of that's really crucial and important, I, I tend to think more about the broader context. So, you know, I gave an example in that article that you mentioned of, you know, a, a, a duck who goes to a lake and that this is a statistician duck. So it's kind of calculated that over the last year, there tends to be, you know, twice as much grain on that side of the lake than on this side, two to one ratio. All right. So now the next day I need to decide on the duck, which side of the lake I go to. And the, the Bayesian duck, um, who has those probabilities would then do the maximal expected value. And they would go to the left side of the lake with probability one because they're all right. But the actual ducks don't do that. They go to probably two thirds to that side of the lake and one third of the other side. They're hedging. Um, but, but it's not just a hedging thing. Hedging would just do occasionally going to the other side of the lake. They're actually getting the right ratio. And then, and so the explanation is that you weren't thinking about the context, right, of this uncertainty. Okay. And it's not just you, the individual duck, probably you evolved in a world where there are many ducks. And if all the ducks went to the same side of the lake, obviously you've missed out on a resource. And so is there an algorithm that allows many ducks to cooperate here? Um, well, if they all have that same uncertainty, then they can sample with probability two thirds and go to this side versus one third. And that's actually a Nash equilibrium of the bigger system. All right. So the right way to think about uncertainty there is that in the context of the population, what should be, how should I use my uncertainty? Uh, another kind of uncertainty, that's kind of the economic side. Another uncertainty in economics is the one I've alluded to information asymmetry. You know, things I don't know, and you have expertise I don't know about, but we're going to work together and I'll maybe give you a contract, a menu of options. Um, but even if I interact with you for a while, I still might not know. You'll, there's things you're going to know that you're not going to give away to me and maybe you'll hedge, you know, you'll lie a little bit. So I don't know about that. That'll never, that's not just sampling. That's a different kind of uncertainty. Okay. And then finally, there's what I like to call provenance, you know, that's more like a database kind of uncertainty. Um, if you, um, if I want to do a medical operation and, um, you're a doctor and you look at the data for people like me, uh, here's the, you know, if you do the operation this way, the probability of survival versus this. And I look at that and say, great. But now you tell me, oh, that data was gathered 10 years ago. And I'm going to say, okay, my confidence interval should go up. All right. Well, classical statistics, you know, could talk about that. In fact, it'd be, I'd be more of a Bayesian to think about that, but it doesn't. It just sort of the data is the data. Uh, and it, and it should be in a bigger system that is data is flowing around. There should always be tagged with metadata about how old it is. And that should be quantitatively brought into the uncertainty quantification. We're not doing anything like that right now. And so the poor LLMs, you know, which are basically doing none of the above, have to strive for that. We have to strike out a little bit in all these directions. If they're going to start to do like what, what humans do. We are pretty good at, um, getting these little bit of provenance. Oh, it's old data. I discount that. Uh, we get a little bit of context. Oh, there's a social environment here. You know, I should just do the same thing. Should randomize. Um, oh, there's some sampling uncertainty, you know, and so on. We put all that together almost seamlessly. And then we do this in a social context where if I don't know how to get from here to the other side of town, I will ask someone who looks Danish. Uh, I know something about how to gather more data and so on. So the poor LLM has none of the above. And, um, so what should it say when you ask, how sure are you? And all it's doing to the best of my knowledge is that it's just, well, in the past, someone asked a human on the internet, how sure are you of that equation you just wrote down? And someone said something, oh, I'm very sure because of this or that. And I think it just mimics that those kind of assertions, but that's not reasoning under uncertainty.
[01:15:19] Speaker 2: And if we did have epistemic, um, you know, quantification, what would be the main uplift from that? Is it about, I know I don't know something, so I'm going to kind of lean in and try and do more epistemic foraging in that area? Well, again, I think we're now in statistics land.
[01:15:35] Speaker 1: You know, the statisticians are all about what species are present on the island. Have I sampled enough to know that there's not a new species? That's because these are classical areas of statistics, uh, optimal experiment design. Uh, you know, for that subpopulation, I don't have enough data. I'm making a bad inference and data collecting in the context of inference in the collection, in the context of, um, you know, making assertions and doing that repeatedly. That's what statistics is long focused on. So I think give them credit for handling a kind of active form of uncertainty reduction. But again, for me, certainly reduction in the, in the large comes about from much broader sets of, uh, of, um, components like a market. Like if I, if I, I use example in the paper where I'm, uh, you know, I want to have a restaurant like this for pizza and I need tomatoes. And so if I had to forage for tomatoes every day, that would be pretty uncertain whether I would have pizza that evening. But because there exists a market where someone else did the foraging, there's a stable amount of tomatoes. Every day I can, I can build my restaurant assuming that that's true. That my uncertainty for finding tomatoes went down. Therefore I can build on top of that and do other things. Markets mitigate uncertainty and they, they don't do it because someone designed an optimal experiment design. Or, you know, ran or did some multi-armed bandit, you know, uh, not directly, uh, but because, uh, the market did try various things out. There's incentives for people to explore and exploit.
[01:17:00] Speaker 2: Professor Jordan, it's been an honor having you on the show. Thank you so much.
[01:17:03] Speaker ?: All right.
[01:17:03] Speaker 2: It's been my pleasure.
[01:17:04] Speaker 1: I've enjoyed talking to you.
[01:17:05] Speaker ?: Thank you.
Related Transcripts from Machine Learning Street Talk