How They Became Leading AI Researchers in Just 1 Year – Sholto Douglas & Trenton Bricken

[00:00:00] Speaker 1: I'm curious how you explain what's happened, like why in a year, a year and a half, have you guys been, you know, made important contributions to your field? [00:00:09] Speaker 2: It goes without saying luck, obviously. And I feel like I've been very lucky in like the timing of different progressions has been just like really good in terms of advancing to the next level of growth. I feel like for the interpretability team specifically, I joined when we were five people. We've now grown quite a lot. But there were so many ideas floating around and we just needed to like really execute on them and have like quick feedback loops and like do careful experimentation that led to like signs of life and have now allowed us to like really scale. And I feel like that's kind of been my biggest value add to the team, which it's not all engineering, but quite a lot of it has been. [00:00:51] Speaker 3: Interesting. So you're saying like you came at a point where like there had been a lot of science done and there was a lot of good research floating around, but they needed someone to like just take that and like maniacally execute on it. [00:01:02] Speaker 2: Yeah. Yeah. And, and, and there's, this is why it's not all engineering. Cause it's like running different experiments and like having a hunch for why it might not be working and then like opening up the model or opening up the weights and like, what is it learning? Okay. Well, let me try and do this instead and that sort of thing. But, um, a lot of it has just been being able to do like very careful, thorough, but quick, um, investigation of different ideas. I just don't get blocked very often. Like if I'm trying to write some code and like something isn't working, even if it's like in another part of the code base, I'll often just go in and fix that thing or at least hack it together to be able to get results. And I've seen other people where they're just like, help, I can't. And it's like, no, that's not a good enough excuse. Like go all the way down. [00:01:41] Speaker 1: I've definitely heard like people in management type positions, talk about the lack of such people where they will check in on somebody a month after they gave them a task or week after they gave them a task and like, how is it going? And they say, well, you know, we need to do this thing, which requires lawyers, uh, cause it requires talking about this regulation. It's like, how's that going? I was like, well, we need lawyers. I'm like, why didn't you get lawyers? [00:02:04] Speaker 3: I think that's arguably the most important quality in like almost anything. It's just pursuing it to like the end of the earth and like whatever you need to do to make it happen, you'll make it happen. If you do everything, you'll win. If you do everything, you'll win. Exactly. I think from my side, uh, definitely that quality has been important, like agency and work. There are thousands, I would even like probably tens of thousands of engineers at Google who are like, you know, basically like we're all like equivalent, like software engineering ability. Let's say like, you know, if you gave us like a very well-defined task, um, then we'd probably do it like equivalent. I mean, a bunch of them would do it a lot better than me, you know, in all likelihood. Um, but what I've been, like one of the reasons that I've been impactful so far is I've been very good at picking extremely high leverage problems. So problems that haven't been like particularly well solved so far. Um, but perhaps as a result of like frustrating structural factors, like the ones that you pointed out in like that scenario before, where they're like, Oh, we can't do X. Cause this what team went to do, why, or like, and then going, okay, well, I'm just gonna like vertically solve the entire thing. [00:03:10] Speaker 1: We should talk about, uh, how you guys got hired. [00:03:14] Speaker 3: Cause I think that's a really interesting story. So like the deal, they are this, that I studied robotics in undergrad. And in the meantime, on nights and weekends, basically every night from 10:00 PM till 2:00 AM. I would do my own like research and every weekend for like at least six to eight hours each day, I would do my own like research and coding projects and this kind of stuff. That sort of switched in part from like for quite robotic specific work to after reading, uh, Gwern's scaling hypothesis post, I got completely scaling pills. And I was like, okay, but clearly the way that you solve robotics is by like scaling large multimodal models. I was trying to work out how to scale out effectively and, um, James Bradbury, uh, who at the time was a, uh, Google and is now an anthropic, um, saw some of my questions online where I was trying to work out how to do this properly. And he was like, I thought I knew all the people in the world who were like asking these questions. Who on earth are you? Um, and, uh, he, you know, he looked at that and he looked at some of the, like the robotic stuff that had been putting up on my blog and that kind of thing. And he reached out and said, Hey, do you want to have a chat and do you want to, um, like explore working with us here? Um, and, uh, I was hired, I, as I understand it later as an experiment in trying to take someone with extremely high enthusiasm and agency and pairing them with some of the best engineers that he knew. Um, and so one, another one of the reasons I could say, like I've been impactful is I had this like dedicated mentorship from utterly wonderful people. [00:04:35] Speaker 1: Um, what you mentioned about being, um, being bootstrapped immediately by these people might've meant that since you're getting up to speed on everything at the same time, rather than spending grad school, going deep on like one specific way of being RL, you actually can take the global view and aren't like totally bought in on one thing. So not only can, is it something that's possible, but like has greater returns than just hiring somebody at a grad, so potentially you come at everything with fresh eyes. [00:04:58] Speaker 3: Um, and you know, it come in lock to any particular field, um, now what, like one caveat to that is that before, like during my self experimentation and stuff, I was reading everything I could. I was like obsessively reading papers every night. Um, and like, actually funnily enough, I, I like read much less widely now that I like my day is occupied by working on things. Um, and in some respect I had like this very broad perspective before where not that many people, even, even like in a PhD program, you'll like focus on a particular area. Um, if you just like read all the NLP work and all the computer vision work and like all the robotics work, you like see all these patterns and start to emerge across subfields. Um, in a way that I guess like foreshadowed some of the, the work that I would later do. [00:05:41] Speaker 1: And Trenton does this map onto any of your experience? [00:05:44] Speaker 2: I think Shalto's story is more, more exciting. Um, mine was just very serendipitous in that I, I got into computational and neuroscience, didn't have much business being there. Um, my first paper was mapping the cerebellum to the attention operation and transformers. Um, my next ones were looking at like, uh, it was my first year of grad school. [00:06:04] Speaker 1: Okay. [00:06:04] Speaker 2: Um, so 22, um, but, uh, yeah, my, my next work was on, uh, sparsity in networks, like inspired by sparsity in the brain, uh, which was when I met Tristan Hume, uh, and Anthropic was doing the solo, the soft max linear output unit work, which was, was very related in quite a few ways of like, let's make the, uh, activation of neurons across a layer really sparse. And if we do that, then we can get some interpretability of what the neuron's doing. Um, that started the conversation. I shared drafts of that paper with Tristan. He was excited about it. And, and then, and, and that was basically what led me to become Tristan's resident and then convert to full time. Um, but during that period, I also moved as a visiting researcher to Berkeley, uh, and started working with Bruno Olshausen. And Bruno Olshausen basically invented sparse coding back in 1997. And so it was like the, the, the, my research agenda and the interpretability team seemed to just be running in parallel, um, in, in, with just research tastes. And, and so it, yeah, it made a lot of sense for, for me to work with the team. Um, and it's been a dream since. [00:07:09] Speaker 1: Well, one thing I've noticed that when people tell stories about their careers or their successes, they ascribe it way more to contingency, but when they hear about other people's stories, they're like, of course it wasn't contingent. You know what I mean? It's like, if that didn't happen, something else would have happened. [00:07:22] Speaker 2: Yeah. But I mean, like I literally met Tristan at a conference and like, wasn't, didn't have a scheduled meeting or anything, just like joined a little group of people chatting. And he happened to be standing there. And I happened to mention what I was working on and that led to more conversations. And I think I probably would have applied to anthropic at some point anyways, but I would have waited at least another year. I, I, I, yeah, I, it's still crazy to me that I can like actually contribute to interpretability in a meaningful way. [00:07:48] Speaker 3: I think there's a big important aspect of like shots on goal there, so to speak, right? Where like you're even just going to con choosing to go to conferences itself is like putting yourself in a position where you're, where luck is more likely to happen. Um, and like conversely, my own situation was like doing all of this work independently in trying to produce or do interesting things was my own way of like trying to manufacture luck, so to speak, um, and like try and do something meaningful enough that it got noticed. [00:08:14] Speaker 1: For the people who are like, assuming that the other end of the job board is like, just like super legible and mechanical. This is not how it works. And in fact, like people are looking for the sort of different way, different kind of person who's authentic and putting stuff out there. [00:08:27] Speaker 3: And I think specifically what people are looking for, there is two things. One is agency and like putting yourself out there. Uh, and the second is the ability to do world class something. Yeah. Andy Jones from Anthropic did an amazing paper, um, on scaling laws as applied to board games. It didn't require much resources. It demonstrated incredible engineering skill. It demonstrated incredible understanding of like the most topical problem of the time. Um, and he didn't come from, uh, like typical academic background or whatever. And as I understand it, basically like as soon as he came out with that paper, both Anthropic and opening, I would like, we would desperately like to hire you. [00:09:00] Speaker 2: There's this line, the system is not your friend. Right. Uh, and it's not necessarily to say it's, it's actively against you or it's your, your sworn enemy. Um, it's just not looking out for you. Right. And so I think that's where a lot of the proactiveness comes in of like, there are no adults in the room or like, and, and like, you have to come to some decision for what you want your life to look like and execute on it. And, and yeah, hopefully you can then update later. Um, if you're too headstrong in the wrong way, but, but I think you almost have to just kind of charge at, at certain things, um, to, to get much of anything done. Not be swept up in the tide of whatever the expectations are. [00:09:37] Speaker 3: There's like one final thing I want to add, which is like, we talked a lot about agency and this kind of stuff, but I think actually like, surprisingly enough, one of the most important things is just caring an unbelievable, um, and when you care an unbelievable amount, you'd like, you check all the details and you have like this understanding of like what could have gone wrong. And you'd like, you, uh, it just, it matters more than you think because people end up not caring or not caring enough. Uh, this is like LeBron quote where he talks about how, when he, he sort of, before he sat in the league, he was like worried that everyone would be like incredibly good. And, and then he gets there and you like realize that actually once people hit financial stability, then they, um, like they relax a bit and he's like, oh, it's going to be easy. Um, I don't think that's quite true because I think in like AI research, because most people actually care quite deeply. Um, but there's caring about your problem and there's also just caring about the entire stack and everything that goes up and down, like going explicitly going and fixing things that aren't your responsibility to fix because overall it makes like the stack better. [00:10:36] Speaker 2: Uh, something that a friend said to me a while back, but I think is stuck is like, it's amazing how quickly you can become world-class at something just because most people aren't trying that hard and like are only working, like, I don't know the actual like 20 hours that they're actually spending on this thing or something. And so, yeah, if you just go ham, then like you can, you can get really far pretty fast.