Advice for machine learning beginners — Andrej Karpathy and Lex Fridman

[00:00:00] Speaker 1: You're one of the greatest teachers of machine learning, AI ever, from CS231N to today. What advice would you give to beginners interested in getting into machine learning? [00:00:15] Speaker 2: Beginners are often focused on like, what to do. And I think the focus should be more like how much you do. So I am kind of like believer on a high level in this 10,000 hours kind of concept where you just kind of have to just pick the things where you can spend time and you care about and you're interested in. You literally have to put in 10,000 hours of work. It doesn't even like matter as much like where you put it and you'll iterate and you'll improve and you'll waste some time. I don't know if there's a better way. You need to put in 10,000 hours, but I think it's actually really nice because I feel like there's some sense of determinism about being an expert at a thing. If you spend 10,000 hours, you can literally pick an arbitrary thing. And I think if you spend 10,000 hours of deliberate effort and work, you actually will become an expert at it. And so I think that's kind of like a nice thought. And so basically I would focus more on like, are you spending 10,000 hours? [00:01:04] Speaker 1: That's what I would focus on. So, and then thinking about what kind of mechanisms maximize your likelihood of getting to 10,000 hours. Exactly. Which for us silly humans means probably forming a daily habit of like every single day actually doing the thing. [00:01:19] Speaker 2: Whatever helps you. So I do think to a large extent it's a psychological problem for yourself. One other thing that I hope that I think is helpful for the psychology of it is many times people compare themselves to others in the area. I think it's very harmful only compare yourself to you from some time ago. Like say a year ago, are you better than you year ago? It's the only way to think. And I think this, then you can see your progress and it's very motivating. [00:01:42] Speaker 1: It's also interesting that focus on the quantity of hours because I think a lot of people in the beginner stage, but actually throughout get paralyzed by the choice. Like which one do I pick this path or this path? Yeah. Like they'll literally get paralyzed by like which ID to use. [00:02:01] Speaker 2: Well, they're worried. Yeah. They're worried about all these things. But the thing is, some of the, you will waste time doing something wrong. You will eventually figure out it's not right. You will accumulate scar tissue and next time you'll grow stronger because next time you'll have the scar tissue and next time you'll learn from it. And now next time you come to a similar situation, you'll be like, oh, I, I messed up. I've spent a lot of time working on things that never materialize into anything. And I have all that scar tissue and I have some intuitions about what was useful, what wasn't useful, how things turned out. Uh, so all those mistakes were, uh, were not dead work, you know? So I just think you should, did you just focus on working? What have you done? What have you done last week? [00:02:38] Speaker 1: Uh, that's a good question actually to ask for, for a lot of things that just machine learning. Um, it's a good way to cut the, the, I forgot what the term we use, but the fluff, the blubber, the, uh, the inefficiencies in life, uh, what do you love about teaching? You seem to find yourself often in the, like drawn to teaching. You're very good at it, but you're also drawn to it. [00:03:02] Speaker 2: I mean, I don't think I love teaching. I love happy humans and happy humans like when I teach, I, I wouldn't say I hate teaching. I tolerate teaching, but it's not like the act of teaching that I like. It's, it's that, um, you know, I, I have some, I have something. I'm actually okay at it. I'm okay at teaching and people appreciate it a lot. And, uh, so I'm just happy to try to be helpful and, uh, teaching itself is not like the most, I mean, it's really, it can be really annoying, frustrating. I was working on a bunch of lectures just now. I was reminded back to my days of 231 and just how much work it is to create some of these materials and make them good. The amount of iteration and thought, and you go down blind alleys and just how much you change it. So creating something good, um, in terms of like educational value is really hard and, uh, it's not fun. [00:03:50] Speaker 1: It was difficult. So people should definitely go watch your new stuff you put out. There are lectures where you're actually building the thing, like from, like you said, the code is truth. So discussing, uh, back propagation by building it, by looking through and just the whole thing. So how difficult is that to prepare for? I think that's a really powerful way to teach. Uh, did you have to prepare for that or are you just live thinking through it? [00:04:13] Speaker 2: I will typically do like, say three takes and then I take like the better take. Uh, so I do multiple takes and I take some of the better takes and then I just build out a lecture that way. Uh, sometimes I have to delete 30 minutes of content because it just went down an alley that I didn't like too much. So there's a bunch of iteration and it probably takes me, you know, somewhere around 10 hours to create one hour of content. To get one hour. [00:04:33] Speaker 1: It's interesting. Uh, is it difficult to go back to the, like the basics? Do you draw a lot of like wisdom from going back to the basics? [00:04:41] Speaker 2: Yeah. Going back to back propagation, loss functions, where they come from. And one thing I like about teaching a lot, honestly, is it definitely strengthens your understanding. Uh, so it's not a purely altruistic activity. It's a way to learn. If you have to explain something to someone, uh, you realize you have gaps in knowledge. Uh, and so I even surprised myself in those lectures, like, oh, so the result will obviously look at this and then the result doesn't look like it. [00:05:04] Speaker 1: And I'm like, okay, I thought I understood this, but that's why it's really cool to literally code. You run it in the notebook and it gives you a result and you're like, oh, wow. Yes. And like actual numbers, actual input, actual code. Yeah. [00:05:18] Speaker 2: It's not mathematical symbols, et cetera. The source of truth is the code. It's not slides. It's just like, let's build it. [00:05:24] Speaker 1: It's beautiful. You're a rare human in that sense. [00:05:32] Speaker ?: You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense. You're a rare human in that sense.