Keynote: AI-Powered App Development - Steve Sanderson - NDC London 2026

[00:00:00] All right. Good morning, everybody, and welcome. Thank you for coming here. I'm always impressed [00:00:15] by the people who make the effort to actually get out of bed even on the last day of a conference [00:00:20] and get in through the rain. So, good job to all of you. So, just before we get started, [00:00:27] before I say anything stupid and embarrass myself, I just want to check what people in the room here are [00:00:33] already doing in this kind of area. Can you do me a quick hands up if you already use AI to generate [00:00:38] code in some way? Yeah. Oh, man, that's like almost everyone. How many of you are using one of the CLI [00:00:44] based tools like CloudCode or Copilot CLI, something like that? Yeah, right. We're talking maybe about [00:00:50] 40% of the room just there. All right. So, that's cool. That's very interesting to me because I work [00:00:55] on this stuff all the time. My name's Steve. I work at Microsoft and I'm on the GitHub Copilot [00:01:02] Coding Agent Runtime team, which obviously needs a shorter name, but that's what we do. I do this [00:01:08] all the time, right? So, I'm thinking about AI coding agents. I'm using coding agents to build coding [00:01:14] agents and talking to other people about coding agents. This is a fascinating area for me. It feels [00:01:20] like possibly the biggest shift that's happened in the technology industry since I joined it. It [00:01:26] definitely is changing the way that I produce code on a normal day-to-day basis. It changes the type of [00:01:33] things that I create and what I feel empowered to do. And I'm sure many of you have got different views [00:01:40] in the room about this. Like, how many of you think this is all possibly a really terrible mistake? [00:01:44] Anyone? Yeah? All right. So, maybe only about 10% of people here are willing to admit it, but, you know, [00:01:50] it is a concern, right? What are we actually getting ourselves into as an industry? What are we going to [00:01:55] become? So, let's think about this a little bit. Let's think about what we want the software industry [00:02:00] to actually be in the first place or why we got into it. So, cast your minds back to some point in the [00:02:06] past when you felt that you were really happy in your career as a software developer, when you were working on [00:02:12] something that you enjoyed. Like, think of a team that you worked on that you thought was good, [00:02:16] a project that you were happy to go in in the morning, fire up your computer, get your coffee, [00:02:21] and start getting to work on something. Because the industry has brought us all into it, probably for [00:02:28] common reasons. You know, I imagine that we've got a lot in common in this room, why we joined this [00:02:33] industry. One of the main reasons is probably just we like computers, right? Like, most of us, since we were [00:02:38] kids, we've just enjoyed doing stuff with computers, and we enjoy using our brains to solve problems with [00:02:45] the computer. You know, we'd like the idea that it's complex, but deterministic. You know, it's being [00:02:51] fair to us. If we can solve the problems, it will work correctly. And we can use our brains to translate [00:02:58] our ideas into something that will make a difference in the real world to our users. And this is all great. [00:03:04] We've had a good career up until this point by working this way. But it starts to feel like [00:03:10] something is shifting yet. Because up until now, we were the only ones who could create working code, [00:03:15] right? We had that unique ability that other people did not. It was like our personal moat that gave us [00:03:20] some special economic advantage. But now machines can create code as well. And maybe even non-technical [00:03:27] users can use machines to create code. Is this a good idea? Or are we running ourselves off a cliff? [00:03:33] So let's think about how we got to this point, right? So it probably started a couple of years ago, [00:03:40] when a range of tools started to appear that were showing the ways that we can use AI to produce code. [00:03:48] All right. So we had things like cursor and VS Code's agent mode, which were IDE-based tools, [00:03:53] showing you how you can have this chat with an agent. And as it goes, it's going to start writing code [00:03:59] for you. We've also got GitHub Copilot on the web, where you can, you know, you can file an issue, [00:04:03] you assign it to the agent, and it's going to make a PR for you. All right. So that's kind of like [00:04:08] a first wave of this technology. But then there was also a second wave that came along fairly quickly [00:04:14] after. The wave of the CLI-based tools. So Claude Code was the first one that came along and sort of [00:04:22] defined this whole product category. And that was rapidly followed by others like Gemini and Codex CLI [00:04:30] and Copilot CLI. And I know a good number of you in this room are using those right now. But it's kind of [00:04:36] weird, right, that we've decided to go back to this older UI technology when we've clearly got much more [00:04:41] modern alternatives. So why are we doing this? It's very strange, right? I think that these CLI-based [00:04:49] coding agent tools, they're kind of like, they're often described as like dopamine machines. It's weird, [00:04:53] right, because they're not tied to any particular IDE, they're not tied to a repo, they're not tied to any [00:04:59] particular web-based thing. You just pop them open on your computer anywhere, you give it a task, and within [00:05:04] five to 10 seconds, you've got some sort of progress on that task and you can feel good about it. Then [00:05:09] you do it again 10 seconds later and another 10 seconds later. And it's weirdly addictive. The [00:05:13] teams building these CLI-based tools have been able to iterate really fast. You know, they've been freed [00:05:19] up from some of the modern UI concerns and been able to just focus on making the agent really good. [00:05:24] And they've rapidly been able to get something much better than you might be used to if you haven't used [00:05:29] one of these kinds of tools. But does it make a difference? Well, people will have different [00:05:35] opinions about it. And if you believe them, some people will tell you that they're able to produce [00:05:40] work a lot faster working in this sort of way. Now, I personally feel that I can relate to that. I [00:05:46] definitely feel like I'm able to produce code faster in a wider range of situations than I did before. [00:05:52] And I don't think it's just me. Here's some stats from the team that I work on. So, this is just [00:05:56] one week running up until a few days ago. And this is a team of about seven to 10 people. And, you know, [00:06:04] within one week, we've merged over 200 PRs on that team. Now, I will admit that about half of these [00:06:09] are just trivial, like, you know, add a line to the documentation or whatever. But I would say, [00:06:13] realistically, 100 of them are serious features that would have been several days of work in the past, [00:06:18] you know, well implemented, full set of tests and all that kind of thing. And so we're getting what, [00:06:22] 100 of these a week out of 10 developers, 10 a week per developer, like 10 non-trivial features [00:06:28] per week per developer. That's, you know, a big step forwards compared to what I'm used to in the past. [00:06:35] Okay. So, let's see what it feels like to do. And let's go on a little journey where we'll have a go [00:06:43] at how fast can we create some kind of feature. And then we'll start to think about where this takes us, [00:06:48] what kind of new ways of thinking it forces us into, and what kind of problems it might [00:06:53] potentially cause for us as well. Because this is not all sunshine and happiness. We might get [00:06:58] ourselves into some difficulties with what we're doing here as well. So, let's try and anticipate [00:07:02] some of that. All right. So, let's try and implement a feature. Let's say that you've just joined the [00:07:07] co-pilot team. And in your first week, you want to add a feature where, well, in your first half an hour, [00:07:13] you want to add a feature where it can tell you what models are available to interact with. So, [00:07:17] we've got co-pilot here on the command line. And let's say that we want to run show models. All [00:07:23] right. Now, the problem with that is it doesn't exist as a feature, right? It's an unknown option. So, [00:07:27] let's implement it ourselves right now. We'll just check that we're in a good clean state. We are. [00:07:32] And we'll pop open co-pilot and we'll try and get to work on that. All right. So, CLI tool comes up, [00:07:40] and we can say to it something like add a dash dash show models command to CLI that lists available [00:07:51] models to process.standard out. Okay. So, it's going to get started on thinking about that. And [00:07:57] how's it going to do it? Well, I don't know. It's non-deterministic. It's decided that it's going to [00:08:02] use the explore sub-agent in this case. So, sub-agents is a new feature that I'll explain to you in a [00:08:07] minute, but that's another agent that's specialized in working out how to do stuff. It's figured out [00:08:13] where the CLI entry point is here on index.ts. It sees a pattern that it can follow. It's just [00:08:20] telling us that it's going to add dash dash show models. It's figuring out where to add it. It's [00:08:26] spotted that there's some kind of similar thing that it can follow. It started editing the file just now. [00:08:32] Okay. Good. Is it done yet? No. It still needs to actually add the implementation of the feature, [00:08:38] which it's doing. It's decided to build it. So, it's going to check if there's any compile errors. [00:08:44] Are there going to be errors? Nope. All right. It's now going to actually run it itself and see whether [00:08:49] the feature that it implemented actually works. It says that it does. It's verifying that it's in the [00:08:53] help and I'm just going to stop it at that point. All right. So, let's see what we've got now. [00:08:57] So, let's go and have a look at what changes it's made. All right. So, it's added dash dash show models [00:09:03] to the list of options and it's added a handler for that as well that's going to log some models. [00:09:09] Let's just check if it actually works. So, if I do copilot show models now, it does. It produces a list [00:09:15] of models. All right. Cool. So, that was pretty good that we were able to get from nothing to a feature [00:09:21] in a minute or so. And you might think, all right, but so what? It's only like a five line feature, [00:09:27] isn't it? And yes, it is. But the hard part of implementing a feature like that, it's not really [00:09:31] the five lines. It's the working out where you're supposed to put them and exactly what constitutes [00:09:36] following the correct patterns and that sort of thing. And obviously, if I'd let this go on longer, [00:09:41] it would have gone on and written some tests for me and other things like that as well. All right. [00:09:46] So, that seems like a pretty good starting point for this sort of thing. So, hopefully, [00:09:53] it doesn't seem ridiculous that we can expect to ship multiple non-trivial features a day. [00:09:59] Obviously, that was a fairly trivial feature, but we could do something much bigger if we were going [00:10:02] to be working for an hour or so. All right. So, if we're able to produce our changes this fast, [00:10:08] it forces us to think a little bit differently about code. We start to take a different relationship [00:10:14] to our project. So, here are some of the ways that, on our team, we now have a different attitude [00:10:21] to code. And to the kind of rules that we've had for our whole career going up until now, [00:10:26] maybe some of them don't really apply anymore. So, I think we quickly start to feel that code [00:10:31] isn't a precious asset anymore. Code is cheap. If you want to make a lot of code, [00:10:36] you can make a lot of code very quickly. If you want to prototype something, particularly if it [00:10:41] doesn't have to be done to a production standard, you can make it unbelievably quickly. Now, [00:10:46] prototypes are basically free. Now, there's still a big gap between that and production-ready code, [00:10:51] but specifically when it comes to prototypes, very, very cheap, very, very fast to do. And when we [00:10:57] start working on something and we find that it's going to require a whole bunch of non-trivial [00:11:01] implementation, in the past we might have thought, oh, well, that's going to be a lot of time to [00:11:06] produce it and it's going to be hard to maintain it and all that sort of thing. We don't necessarily [00:11:10] feel that way anymore. We might just feel, all right, so I need a thousand lines of code to [00:11:15] parse something, so what? You know, the Opus model will produce that for me in a few minutes. And if I [00:11:20] don't like it later, I'll throw it away. If I need to rewrite it, I'll rewrite it. If I need to change the [00:11:23] language, if I need to change libraries, whatever, I'll just regenerate it later. The code itself is cheap. I don't need to be [00:11:30] precious about it. And when it comes to design processes, in the past, if you were going to do [00:11:36] like a week-long piece of feature work, you would have probably scheduled a bunch of meetings and [00:11:41] wanted to like really build consensus for this feature before you started working on it and worked [00:11:46] out what all the architectural concerns were going to be and anticipate the problems up front. We don't [00:11:52] really do that anymore because what we find now is if you've got a feature that you're going to produce in [00:11:58] a couple of hours, just do it, right? You know, the agent will spit out an implementation of something [00:12:04] and in fact, there might be multiple different ways of doing something. Do all three. It's fine. And [00:12:08] then you can compare all three different ways to implement something. Learn by actually doing the [00:12:13] work and then seeing what the outcomes of it are, what the architectural problems actually turned out [00:12:17] to be. And then at the end of it, you might just throw it all away because it turns out to be a mess in [00:12:22] some way. But if it doesn't, maybe one of those three is what you want. So it is a different way [00:12:27] to approach the work that we do. All right. So let's get a little bit deeper into this. Let's start [00:12:34] thinking about some of the ways that you can customize and tune what an agent is going to be doing and [00:12:40] hopefully make yourself more successful than you would have been. Because as we go through this, we are [00:12:46] going to find that some things are actually going to cause new challenges for us. And there are going [00:12:52] to be a whole bunch of unknowns that we'll encounter. And we need to start of trying to anticipate what [00:12:57] we're getting ourselves into and what we're going to need in order to be more successful. [00:13:02] Okay. So when it comes to coding agents, there are a lot of different features and terminologies that [00:13:09] you might or might not be familiar with. Here are a load of words. And these words correspond to [00:13:16] features in coding agent tools. Now, obviously, it varies a bit from one to another. Claude Code is [00:13:20] going to have different features compared with Copilot or Gemini or Opus or Opus as a model or like [00:13:25] OpenCode. They're all going to have slightly different features. And I'm not here to try and push you [00:13:30] towards one or the other. I think all of them are going to be something that you can be successful [00:13:34] with. I'm going to show you Copilot, so I'm actually on that team. But you know, there are similar [00:13:38] features in all of them. All right. Now, I'm going to start by showing you the subagents feature in a [00:13:43] bit more detail. But before I explain it, let me just start a demo because it takes a minute to run in the [00:13:48] background. So I want a large code base to start with to show you something. And so here, I've got a game [00:13:56] engine. It actually implements Doom, which I'm sure will be familiar to most people here. It's written [00:14:02] in .NET, actually. And here's a pull request that came in a few weeks ago. And let's say that your job [00:14:09] was to review this pull request. Well, you might look at it and say, oh, two files changed. That's going [00:14:14] to be quite an easy review for me to do. I'll get that done in a minute or so. So you go in and you have [00:14:19] a look at the changes that they've made in these two files. And you see, oh, you know what? This is [00:14:25] not going to be so easy to review, actually. This is actually kind of terrifying, especially if you [00:14:31] really want to know that every single one of these lines of code doesn't have like an off by one error [00:14:36] in it or something like that. That is going to be a non-trivial process. So what if we could get [00:14:42] our coding agent to do the review for us? Well, maybe we can or maybe it's just going to hallucinate some [00:14:46] garbage. How would we know? So let's try it. I'm going to go over here to Copilot, and I'm going to [00:14:53] spin that up. And I'm going to tell it that I want it to review this. So I'm going to say, [00:15:00] I would like you to review the PR, paste that in, in parallel across Opus, Haiku, and Gemini. Now, [00:15:10] those three things, those are different AI models, and they're going to have slightly different takes [00:15:16] on this code. Some of them will be stricter than others. Some of them are going to hallucinate in [00:15:20] some way, and others might not. But by running this parallel code review across multiple agents, [00:15:26] I can get multiple levels of feedback and compare them and try and spot what's the real information [00:15:31] and what's not. So you can see it's fetched the pull request from GitHub. It's using its large output [00:15:37] handling here, actually. So because models have got a limited context window, it's decided that this is [00:15:43] too much to go in the context window. So it saved it to a file on disk. And then each of those different [00:15:48] agents can read different parts of those files on demand. All right. So it's got the PR content, [00:15:54] and it's started the parallel review across these three different agents. And then it's going to have [00:16:00] to wait for them to produce their output. All right. So while that's running, let's go back over here. [00:16:06] Right. So sub-agents. That's this thing that I've just shown you here, where an agent can start [00:16:11] multiple tasks in parallel with different context windows, different tools, different models, and so on, [00:16:16] and combine the results and orchestrate things for you. Plan mode I'm going to get to in a minute. I'll [00:16:21] demonstrate that for you. That's a way that you can anticipate or work with the model to make a [00:16:27] comprehensive plan for the work that you're going to be doing and try and avoid the case where it [00:16:31] does the wrong thing. Skills is a relatively new enhancement. How many of you have seen or used skills [00:16:38] so far? Yeah, pretty small number. All right. So that's going to be worth digging into a little bit. [00:16:43] So skills are a way that you can create reusable capabilities for coding agents and then share them [00:16:49] across different people on your team. I'll show you an example. Delegate is a way of sending stuff up [00:16:54] into the cloud after you've worked on it locally so that you don't have to use up your own machine. [00:16:59] Memories. That's a way of keeping track of facts that can span your team. So if one of your colleagues [00:17:06] tells the agent something like, oh, never use this way of writing a test, always write tests in this [00:17:10] other way, then that memory can be stored and tracked within your project so that other coding agents [00:17:16] will automatically pick that up. Hooks gives you deterministic callbacks so you can change what an [00:17:22] agent does. The other stuff is all cool too. But let's see how we're getting on with this [00:17:27] review. I can see that Haiku, oh, actually they've all just finished. Right, so Haiku finished first and [00:17:32] then it was Gemini and then Opus. And it's now writing up this combined review across all of them. Opus has [00:17:39] found a particular problem. And let's see. Okay. All right. So it's given us this report. Opus has found [00:17:46] an issue. Haiku didn't find any issues. Gemini found three issues. I don't want to read them individually. [00:17:52] I'm just going to tell it I want to know how they compare. So what were the main disagreements? [00:18:00] All right. So let's see how these different reviews compare with each other and what we can learn about [00:18:06] this. So we've got there's something to do with element bounds.empty where Opus says there's a bug. [00:18:14] Haiku doesn't. All right. So what does it say? It says Opus is more rigorous. The math does overflow. [00:18:24] The overall assessment. Opus found an issue. Haiku said there's no problems at all. And Gemini found three problems. [00:18:31] All right. So this kind of lines up with my experience of these models. If you don't know, Haiku is the smallest [00:18:36] one of these models. It responds very quickly, but it doesn't have very good deep insights. [00:18:40] Gemini is a slightly more anxious model. It will tend to complain about a lot of things and it will pick up on a lot of [00:18:47] architectural concerns, even if they're not really real issues. But it's interesting to get its points of view. [00:18:53] And Opus is sort of the big daddy of the coding agent world at the moment. And that's probably the [00:18:57] one whose opinions I'd be most interested in. All right. So that's how we can orchestrate multiple [00:19:03] agents quite trivially from a coding agent tool like that. All right. So let's have a go at digging a [00:19:12] bit deeper into some of these features. And also, let's just just think a little bit about some of the [00:19:18] challenges that this is going to give us right now. So I don't know about you, but back in the day when [00:19:22] JavaScript frameworks are coming out every three days, I used to find that was quite hard to keep [00:19:27] up with. When it comes to coding agents, even more so the case, among this list of features here, [00:19:33] probably about half of them is stuff that's all been invented in the last three weeks or something. [00:19:37] That's not an exaggeration. So like trying to keep up with this is almost a full-time job in itself. [00:19:43] And it is my full-time job and I struggle with it. So I pity the people who don't get to spend that [00:19:49] being their full-time job. And it's not just stuff that's shipping in the products themselves. [00:19:53] Other people are coming up with even more complex things that you can do on top of coding agents. [00:19:58] Here's a couple of examples. All right. So this thing called Gas Town started getting a bit of [00:20:03] traction on hack and use recently. Steve Yeggi is a well-known person in the software industry. He's [00:20:09] created this sort of fictional world of coding agents where you've got a mayor and they've got convoys of [00:20:15] beads which represent work and they send them to different agents. And, you know, it's this whole [00:20:19] like fictional let's pretend world of coding agents. And does it work or not? I don't really know. [00:20:25] But people are playing around with this sort of thing. And another one that is became incredibly [00:20:31] hot about two weeks ago for maybe three days was Ralph Wiggum, which I'm sure some of you have heard [00:20:37] of and some of you are probably just thinking like, why am I here? Why did I get out of bed to have this guy [00:20:42] going on about Ralph Wiggum? Why is that even relevant? Well, the thing is that coding agents [00:20:48] can be really lazy. And Ralph Wiggum tries to solve that problem. So you'll tell your agent, [00:20:53] hey, do this task. It's got six parts in it. And it will do it for a bit. And it'll say, I've done [00:20:58] four of the six tasks. So I'm finished. And you say, what? No, go and do the other two tasks. And it [00:21:05] says, yes, you're absolutely right. I'll do those two other tasks. And then a minute later, it'll go, [00:21:09] those are quite hard. So I'll leave those for the future. And you'll say, no, do them. And it will [00:21:16] say, okay, yes, I will. And then it'll say, I've added to do comments to say that we'll do these in [00:21:20] the future. And you think, why am I doing this? So Ralph Wiggum tries to just force it to actually [00:21:27] do the work. It's simply a while loop that just keeps saying, do it, do it, do it, do it over and [00:21:33] over to the coding agent. And the idea is you leave it running overnight or whatever. And who knows how [00:21:38] many tokens it consumes. But when you wake up in the morning, you've ported your entire operating [00:21:42] system to Rust and created AGI. That's the idea of it. Does it work? Does it not? Why is it called [00:21:49] Ralph Wiggum? Well, these are all mysteries. But I would encourage you to think about this stuff, [00:21:54] but not get too caught up in it. Because this is just going to keep changing all the time. And I don't [00:22:00] know about you, but you will probably start to experience a bit of FOMO when it comes to all of [00:22:05] this stuff. Like, how can you know what is worth thinking about and what is not worth thinking about, [00:22:11] given how quickly we're iterating? So to try and counteract some of this FOMO effect, I want to take it [00:22:18] back to some fundamentals that I think are the actual gems within the pile here. From experience [00:22:26] of working on a coding agent team for the last six months, what do we think are the fundamentals that [00:22:31] actually make a real difference to whether an agent succeeds or fails with a given task? And we're going [00:22:37] to start with planning. And you might notice, actually, that all of these are things that are [00:22:41] generally good for humans as well. So it kind of makes sense. Right, planning. One of the main reasons why [00:22:47] tasks assigned to coding agents might not turn out in the way you want is that the problem might be [00:22:51] underspecified. And agents are very lazy. If they can get away with just saying like, oh, I did this in [00:22:57] a sort of whatever was the easiest way, whether you wanted it or not, they'll kind of do that. So [00:23:01] planning is a way of ensuring that you've got a full communication of what you want the agent to do [00:23:07] up front. Let me show you that. Right. So I need another application to work on. So I've gone with [00:23:13] literally the most mainstream business scenario I can think of, which is just standard e-commerce, [00:23:19] right? So I've got this open source.net e-commerce platform here, totally normal. The only unusual [00:23:26] things are the insane prices in the test data and the fact that the languages seem all messed up. But [00:23:31] apart from that, it's just normal e-commerce. All right. Now, let's say I want to add a non-trivial [00:23:37] feature into this application right now. So I could go into my coding agent here within that repo. Let's [00:23:43] start up co-pilot. And I could tell it to start work. Okay. But how do I tell it exactly what to do? [00:23:51] You'll notice down here that we've got this shift and tab to cycle mode thing. And I can do that to [00:23:57] switch myself into plan mode. And while I'm in plan mode there, it knows that whatever I tell it to do, [00:24:03] I don't really want it to start implementing it. I want it to work with me on working out a plan. [00:24:08] All right. So let's tell it a complicated feature to work on. Add a product customization feature. [00:24:17] Okay. Now that is potentially an enormous piece of work for it to do. Now, what does that even mean, [00:24:23] product customization? So it's going to start thinking about it. It's going to research our product [00:24:28] code, hopefully. Yeah. So it's listing the files. It's using that explore sub-agent again to explore [00:24:34] the architecture. And it's thinking about this project and what that might mean. It says product [00:24:40] customization is ambiguous. I need to clarify what the user means. So how is it going to do that? Okay. So it [00:24:46] brings up a multiple choice option for us. And we can type in an arbitrary answer if we want to. [00:24:51] But let's say I'm just going to pick that I want the build your own product option here. So let's put [00:24:56] that in. And it's going to think a little bit more. Maybe it's ready to write a plan now. Maybe it's not. [00:25:02] Okay. It says, what's the use case for build your own? I'm just going to go with custom PC builder. [00:25:07] All right. Maybe it's ready to start now. I don't know. Let's see. All right. Should the system [00:25:14] enforce compatible? Okay. I don't want to think about this anymore. This is a demo, right? So I don't [00:25:18] need it to be a complete thing. So I'm just going to tell it, no more questions now. And it will now [00:25:26] have to get started working on that. All right. So it's going to think about this feature. And hopefully, [00:25:32] it will come up with a plan about how to actually do it. All right. So let's just wait for a moment [00:25:37] while it contemplates that. Create implementation plan. It's listing files. It's grepping for stuff. [00:25:45] I have enough information to create a comprehensive plan. Let me create the plan file. Go on then. [00:25:50] Okay. So it's created this file. And you can see this gone into a sort of temporary directory [00:25:54] rather than my actual project, because it might just be an ephemeral plan. We can move it into the [00:25:59] project later if we want to. But it's done that in the background. And it says you can press Control Y [00:26:03] to actually read the whole thing. So I'll do that. And then hopefully that will be in my IDE. Okay, [00:26:09] over here. All right. So here's what it's just come up with. Problem statement. Add a build your own [00:26:14] feature. Here's my proposed approach. Here are some entities that we'll add. Here's the work plan, [00:26:18] split up into multiple phases. Here's the technical architecture stuff we'll add, database stuff, [00:26:25] open questions, and so on. Now, this is a really important document if you're working on a big [00:26:31] feature like that. And if I was doing this for real, I would spend a long time working on this [00:26:35] document. I would easily spend a full hour just working with the agent on refining this plan, [00:26:40] and actually breaking down each of these pieces of work, checking that I understand what the [00:26:44] architecture is going to be like, anticipating any problems, telling it about other stuff, [00:26:49] and so on. And it can update this plan for us automatically. Or you can just literally edit [00:26:54] the file yourself if you want to. In this case, I'm just going to add something that I often add to [00:26:59] these things. For each phase, add a validation line to the bottom saying what I'm going to be able to see [00:27:11] in the UI to check it works. Okay. So, this is a tip that I would give you. Often, it's really helpful [00:27:19] for your coding agent not to just produce code, but to structure its work in such a way that at each stage, [00:27:25] there's something that you as a human can objectively observe in the UI to make sure that it's actually [00:27:30] doing something useful. You don't want it to just produce code, and it compiles great, but how do you [00:27:35] know it's the right code? You want to see it do something. All right. So, you can see it's doing [00:27:39] a bunch of further edits to that plan document. It's now added like this validation step, and it will [00:27:45] keep on adding more validation steps. And then while it's actually working on this task, it will keep [00:27:51] the plan up to date. So, if it learns new things, it will add them into the plan, and it will mark off [00:27:56] the tasks as done as it does them. And that will allow you to keep track of its work, and it also [00:28:01] allows it to keep track of its work over a long time horizon. Okay. So, that's the first one of these [00:28:08] big techniques. Next one I want to talk about is composition. All right. So, when you, as a human, [00:28:16] are taking on some difficult task, you don't learn it from scratch every single time. You bring your [00:28:21] skills and your tools that you already know about from the past that you can apply, and that allows [00:28:26] you to work much faster. And the same thing can be true with coding agents as well. And skills are a way [00:28:32] of defining certain abilities that an agent can have that you can then keep track of and share across people on your [00:28:38] team or even across the whole industry if you want to. So, let me show you how you can get skills, [00:28:44] how you can create skills, and how you can use them. So, the first way you can get skills is by [00:28:48] installing skills that other people have already published onto the internet. So, let me give you [00:28:53] an example of that. So, I'm going to go back over here. And in a new session, I'm going to first list [00:29:00] the skills that I've got installed. And you can see that I've just got one skill initially. Now, [00:29:06] forget about this. This is going to be part of a demo that I do in a few minutes. So, pretend that you [00:29:09] didn't see it, and that there's just no skills installed right now. And I'll show you how we can [00:29:14] install skills that someone else has made. All right. So, the first way that we can do it is that we can get [00:29:19] them from something that's called a marketplace, which is basically just another word for a Git repo that [00:29:24] contains skills. All right. So, this is a Git repo, Anthropic slash skills. And if I go over here in [00:29:30] my browser to it, that's this one here, then you'll see that this kind of provided by Anthropic is a [00:29:37] collection of skills that you can plug into your coding agent. And we've got things like MCP builders, [00:29:43] PDFs, PowerPoint, stuff to do with themes and other stuff. All right. So, we can dig into what each of [00:29:49] these are, but let's just have a go at installing this collection of skills firstly. [00:29:54] All right. So, we need to just paste in a command because it's too hard for me to remember. [00:29:59] So, what is it that I need? It's this. So, plugin install document skills, blah, blah, blah. All right. [00:30:06] And so, that's installed this document skills plugin. And now, if I list my skills, you can see I've got [00:30:11] loads of them and they correspond to stuff that we saw on that repo a minute ago. Now, what even are [00:30:17] skills really? What are they mechanically? Well, you'll see that when I create one in a minute, [00:30:23] but another way to look at it is by going to this website that gives you the spec for it. And it [00:30:28] shows you that a skill is basically, at root, just a markdown file. It's a markdown file with a bit of [00:30:33] metadata in it. If we can see an example of one, here's PDF processing. It's got a name, it's got a [00:30:38] description, and then it's got loads of information about how to do it. And this might be a really long [00:30:42] document. So, the clever thing about skills is the way they are progressively disclosed to the model. [00:30:48] Models have got a limited context window. And if you had a thousand skills and each of them had got [00:30:53] a massive document associated with it, you would not want to load all that into context upfront. It [00:30:57] would just confuse the model and make it perform badly. So, the great thing about skills is it only [00:31:02] has to load this description into context upfront. And that's just enough for the model to realize when [00:31:08] it needs to use a particular skill. And when it realizes that, it can research it and get the full [00:31:13] information. And it's not just markdown content. It can be tools and scripts and things like that [00:31:17] as well, as I'll show you. So, best way to understand this probably is to create our own skill right now. [00:31:24] So, let's say you want to make the ability to share across your team for coding agents to check on your [00:31:29] product's build status. All right. So, I'm going to do that now. I am going to firstly show you that we [00:31:37] just installed this skill creator skill. So, easiest way to create a skill, use the skill creator skill. [00:31:43] All right. So, let's say use skill creator skill to create a skill in and that location. It should ask [00:31:52] user which project and we'll do .NET ASP.NET Core or GitHub Copilot SDK and get build status for it. [00:32:04] Right. So, that's going to, it needs to know to use the skill creator skill, right? So, it doesn't [00:32:10] know that initially. It's going to realize that this task corresponds to skill creator and it's going [00:32:14] to load that into its context. And once it's done that, it's going to do a bit of research and find [00:32:20] out how exactly you're supposed to create skills. And then hopefully it will actually do that. It's [00:32:25] running the Python script that it's like a project template for skills. And if we go over here, I should [00:32:29] be able to show you actually creating that. So, it's going to appear in here. And it's an [00:32:33] empty file right now. Okay. So, it's checking what it's doing. What's it doing? The file is empty. Yes. [00:32:41] Okay. It's done it now. Right. So, it's added this build status skill here with this description. [00:32:48] And it's got a workflow that corresponds to what I just told it, which is ask the user which project [00:32:53] and then check. Now, obviously, I can edit this file directly if I want to change how the skill works, [00:32:58] but I can also just instruct the agent to update it as well. So, let's say I only want it to get build status from [00:33:03] the main branch. So, let's say actually only get status from main. Okay. So, it's going to figure out [00:33:11] how to do that. And it's going to make some edits to this hopefully. So, I think, okay, yeah, it's done [00:33:17] it. It's just added this branch main thing onto the skill right now. Okay. So, now that's something that [00:33:23] both I and other people on my team would be able to use because this will be stored in source control in [00:33:29] my project. All right. So, let's have a go actually using it, shall we? So, I'll go into a new session [00:33:35] and check that we've got it installed. And you see that we do by default because it's in the project [00:33:39] there, build status. And now I can use that. Notice also that it shows up as a slash command. So, [00:33:45] we can auto complete on the name of that skill as well now to invoke it really quickly. So, let's [00:33:52] submit that. It's going to think about it. Obviously, it knows that we want it to use the build status skill [00:33:57] there because we've used the slash command. It's going to load that into context and it's going to [00:34:01] ask us, all right, what project do you want to check status for? Let's do copilot SDK. And it's going to [00:34:08] invoke that skill now. So, it's going to follow whatever instructions are specified in there and [00:34:14] merge that in with all whatever other tasks it's doing. It's, what is it doing? Okay. So, it's use GitHub [00:34:21] actions to fetch the build status and then it's parsing that and it's producing this happy wall of green [00:34:28] text. So, good. It works. And that's something that you could easily start reusing amongst other people [00:34:34] on your team. All right. Great. So, we're making good progress here on how to guide an agent to be [00:34:41] productive in our scenarios. Let's take on this last one now. I think this is probably the most important [00:34:46] one of them all. So, another really annoying anti-pattern that coding agents can get into [00:34:52] is you tell it to implement something and it goes, yay, I did it. I implemented it. And you go and [00:34:57] check it and it doesn't work. So, you go back and you say, it doesn't work. And it says, [00:35:00] oh, I'm really sorry. I'll fix that for you. And it does some change. And it says, I've done it now. [00:35:04] And you try it and it still doesn't work. And it's just really annoying being in that cycle. [00:35:08] But you can break out of that cycle by giving a feedback loop to the agent. So, it can check its [00:35:13] own work as it goes. And that can be done at multiple levels. So, at the simplest level, [00:35:17] make sure it's able to actually compile your code. Because then if there's any build errors, [00:35:21] it can see them and address them itself. If you've got linters, if you've got a test suite, [00:35:26] again, make sure it's able to actually run them itself so that it can detect any problems. But the [00:35:31] biggest power move, I think of all, is to give it the ability to actually run your real application [00:35:37] and interact with it like an end user. Because that way, it can do what you would do as a developer. [00:35:42] When you implement something, you actually try it in your application, right? Why can't the agent do [00:35:47] that? Well, it can, if you give it a skill to do so. So, to demonstrate that, I am going to go back to [00:35:53] this simple commerce thing again. And imagine I want to add some other feature into that. And I'm going [00:36:00] to do so using a skill that I prepared earlier, because it takes too long to create this. [00:36:06] So, I've got this simple commerce automation skill. I used the skill creator to create it, [00:36:12] obviously. It took me about half an hour or something. And I've taught it how to automate a [00:36:18] browser using Playwright so it can interact with my application. And I'm giving it lots of info that's [00:36:23] specific to my app. Like, here's the standard admin credentials for our test data. Here's how to set the [00:36:29] site up in the first place. Here's how to start a browser. You don't have to work it out. I've provided some [00:36:34] actual scripts for you that do that. I've got all these other little JavaScript snippets that teach [00:36:39] you how to explore and list links that are on pages and navigate around, register users and [00:36:45] all kinds of stuff like that. And you can build up more and more capabilities over time. So, let's try [00:36:51] using it, shall we? I'm going to get out of here. And I'm going to start a new session. [00:36:59] And then in here, I am going to go to my list of skills. And you can see that I've got this simple [00:37:06] commerce automation skill in there. All right. So, let's use that to start a browser. All right. So, [00:37:15] hopefully, it will discover that it needs to bring that skill into its context window. And it does. [00:37:20] And then it should be able to use it. So, it's using a script that's provided in there. And you [00:37:25] can see a real browser has just popped up here. All right. That's good. Now, based on all of that [00:37:31] knowledge, I can tell it to do other things with it as well. So, let's say, for example, I would like it [00:37:36] to use that skill to register a new user, Steve. All right. And because there's a built-in script for [00:37:44] doing that, it should be able to do this quite reliably. So, it discovers which script to use. [00:37:50] And it goes through that process. It was pretty quick. But you can see it's registered this user [00:37:54] and logged in as that right now. Okay. And I could use this to make it do more complicated things as [00:38:00] well. So, let's say that I wanted to work on some sort of order management thing. And I need some test [00:38:06] data. Or I want it to be able to check the order management actually works. I could say to it [00:38:11] something like, use that skill again. And as admin, create a new product. Let's call that the super [00:38:21] headphones. And then place five orders for it as Steve, five separate orders, actually. Okay. So, [00:38:32] that's a fairly non-trivial flow that it's going to have to go through. But it should be able to do it, [00:38:37] because our skill tells it all about how to explore the site, how to log in as admin, and things like [00:38:42] that. So, it's generated a script. It's logged in as admin. It's gone to the product creation wizard. [00:38:49] It's going through that flow. It's created super headphones. It's logged in as Steve again. And it's now [00:38:55] starting to place orders as Steve. All right. So, clearly that is useful if you just want to automate [00:39:02] stuff for testing purposes. But the bigger point that I'm trying to make with this is that you really [00:39:09] want the agent to be able to do this itself because it chooses to. You want it to have that power whenever [00:39:15] it implements something to go, "Now, I'm actually going to check this works. Let me pop open a browser [00:39:20] and click through the flow and make sure that this feature is really implemented properly." And if [00:39:24] there's any errors that come up, it can see those errors and it can go back to the code and start [00:39:28] making changes to it. Okay. So, that's some of the fundamentals that I want to get to. All right. [00:39:37] So, so far, everything's been great. We've been moving forwards rapidly. We're just more productive. [00:39:43] We're happier. Nothing could go wrong, right? Well, I would say that this is great, but there are some [00:39:50] real drawbacks that we want to start thinking about as developers as well. And there are some difficult, [00:39:56] hard questions that are probably even beyond the scope of what we can get to. But just at a simple [00:40:00] level, one of the biggest challenges that I see around me right now on the team I'm on is just the [00:40:06] sheer rate at which we're producing code. So, here's a snapshot of our list of pull requests from a [00:40:13] couple of days ago. And this was just the first two hours of the day. So, we've had six new PRs come [00:40:19] in, five of which are waiting for reviews. And this is while just the Europeans are online. So, there's [00:40:24] only three of us in Europe. The rest of the team's all in the US. They're asleep right now. And we've [00:40:28] already got six PRs out and we're waiting for each other to review them. How are we going to manage this? [00:40:34] Like, this sheer rate at which code is being created, who's going to read it? How do we know [00:40:38] that it's actually correct? Like, I don't think that this is a solved problem. I'm not here to say, [00:40:43] the answer is this, because we don't know the answer right now. This is something we're working [00:40:47] at. Obviously, we're thinking, can we throw more AI at this problem? But we don't know, right? It's an [00:40:51] unsolved problem right now. It's a challenge for us. And another challenge that I want to bring up, [00:40:56] and I don't know if you'll relate to this or not, but it's something that bothers me a little bit, [00:41:01] is when I go back into my IDE and I start trying to write code with my actual fingers like an animal, [00:41:08] it's a weird experience to me now. Like, I'm not sure what buttons to press. And I just, like, [00:41:14] look at my hands and think, like, make the code come out of them somehow. And I'm not sure how to do it. [00:41:20] Because I guess genuinely my, like, muscle memory is starting to change in some way. And I feel like [00:41:26] when I want a load of code to come up on the screen, I should say it in English. I don't, [00:41:31] it feels weird to me to type it out now. And I'm not proud of that. I don't think that's necessarily [00:41:35] a great thing. I don't know what to do. Like, is it a problem or not? Do I need to do some, like, [00:41:39] daily manual coding exercises to keep myself sharp or something? Or is there no point? I honestly don't [00:41:45] know. But I'm very interested to hear what the rest of you think about that sort of thing. And [00:41:50] obviously, there are deeper questions still. Like, you know, where are we really going in the long [00:41:55] term as an industry with this? What difference is it going to make to our, like, role in the world, [00:42:01] the supply and demand of software engineers, that sort of thing? We don't have the answers to these [00:42:05] things right now. I don't feel qualified to predict the future, so I'm not going to. But I would love to [00:42:10] chat with any of you about this and see what sort of things you are thinking about this. But despite [00:42:15] all of these unknowns, I don't think that I'd be willing to let anyone take this away from me now. [00:42:21] Like, the sheer sense of productivity that I get is not something I'm going to give up on. I do feel [00:42:28] that I can do stuff that I wouldn't do before. So, quick example of this. Last week, I was using some [00:42:35] screen recording software, and it was crashing. It's open source. So, I went to the GitHub repo and I [00:42:40] wanted to see if other people reported this issue. Is there some sort of workaround? And other people [00:42:44] had reported it, but there was no workaround. This thing was written in Rust. I don't speak Rust. So, [00:42:50] normally, I would not have been able to do anything about that. But I thought, oh, let's just see what [00:42:54] happens if I just get Copilot on this. So, you know, I told Copilot, go to this repo, clone it, [00:43:00] investigate the following issue. If you can find a fix, submit a pull request. And that's it. I didn't [00:43:06] tell it how to clone it, how to submit a pull request, any of that stuff. And genuinely, in under 10 [00:43:09] minutes, it did track it down. It said, you know, there's a UTF-8 parsing bug here. Here's a three-line [00:43:14] fix for it. I've submitted a PR for you. And the PR was merged the following day. That's not something [00:43:21] that I would have attempted in the past. Like, maybe if I really wanted to, I could maybe have done that. [00:43:26] But, like, the overhead of dealing with the fact that it was written in Rust would probably have [00:43:30] meant I wouldn't have attempted it. But I am now empowered to do that sort of thing. So, that does [00:43:36] feel good to me, despite all the uncertainty. Now, everything we've talked about so far is about [00:43:43] making you, as a developer, have additional powers. And that's great. But there are other people in the [00:43:49] world besides software developers. And maybe you might want to bring some of these kind of powers to your [00:43:55] users as well. And obviously, it's been possible for several years to use AI within your application. [00:44:00] But a relatively new thing is being able to use the actual agent loops that come from coding [00:44:06] agents within your application as well, which brings a lot of built-in smartness that's pre-optimized. [00:44:13] So, something that we just shipped in the last couple of weeks is an SDK for Copilot. It's [00:44:19] available for Node, Python, .NET, and Go. And there are community projects for other languages as well. [00:44:24] And this allows you to use the same agentic harness that powers Copilot CLI, but within your application [00:44:31] for other features. And it brings all the tools and prompts and all the optimization that we've done. [00:44:35] So, you get something that's very smart out of the box. Let me show you a quick example of doing [00:44:40] something with that. And I'm going to start with another software-related thing, just so it's more [00:44:45] relatable. But I'll show you it's not limited to that in a minute. So, over here, I've got a .NET [00:44:50] project called Issue Sizer, which doesn't have much implementation in it right now. And I am going [00:44:56] to implement something that can estimate how difficult a particular issue is going to be to solve. And it's [00:45:03] going to use all the smartness of the Copilot coding agent, but I'll be able to control it from my own code. [00:45:09] All right, so let's start by adding an instance of Copilot client. And that's just going to use whatever [00:45:16] thing I've got installed locally with whatever credentials I'm logged in as, but you can customize [00:45:21] that if you pass options. All right, and it's going to create a new session. And then I'll show you [00:45:25] incredibly trivially, we can just make it list the files in this directory. Now, I don't have to give it [00:45:31] any special tools to be able to access the file system or whatever, because it's using the real coding [00:45:37] agent to do this. And obviously, that's already got the ability to see files around it and make [00:45:42] decisions about how to investigate them. Now, it's going to take a moment to run the first time, [00:45:47] it always does. And we're not going to see any output until it finishes. But I'll solve that in a second. [00:45:52] All right, you can see it was running inside the build output directory there. And so it's seeing an [00:45:56] executable and a bunch of DLLs and other stuff. Okay, so that's fine. But let's get a little bit more [00:46:02] information about what it's doing. And one way that I can do that is I can subscribe to some events. [00:46:07] So we've got this on callback, and it will give us an event info. And we've got about 20 different [00:46:13] types of events, they've all got nice strongly typed types to represent them. And I'm going to detect [00:46:18] if it's a tool, and I'll display the tool name. And if not, I'll just display the event name in general. [00:46:23] So if I run that again, now, we should see a little bit more information as it goes, we can see we've [00:46:27] modified the messages, there's a user message, the assistant has started some work, the assistant's [00:46:32] done some reasoning, it's listed its intent as listing directory contents. And then it's doing a [00:46:38] view call to find the files at that location. All right, that's fine. So we can see a bit more [00:46:44] information and we could build a UI that updates in a slightly more dynamic way to show what the agent's [00:46:49] doing if we want to. All right. But so far, that's just using the built in tools for the coding agent, [00:46:55] you can of course, add your own tools and capabilities to it as well. Since this is a C [00:47:00] sharp application, I'll do it using C sharp. So I'm going to define a new method here called get [00:47:05] issue size label, just to show that we can, it's going to take in a number of days work. And based [00:47:10] on that, it will use this absolutely outrageous syntax, what on earth is that to return a different [00:47:16] string, depending on what number comes in. All right, let's put a breakpoint so we can see that. And [00:47:22] then I'm going to have to make that available in my session. So here, I'm going to pass a set of tools. [00:47:29] I'm using the Microsoft extensions AI library to turn that method into something that AI can call. And [00:47:35] once I've done that, I'm just going to prompt it to actually use that. So I'm going to say, [00:47:39] what's the label for an issue that takes six days? And if I run that with the debugger now, [00:47:45] and then hopefully, it will have access to that tool, and it will choose to call it in order to [00:47:50] satisfy this particular request. All right, so you can see it's invoked my method, it's passed in six [00:47:56] as the number of days of work. And then when it gets the string that comes back, it says the label [00:48:01] is size M. All right, great. So it works. Cool. Now let's use this to implement something that actually [00:48:07] estimates the size of a piece of work. And it might seem like that would be a hard problem to solve, [00:48:12] but it's not really because coding agents have pretty much got that capability already. [00:48:17] So I could add a nice big prompt like this that says, you're a software engineer, estimate the [00:48:22] complexity of the issue. Check if the issue corresponds to source in the current directory. And if so, [00:48:27] investigate the code to provide a more accurate estimate. And it will use all the capabilities like [00:48:33] the explore subagent or whatever else in order to do that. Okay, and it's going to take this issue URL. [00:48:38] So let's run that. And it says enter your issue URL. Let's find an issue that we can do. Let's go to the [00:48:46] Copilot SDK repo. Go into the list of issues. Find something in there that looks quite big. All right, [00:48:55] this one's quite big. Connect to active CLI sessions. Okay, so if I go back over here and paste that in, [00:49:02] then it should use all of the capabilities that the coding agent already has by default. So it's [00:49:08] got the ability to, you know, like fetch stuff from the web, and deal with if it's a large output and [00:49:15] inspect that content, it's able to look at the files on disk and it can sort of reason about whether this [00:49:19] is the code corresponding to that repo, which isn't. So in this case, it's just going to skip ahead to [00:49:25] giving an estimate in general. And it's decided that it's a medium sized issue, which is probably [00:49:30] reasonable five days of work. All right, cool. So that's how you could control one of these coding [00:49:39] agents from within your application and build upon all the skills and capabilities that are already [00:49:44] available to it. But it's not limited to just software engineering type tasks. So to demonstrate that, [00:49:51] we've also created this other little demo that I'll be able to bring up over here. All right. So up here [00:49:58] in the top right of PowerPoint, you see, I've added this little extension here, GitHub Copilot. Now, [00:50:03] I just want to be clear, this is not a feature that we're actually shipping. This is not a product [00:50:07] announcement or whatever. This is just a fun little demo that we put together, where we've added, [00:50:11] we've used the SDK that I showed you there to create a sidebar, where I can type in some instruction [00:50:16] and it will use coding agent features, but it's also plugged into the Office APIs, so it can interact [00:50:23] with the document itself. So I could say something like make a five slide presentation, another text is [00:50:31] too small, let's zoom in. Make a five slide presentation about NDC London, use cool layouts and lots of London puns. [00:50:45] All right. So I'm using Haiku here, which is a small model. I'm using that just because it's fast. [00:50:52] I find in practice the layouts it produces are a bit of a mess, so it might be ugly. [00:50:55] If I used something like Opus, it would generally produce really quite good designs, but it does take [00:51:00] like a whole minute to respond and I don't want you to have to wait for that. So it's working and hopefully [00:51:06] it's going to come up with an actual answer for us in a minute, which is unusually slow right now. [00:51:15] I'm just going to start a new instance of that just in case I can make this go faster. [00:51:23] Okay, so I'm trying that again. Hopefully we will get some output from that. Okay, [00:51:31] so it's adding a slide. Code on the Thames where code meets culture. All right, let's see what else [00:51:37] it comes up with. What is NDC London? The most prestigious constants. It's not a London fog of [00:51:42] confusion. All right. Cool. Fine. Conference highlights. Great. All right. So as you can see, [00:51:48] it is working. The puns are not great. I think we need some slightly more advanced AI if we want. [00:51:57] We're not pulling your big what? Let's move away from that quickly. All right. [00:52:08] I'm deeply sorry. I do apologize. That's one of the worst things I've ever done. All right. All right. [00:52:15] Point is, you could apply the intelligence of a coding agent to tasks that might not just be coding [00:52:24] using an SDK like this. All right. So I think we're coming up to being done. And I'm sure many of you [00:52:31] are very, very keen to get out there and get some coffee. So I just want to leave you with some thoughts [00:52:35] about what this means for us in our industry right now. What are we anymore? Like, what are we trying [00:52:39] to become? I think at the end of it, it does not change the fact that we're still engineers. It gives [00:52:45] us additional tools, additional powers, and allows us to apply our talents to a wider range of things. [00:52:51] But fundamentally, we are still doing the same job, right? We're still using our brains to make decisions [00:52:55] about how to solve problems. There's still an almost infinite space of possibilities for what you can [00:53:00] actually build. And it's up to you to apply your decisions of taste and your decisions of wisdom to [00:53:07] what you're actually going to use this more powerful tooling to achieve. And hopefully, it will allow [00:53:11] you to take on problems that you wouldn't have done before. So that's my challenge to you. Get one of [00:53:16] these CLI tools, I don't care which one it is, and try doing something that you wouldn't have done before. [00:53:21] If you want to try fixing a bug in some code that you don't even speak the language of, [00:53:26] try that. If you want to try implementing a prototype for something in your work, do it. [00:53:30] If you want to try that really difficult feature that people have been saying for a year can't be [00:53:34] done because it's too hard, just see what happens if you just throw the problem at your coding agent. [00:53:40] I don't know if it's going to work, but you will learn something. And I would say that for a lot of [00:53:44] these prototyping things, you're going to be pretty delighted with what you can get in an hour or two's [00:53:49] work. All right, so I'm going to leave you with that. I hope you have a really good last day of the [00:53:55] conference. If you're around, I would love to chat to you about any of this stuff. But other than that, [00:53:59] just have a really good day and yeah, enjoy your time.