The Swimm Podcast
Understanding Code Quality and AI Compatibility with Adam Tornhill
Episode 04
Notes
In this episode
- Code Quality and AI Compatibility: Adam explains that "CodeHealth"—a multi-faceted metric developed at CodeScene—is a key predictor of how well AI agents will perform. High-quality, well-designed code (approaching a 10/10 rating) is necessary for agents to function effectively without significantly increasing defect rates (0:01:35 - 0:02:27).
- The Risks of AI in Legacy Code: Using AI agents on legacy systems with low code health often leads to a sharp increase in defects (40-50%) without improving development speed. The lack of clear abstractions and documentation makes it difficult for agents to grasp the context (0:04:03 - 0:05:24).
- Strategic AI Adoption: Instead of applying AI blindly, engineering teams should identify the "business-critical" parts of their codebase—often the ones that are most frequently modified—and focus on improving the quality of those specific areas first (0:11:35 - 0:12:19).
- Engineering Discipline: AI does not replace the need for strong software engineering foundations; it actually makes them more vital. As AI accelerates development, it can also accelerate the creation of "bad code" if not managed with guardrails, automated testing (ideally nearing 100% coverage), and human-in-the-loop workflows (0:16:44 - 0:17:46).
- Psychology in Software Design: Adam highlights that because human developers and AI models (trained on human code) share similar cognitive bottlenecks, good code design must prioritize human-understandable patterns. He notes that while AI makes coding more productive, the rapid decision-making required by agents can lead to mental fatigue and a loss of focus (0:21:43 - 0:24:49).
Chapter timestamps will appear here when the video provides them.
Transcript Show Hide
<?php //Remove minutes and second lines ?>
0:01 Hello and welcome to the swim podcast.
0:03 My guest today is Adam Tornhill, founder and CTO of Code Scene and the author of your code as crime scene and software
0:11 design X-rays. What makes Adam unusual is that he comes at software from two directions. He has degrees in both
0:19 engineering and psychology and that combination is pretty much the lens through which he sees everything about how code and organizations evolve
0:28 together. He’s one of the people who made behavioral code analysis a real field and I’m curious to dig in to how
0:36 he thinks about everything. Adam, welcome.
0:40 Thank you very much and thanks for that lovely introduction.
0:44 Um, so Adam, let’s uh let’s dig in right away. What actually makes a codebase AI friendly? I know you’ve uh you’ve
0:52 written something about it recently, like concretely. What does that look like versus one an agent can’t really work in?
1:02 Yeah, that’s a really important question. So, it’s something we have been doing some research on and we have published uh papers on it and our
1:10 hypothesis was that um the patterns the coding patterns that negatively impact the human that makes
1:18 code hard to understand also have a negative effect on coding agents. So, that’s what we studied. We basically
1:25 looked at can code quality predict whether an AI agent will do a good job or whether it will just amplify defect creation.
1:35 And uh for that we used a metric called code health that we have developed that we have um implemented in code scene.
1:42 And what we found that was quite dramatic was that for an a agent to do a good job and keep defect rates within
1:49 reasonable limits you need to push code quality almost to a perfect 10. Hm.
1:55 And once code code health starts to decline, defect rates go up and they go up quite steeply. So it’s a nonlinear
2:02 relationship. So what happens in that case is that or rather what you observe is that some organizations they really
2:11 really get large benefits out of coding agents, right? are able to accelerate work and others they really struggle and
2:19 uh it’s very easy to default and blame the tools but it’s not the agent’s fault. It’s the surrounding context which is your code that simply isn’t up
2:27 to par when when you when you say code quality what do you mean how clean is the code written? Can you like explain a bit more
2:36 about what this quality uh also health uh mark you’re giving now like to code
2:44 bases? Can you can you give us more about that?
2:48 Sure definitely. So when it comes to code quality in my my experience and basically the whole research community
2:54 there’s never a single metric that can u predict something as complicated as code quality. It’s a multifaceted concept. So
3:02 what we do when I talk about code health, what I mean is the the metric, not only a concept. And what we do in code health is that we sample multiple
3:11 properties. Each one of them correlates to one factor that makes code hard to understand. And to give you some examples, it could be things like
3:19 missing abstractions, right? So you have uh lots of really complicated boolean logic in a in an expression or low
3:27 cohesion. You have stuff too many business responsibilities into one module. or you have uh duplicated a lot of code. So factors like that
3:35 we sample these and then we have algorithms that can simply convert them into a numeric range 10.0 perfectly alpha code all the way down to one which
3:44 is the worst possible technical depth you can imagine.
3:48 All right. Uh so let’s take it that take this approach into the real world challenges and and I know you’ve tested
3:56 agents going into legacy code and guided what what happened
4:03 well what what happens is that uh defect rates sore and there are uh plenty of uh
4:10 pretty good studies on this and our own included of course. What you typically observe is that you get a large increase
4:18 in defects and the the typical number tends to be around 40 to 50% increase in defects without in any increase in throughput whatsoever.
4:28 That basically means that code quality and quality of the product quality of the product goes downhill but um you’re
4:37 not even getting there faster. Right? So that’s uh one typical thing that happens and the the reason for that is of course
4:44 that legacy code tends to lack abstractions and it tends
4:51 to be really really hard to understand for humans. Now a coding agent comes in and it’s very hard for the agent to even
4:58 bridge the gap between the the problem you’re describing through a prompt versus what the code reflects. There’s no natural mapping between them. Even
5:06 the the search operation of an agent is complicated and when the agent is then tasked to change the code it will be
5:14 more or less a hit or miss and what we see is that in if the code is unhealthy enough hallucination and error rates might be as high as uh 80 90%.
5:24 Wow that’s incredible numbers. Uh so we’re talking now at swim with a lot of uh managers sometimes by the way it can
5:33 be the CEO himself or herself and the CTO of course uh that are trying to uh
5:41 bring AI into legacy systems right so they you know they want to see if you can work with or modernize it or
5:50 different uh outputs that you could hope with AI to achieve. So in your eyes where are engineering managers most
5:59 wrong right now about the legacy problem?
6:04 So when it comes to a legacy problem, I think what what most people get wrong is that they think that all code is equal and they
6:12 think that um the marketing promises of stuff like you know [snorts] 55% faster and whatn not right they think that that’s universally
6:21 applicable but it’s not because uh your existing system your existing code base it’s such an critical part of any
6:29 agent’s uh context and it’s very easy to underestimate that perhaps because you don’t know where you are today. You
6:36 don’t know how bad it is. So that I think is part of the challenge. That said, I’m personally I mean I’ve been
6:43 working with uh trying to remediate technical debt for like 15 years now.
6:48 It’s basically what I built my career on and uh I’m quite enthusiastic about coding agents because I think it’s
6:55 finally the tool we need to uh solve this big problem of uh legacy code and technical depth. But it it won’t come easy. In my experience, uh succeeding with coding agents requires even more discipline than doing stuff manually.
7:10 H that that’s a quite an interesting statement. So when when you say the legacy code lacks um this you know this
7:19 context or this uh layer that enables agents to work with. Uh what do you mean
7:27 exactly? Is that the fact that it was written in old older languages or implemented in a different way or the
7:36 lack of documentation or you know what is missing there? What is the missing layer?
7:43 So language itself I don’t see as a as a big enough problem. What I think that the problem is about is a lack of uh good enough design in the code.
7:54 So typically what you tend to see in in legacy systems is that the implementation is very low level. It’s almost like a brute force. You tend to
8:03 have really really long methods spanning hundreds, sometimes even thousand lines of code, right? And there are very few abstractions. So there’s usually a lot
8:12 of um logic, lot of if statements, nestedly if statements, loop, but no nothing in the code that tells you that
8:19 this is the actual problem that this code solves. M and that means that without having without having good abstractions like uh
8:27 good functions, properly named classes and modules, it’s going to be really really hard for a coding agent to build up any context and understand what that
8:36 code actually does and that’s why it’s so risky to change it both for humans and machines in my opinion.
8:42 Understood. There is no clear statement of why it is doing what it is doing and nothing also about what it is actually doing. Exactly. Exactly.
8:53 Okay. So, and I mean some sometimes uh we can partly we can compensate for that by having really really good test right but
9:01 uh that also tends to be the hallmark of legacy that you don’t have any automated tests. Okay. So now trying to
9:09 extrapolate from this problem and and making agents work in with this you know complex code bases. So what kind of
9:17 context I don’t know if that’s the right word you like to use makes the biggest difference for an agent to be able to work in complex code bases.
9:28 Yeah, that’s that’s an excellent question. So uh based on everything I’ve learned, I would say that uh code health is really
9:37 really vital because if you have a healthy code, then you can also let an agent create the tests and these tests
9:43 will be quite good and you can also use the agent to start to accelerate feature development. So what we see in our research is that code health needs to be
9:52 at least 9.5 on a 10point scale in order to do fully autonomous feature development with agents. So what I think
10:00 is missing is the situational understanding right of which parts of your code are AI ready
10:09 today because I’m pretty pretty confident that that most legacy code bases have parts of the code that are actually in quite good shape. Mhm.
10:17 And if you can identify those up front then that those are the parts we can already today start to accelerate with agents. Great.
10:24 And other parts that might not yet be ready. It’s super important to be aware of that and have realistic expectations and plan accordingly.
10:33 So, so the strategy you recommend is a lowhanging fruit, right? So, like first understanding within the codebase what
10:40 is better or healthier and can work out of the box uh with agents and then the rest you need to deal with. So, let
10:48 let’s talk about the rest. Let’s say there is a company 12 15 year old maybe 20. It doesn’t have to be a 60 year old
10:55 bank, right? uh and they have I don’t know five to 10 million lines of code um
11:03 it’s not green field right and they realize their codebase is is blocking them because of this code health
11:09 situation what do they do how how do you help them you know retrieve uh this context to you know unlock
11:19 agents on their legacy code yeah so what I do in those situations is
11:26 that uh first of all I do this pure analysis of code health just figure out uh where are the problems and where are the parts that are already friendly
11:35 after that I start to look at the business relevance because if you have a code base of 10 million lines of code chances are that not all of it is
11:42 actively maintained in fact 90% of the code tends to be quite stable so that’s the parts of the code where you can
11:51 maybe accept existing technical adapt you can accept the lower quality because chances are you don’t really work on that part of the code. So it’s all about
11:58 identifying the business critical parts of the code and there are techniques you can use to guide you on that. One of them is the hotspot concept that I talk
12:06 a lot about in your code as crime scene that basically identify those parts of the code. So now when you have identified the business critical parts
12:14 of the code those are the parts of the code that you want to make AI ready in case if you really want some return on investment.
12:21 And from here there are multiple technical techniques I would use and it would also depend on code health
12:29 because what we also see in our research is that if you have on this 10point scale you can have quite severe quality issues as long as there are they are
12:37 above say roughly 6.57 then you can actually do an agentic uplift. Mhm.
12:43 Coding agents are quite good with that if they get this information, this context and um direction and uh the way
12:51 we solve that in practice is that we provide code health through an MCP server to the agent so it gets a direction and a goal.
12:58 If you’re below that limit, if you’re in truly unhealthy code, then you need to be quite strict with having a human in the loop to guide it.
13:06 The big difference is that you can now do these costly code transformations.
13:10 You can automate them through an agent, but you need to do them super iteratively and and build up just like you would do any refactoring. The
13:18 difference in my experience is that refactorings that used to take weeks, you can typically do them within a day with agents because the feedback loop is
13:27 so rapid and especially in the parts that you have identified as healthier, right?
13:34 That’s the places where you would start and that I guess through your technology you can also recommend your clients to
13:41 start you know working with agents on this more you know healthier parts and more important parts maybe in terms of
13:47 the business. Uh very cool. Um so in in the prep call we had uh you told me
13:54 you’re also you you are uh working yourself you know using agents and you know uh hands-on. Um so I have a I have
14:04 a question about you know this experience after a year of agentic development. Um where do you trust the
14:10 output without looking except accept except or u when do you feel you still need a human in the loop besides these
14:19 areas where the context is extremely rare and and challenging?
14:25 Yes. So I would say that uh there are two different types of tasks that I generally work on. one are like pure
14:32 maintenance tasks. So to give you one example uh we can let’s focus on the code off metric since we talked about that. So under the hood code elf is
14:41 supported for I think 29 30 different programming languages which means for each one of these languages we need to
14:48 you know we need to parse the languages we need to do an analysis and interpretation of it and programming languages aren’t stable there are
14:56
14:56 changes to them all the time so there’s a new operator introduced in C++ a new I don’t know a new syntax in Java or
15:03 whatever and this used to be quite time consuming but this is a part that I have found that agents are quite good at and the
15:11 reason the agents are good at that is because um the design in our product is there are some predictable patterns for
15:18 the agents to follow. There’s a very strong test suit that can serve as good examples for new tests that need to be added and that can work as validation
15:26 criteria as well. So those are the parts where I can give an agent full autonomy.
15:32 The parts where I have found that I need to be much more tightly involved in the loop is when I do something novel that
15:39 we haven’t done before. So don’t know no not really any existing context not any
15:45 really not really any existing examples and that’s where I need to you know take much much smaller increments and validate as we go along in general.
15:56 Interesting. So as a specialist about you know code health and complexity of code bases I have this discussion with a
16:04 lot of engineering managers about how fast new code will become legacy code now right because it’s being written so
16:13 fast uh with agents and at scale because everybody becomes a developer now uh which is another challenge that
16:20 engineers uh will need to deal with uh but you’ve mentioned that you know in legacy systems in past there was this
16:28 lack of design maybe or and the missing context. Is it better now as it’s being written faster and you know at larger
16:37 scale or it’s just bringing the the problem faster than it it used to to be?
16:44 Uh I would definitely say that uh AI really just accelerates existing uh trends and existing outcomes. So we get to bad code uh faster.
16:54 Interesting. And uh that’s to me to me it’s it’s really really dangerous. I’ve seen so many companies that um nly adopt
17:01 AI and it’s easy to you know it’s easy to do a a proof of concept right so even if you do it u by the book and say okay
17:09 let’s evaluate this with a small team and uh you might even look at benchmarks and whatnot the problem with almost all these tasks is that there are oneoff
17:17 tasks and that’s not how software development works right with software development 95% of the costs come after the first release of the product
17:26 so that’s the maintenance phase And this is where agents need so much more guidance. And this is where engineering discipline is more important than ever.
17:35 So it’s uh so in in my take it’s it’s not only about the new tools, it’s also about the reinforcement of the
17:42 fundamentals of software design and the fundamentals of software engineering. It’s more important than ever.
17:48 So okay, let’s talk about this discipline element because that’s something that is like less sexy than running with agents and building, right?
17:56 uh and so I think the models are super uh exciting nowadays and I think we’re past the model choice right that the
18:04 main variable uh but what does the right so if we’re talking about the uh the discipline what is the right
18:12 infrastructure around a model that what does it look like uh what engineering managers need to be aware if they are
18:20 starting to release products that were fully written by agents right and they’re going to hit quite quickly as you mentioned the maintenance challenge.
18:31 Uh what do they need to prepare for?
18:36 So uh I would say that there are a couple of things. The first is to have uh safeguards and automated quality gates at multiple levels.
18:46 So what what we do internally is that we have a bunch of safeguards that run already as the code is being written.
18:52 Right? So these safeguards are implemented via MCP servers and CLI tools. So they give feedback to the AI
18:59 as the code is being written. The other thing we insist on is to have uh good tests. So we we’ve always known that test automation is a good thing.
19:09 Everything we can check automatically we should but we never really cared about about the extent of it to as we have to
19:18 do now. So what I found is that we have to get almost to 100% code coverage for everything we do in order to be able to
19:25 trust it. And that that’s used to be quite extreme but now it’s a must. And then of course uh rapid feedback loops are really really important. It doesn’t
19:34 help if you even if you code twice as quick and then you have to wait uh two hours for a build pipeline. It’s an extremely long feedback loop.
19:43 To me the big uh open questions in this space when it comes to safeguarding is uh around software architecture and uh design.
19:52 So that’s where things also can deteriorate and that’s a space where I’m not aware of any good uh automation or tools uh today but I think it’s the the next frontier.
20:02 So it’s still a heavily human in the loop in terms of the design. That’s what you think.
20:08 Yeah, I think so. And uh again a good design can serve as a guardrail in itself, right? Because it sets an
20:15 example. It establishes a pattern that they can follow. But we we need to get that right. So what what I tend to do is
20:23 when I do something novel, I always spend much much more time up front on it than I used to do because only so can I accelerate later.
20:31 20:31 So the load of the work changed a little bit from working on the output and it goes uh upstream you would say to the
20:40 design and preparation before sending your agents do the work.
20:46 Yeah, I would say so. And I also um tend to be much more strict in how I work, right? So I I found that some things I
20:54 used to do might no longer be meaningful. For example, I no longer do test re development in the sense I used to, but I do insist on having end to end
21:04 tests written before any code whatsoever is produced because that’s how I validate that I have managed to specify the right thing to the agent.
21:13 Okay. So that’s the Spider-Man with great powers comes great responsibilities I guess. Oh, a bit like that. Yes.
21:21 Okay. Um so as I mentioned in my introduction you come at this from psychology also as much as engineering
21:29 Right. Um can you tell us a little bit how you see the world and what does psychology actually explain about code
21:37 bases that you know pure technical approach misses.
21:43 So I think what psychology brought me was on uh a better understanding for how we people uh think, reason and solve problems and it has influenced a lot of
21:52 the stuff I’ve done. So the whole uh hotspot concept it’s a behavioral analysis technique that one I actually
22:00 picked out of forensic psychology. So it’s based on a method for tracking uh serial offenders. Right.
22:06 Yeah. In our case, we’re tracking code that is being modified frequently. But it has also in um influenced how I view
22:14 code at a deeper level. So um the human brain as fascinating as it is is also quite limited. We have so many cognitive bottlenecks, things like working memory.
22:25 How many things can we hold in our head at once? And uh that has definitely influenced how I approach software design. So I think a good design is a
22:33 design that works with the human brain rather than constantly trying to fight it.
22:38 And if we have followed those recommendation then that pays off now because coding agents they I mean they are trained on humanly written code
22:46 right even synthetic uh training data looks very much like humanly written code.
22:52 So that means that they are sensitive to exactly the same problems as we humans are.
22:57 So I think psychology is more important than ever. And then of course now with coding agents I reflect a lot about how how does it actually impact us as human
23:07 developers this massive massive shift that happened over the past year.
23:11 Can can a coding agent can also be your psychologist or how does it work?
23:18 Oh [laughter] I’m not sure about that. It’s it’s interesting with agents. I I have to say that I never had as fun writing code as
23:26 I have now with agents because I can take on larger problems and I can get there faster. But I also find find that
23:33 it’s quite draining. So I used to be able to code for a whole day. There’s no way I can do that in aentic work. You know after two hours I’m completely fatigued mentally.
23:43 Yeah.
23:44 Because uh what an agent does is basically it compresses the amount of decisions you need to do into a very very short time
23:51 And that is uh that is draining, right? It’s it’s uh rubber duck debugging on steroids on steroids, right?
24:00 Yeah. Yeah. It’s a bit like that. Then you have to pay attention constantly and then you also have these um you know you
24:07 have these interruptions that uh that I think is a big problem with theic workflows today that you you know you instruct the agent hey do this and you
24:16 never know how long the agent is going to need but you assume that yeah it’ll probably be a minute so you go to uh LinkedIn or Rex or whatever or Reddit
24:25 and now you lost uh 10 minutes uh surfing the internet right so it’s and you start to multitask
and now you lost uh 10 minutes uh surfing the internet right so it’s and you start to multitask
24:32 So that I think is the problem right now that agents are like on open invitation to multitasking and lost focus.
24:40 You lose the zone like experience that developers used to have right with the headset. Exactly.
24:46 Because there’s so much wait time in the loop today.
24:49 Mhm. Understood. Interesting. Uh okay, last question. Um you’re one of the founders or the founder of Code Scene,
24:56 right? And if you were to build it today from scratch with AI, uh what would you do differently?
25:05 What would I do differently? I mean there we always been good at coding focusing on engineering discipline. But
25:13 what I would have done would that I would have turned that discipline from 10 to 11 on day one. M and when it comes to it comes to feature
25:22 set, there are probably features that I uh wouldn’t have considered because they might no longer be relevant in the AI
25:28 era, but the fundamentals are more relevant than ever. So, I would have doubled down on those.
25:35 Yeah, that’s interesting. I think uh a lot of companies either their business went you know down because it’s not
25:43 relevant anymore or some the the essence of the business is even truer now it’s more important. We started with
25:51 documentation and suddenly it becomes like a must rather than something that people you know put aside. Um Adam thank you so much for for joining the podcast.
26:02 I learned a lot and you know I think uh your unique approach and uh experience and also the way you’re uh articulating
26:12 um your experiences and how you encounter these challenges I think it’s it’s uh humble and interesting and uh
26:20 and uh helpful I think for many managers that will um see that hear this podcast and you know think about how do I build
26:30 the right discipline into my you know my team in my in in my workflows in my infrastructure um and we’ve I think we’ve passed the
26:38 the hype right and now there is uh room for a more structured serious uh look at
26:47 how we are going to deliver the software of tomorrow. So thank you for your input and uh for the learnings and hopefully we can uh see you soon again.
26:57 Thank you so much and thanks for having me. Pleasure
From the episode
▶ Watch full episode on YouTube
When we talk about making a codebase AI-friendly, we must first understand what constitutes code quality. Adam emphasizes that code quality is not a single metric but a multifaceted concept that requires a comprehensive assessment. He and his team at CodeScene have developed a metric they call “CodeHealth,” which quantifies code quality based on various properties that make code challenging to understand.