Adam Tornhill: code quality in the AI era

00:01Oren Toledano, Swimm CEO

Hello and welcome to the Swimm podcast. My guest today is Adam Tornhill, founder and CTO of CodeScene and the author of your code as crime scene and software design X-rays. What makes Adam unusual is that he comes at software from two directions. He has degrees in both engineering and psychology and that combination is pretty much the lens through which he sees everything about how code and organizations evolve together. He's one of the people who made behavioral code analysis a real field and I'm curious to dig in to how he thinks about everything. Adam, welcome.

00:40Adam Tornhill

Thank you very much and thanks for that lovely introduction.

00:44Oren Toledano, Swimm CEO

So Adam, let's let's dig in right away. What actually makes a codebase AI friendly? I know you've you've written something about it recently, like concretely. What does that look like versus one an agent can't really work in?

01:02Adam Tornhill

Yeah, that's a really important question. So, it's something we have been doing some research on and we have published papers on it and our hypothesis was that the patterns the coding patterns that negatively impact the human that makes code hard to understand also have a negative effect on coding agents. So, that's what we studied. We basically looked at can code quality predict whether an AI agent will do a good job or whether it will just amplify defect creation. And for that we used a metric called code health that we have developed that we have implemented in code scene.

And what we found that was quite dramatic was that for an a agent to do a good job and keep defect rates within reasonable limits you need to push code quality almost to a perfect 10. Hm.

01:55Oren Toledano, Swimm CEO

And once code code health starts to decline, defect rates go up and they go up quite steeply. So it's a nonlinear relationship. So what happens in that case is that or rather what you observe is that some organizations they really really get large benefits out of coding agents, right? are able to accelerate work and others they really struggle and it's very easy to default and blame the tools but it's not the agent's fault. It's the surrounding context which is your code that simply isn't up to par when when you when you say code quality what do you mean how clean is the code written?

Can you like explain a bit more about what this quality also health mark you're giving now like to code bases? Can you can you give us more about that?

02:48Adam Tornhill

Sure definitely. So when it comes to code quality in my my experience and basically the whole research community there's never a single metric that can u predict something as complicated as code quality. It's a multifaceted concept. So what we do when I talk about code health, what I mean is the the metric, not only a concept. And what we do in code health is that we sample multiple properties. Each one of them correlates to one factor that makes code hard to understand. And to give you some examples, it could be things like missing abstractions, right?

So you have lots of really complicated boolean logic in a in an expression or low cohesion. You have stuff too many business responsibilities into one module. or you have duplicated a lot of code. So factors like that

03:35Oren Toledano, Swimm CEO

We sample these and then we have algorithms that can simply convert them into a numeric range 10.0 perfectly alpha code all the way down to one which is the worst possible technical depth you can imagine. All right. so let's take it that take this approach into the real world challenges and and I know you've tested agents going into legacy code and guided what what happened well what what happens is that defect rates sore and there are plenty of pretty good studies on this and our own included of course.

What you typically observe is that you get a large increase in defects and the the typical number tends to be around 40 to 50% increase in defects without in any increase in throughput whatsoever. That basically means that code quality and quality of the product quality of the product goes downhill but you're not even getting there faster. Right? So that's one typical thing that happens and the the reason for that is of course that legacy code tends to lack abstractions and it tends to be really really hard to understand for humans.

Now a coding agent comes in and it's very hard for the agent to even bridge the gap between the the problem you're describing through a prompt versus what the code reflects. There's no natural mapping between them. Even the the search operation of an agent is complicated and when the agent is then tasked to change the code it will be more or less a hit or miss and what we see is that in if the code is unhealthy enough hallucination and error rates might be as high as 80 90%.

05:24Adam Tornhill

Wow that's incredible numbers. so we're talking now at Swimm with a lot of managers sometimes by the way it can be the CEO himself or herself and the CTO of course that are trying to bring AI into legacy systems right so they you know they want to see if you can work with or modernize it or different outputs that you could hope with AI to achieve. So in your eyes where are engineering managers most wrong right now about the legacy problem?

So when it comes to a legacy problem, I think what what most people get wrong is that they think that all code is equal and they think that the marketing promises of stuff like you know [snorts] 55% faster and whatn not

06:19Oren Toledano, Swimm CEO

Right

06:19Adam Tornhill

They think that that's universally applicable but it's not because your existing system your existing code base it's such an critical part of any agent's context and it's very easy to underestimate that perhaps because you don't know where you are today. You don't know how bad it is. So that I think is part of the challenge. That said, I'm personally I mean I've been working with trying to remediate technical debt for like 15 years now. It's basically what I built my career on and I'm quite enthusiastic about coding agents because I think it's finally the tool we need to solve this big problem of legacy code and technical depth.

But it it won't come easy. In my experience, succeeding with coding agents requires even more discipline than doing stuff manually.

07:10Oren Toledano, Swimm CEO

H that that's a quite an interesting statement. So when when you say the legacy code lacks this you know this context or this layer that enables agents to work with. what do you mean exactly? Is that the fact that it was written in old older languages or implemented in a different way or the lack of documentation or you know what is missing there? What is the missing layer?

07:43Adam Tornhill

So language itself I don't see as a as a big enough problem. What I think that the problem is about is a lack of good enough design in the code. So typically what you tend to see in in legacy systems is that the implementation is very low level. It's almost like a brute force. You tend to have really really long methods spanning hundreds, sometimes even thousand lines of code, right? And there are very few abstractions. So there's usually a lot of logic, lot of if statements, nestedly if statements, loop, but no nothing in the code that tells you that this is the actual problem that this code solves. M

08:23Oren Toledano, Swimm CEO

And that means that without having without having good abstractions like good functions, properly named classes and modules, it's going to be really really hard for a coding agent to build up any context and understand what that code actually does and that's why it's so risky to change it both for humans and machines in my opinion.

08:42Adam Tornhill

Understood. There is no clear statement of why it is doing what it is doing and nothing also about what it is actually doing. Exactly. Exactly.

08:53Oren Toledano, Swimm CEO

Okay. So,

08:53Adam Tornhill

And I mean some sometimes we can partly we can compensate for that by having really really good test right but that also tends to be the hallmark of legacy that you don't have any automated tests. Okay. So now trying to extrapolate from this problem and and making agents work in with this you know complex code bases. So what kind of context I don't know if that's the right word you like to use makes the biggest difference for an agent to be able to work in complex code bases.

09:28Oren Toledano, Swimm CEO

Yeah, that's that's an excellent question. So based on everything I've learned, I would say that code health is really really vital because if you have a healthy code, then you can also let an agent create the tests and these tests will be quite good and you can also use the agent to start to accelerate feature development. So what we see in our research is that code health needs to be at least 9.5 on a 10point scale in order to do fully autonomous feature development with agents.

So what I think is missing is the situational understanding right of which parts of your code are AI ready today because I'm pretty pretty confident that that most legacy code bases have parts of the code that are actually in quite good shape.

10:16Adam Tornhill

Mhm.

10:17Oren Toledano, Swimm CEO

And if you can identify those up front then that those are the parts we can already today start to accelerate with agents.

10:23Adam Tornhill

Great.

10:24Oren Toledano, Swimm CEO

And other parts that might not yet be ready. It's super important to be aware of that and have realistic expectations and plan accordingly.

10:33Adam Tornhill

So, so the strategy you recommend is a lowhanging fruit, right? So, like first understanding within the codebase what is better or healthier and can work out of the box with agents and then the rest you need to deal with. So, let let's talk about the rest. Let's say there is a company 12 15 year old maybe 20. It doesn't have to be a 60 year old bank, right?

and they have I don't know five to 10 million lines of code it's not green field right and they realize their codebase is is blocking them because of this code health situation what do they do how how do you help them you know retrieve this context to you know unlock agents on their legacy code

11:23Oren Toledano, Swimm CEO

Yeah so what I do in those situations is that first of all I do this pure analysis of code health just figure out where are the problems and where are the parts that are already friendly after that I start to look at the business relevance because if you have a code base of 10 million lines of code chances are that not all of it is actively maintained in fact 90% of the code tends to be quite stable so that's the parts of the code where you can maybe accept existing technical adapt you can accept the lower quality because chances are you don't really work on that part of the code.

So it's all about identifying the business critical parts of the code and there are techniques you can use to guide you on that. One of them is the hotspot concept that I talk a lot about in your code as crime scene that basically identify those parts of the code. So now when you have identified the business critical parts of the code those are the parts of the code that you want to make AI ready in case if you really want some return on investment.

And from here there are multiple technical techniques I would use and it would also depend on code health because what we also see in our research is that if you have on this 10point scale you can have quite severe quality issues as long as there are they are above say roughly 6.57 then you can actually do an agentic uplift. Mhm.

12:43Adam Tornhill

Coding agents are quite good with that if they get this information, this context and direction and the way we solve that in practice is that we provide code health through an MCP server to the agent so it gets a direction and a goal.

12:58Oren Toledano, Swimm CEO

If you're below that limit, if you're in truly unhealthy code, then you need to be quite strict with having a human in the loop to guide it.

13:06Adam Tornhill

The big difference is that you can now do these costly code transformations. You can automate them through an agent, but you need to do them super iteratively and and build up just like you would do any refactoring. The difference in my experience is that refactorings that used to take weeks, you can typically do them within a day with agents because the feedback loop is so rapid and especially in the parts that you have identified as healthier, right?

That's the places where you would start and that I guess through your technology you can also recommend your clients to start you know working with agents on this more you know healthier parts and more important parts maybe in terms of the business. very cool. so in in the prep call we had you told me you're also you you are working yourself you know using agents and you know hands-on. so I have a I have a question about you know this experience after a year of agentic development.

where do you trust the output without looking except accept except or u when do you feel you still need a human in the loop besides these areas where the context is extremely rare and and challenging?

14:25Oren Toledano, Swimm CEO

Yes. So I would say that there are two different types of tasks that I generally work on. one are like pure maintenance tasks. So to give you one example we can let's focus on the code off metric since we talked about that. So under the hood code elf is supported for I think 29 30 different programming languages which means for each one of these languages we need to you know we need to parse the languages we need to do an analysis and interpretation of it and programming languages aren't stable there are changes to them all the time so there's a new operator introduced in C++ a new I don't know a new syntax in Java or whatever

15:05Adam Tornhill

And this used to be quite time consuming but this is a part that I have found that agents are quite good at and the reason the agents are good at that is because the design in our product is there are some predictable patterns for the agents to follow. There's a very strong test suit that can serve as good examples for new tests that need to be added and that can work as validation criteras as well. So those are the parts where I can give an agent full autonomy. The parts where I have found that I need to be much more tightly involved in the loop is when I do something novel that we haven't done before.

So don't know no not really any existing context not any really not really any existing examples and that's where I need to you know take much much smaller increments and validate as we go along in general.

15:56Oren Toledano, Swimm CEO

Interesting. So as a specialist about you know code health and complexity of code bases I have this discussion with a lot of engineering managers about how fast new code will become legacy code now right because it's being written so fast with agents and at scale because everybody becomes a developer now which is another challenge that engineers will need to deal with but you've mentioned that you know in legacy systems in past there was this lack of design maybe or and the missing context. Is it better now as it's being written faster and you know at larger scale or it's just bringing the the problem faster than it it used to to be?

I would definitely say that AI really just accelerates existing trends and existing outcomes. So we get to bad code faster.

16:54Adam Tornhill

Interesting. And that's to me to me it's it's really really dangerous. I've seen so many companies that nly adopt AI and it's easy to you know it's easy to do a a proof of concept right so even if you do it u by the book and say okay let's evaluate this with a small team and you might even look at benchmarks and whatnot the problem with almost all these tasks is that there are oneoff tasks and that's not how software development works right with software development 95% of the costs come after the first release of the product

17:26Oren Toledano, Swimm CEO

So that's the maintenance phase And this is where agents need so much more guidance. And this is where engineering discipline is more important than ever. So it's so in in my take it's it's not only about the new tools, it's also about the reinforcement of the fundamentals of software design and the fundamentals of software engineering. It's more important than ever.

17:48Adam Tornhill

So okay, let's talk about this discipline element because that's something that is like less sexy than running with agents and building, right? and so I think the models are super exciting nowadays and I think we're past the model choice right that the main variable but what does the right so if we're talking about the the discipline what is the right infrastructure around a model that what does it look like what engineering managers need to be aware if they are starting to release products that were fully written by agents right and they're going to hit quite quickly as you mentioned the maintenance challenge.

what do they need to prepare for? So I would say that there are a couple of things. The first is to have safeguards and automated quality gates at multiple levels. So what what we do internally is that we have a bunch of safeguards that run already as the code is being written. Right? So these safeguards are implemented via MCP servers and CLI tools. So they give feedback to the AI as the code is being written. The other thing we insist on is to have good tests. So we we've always known that test automation is a good thing. Everything we can check automatically we should

19:12Oren Toledano, Swimm CEO

But we never really cared about about the extent of it to as we have to do now. So what I found is that we have to get almost to 100% code coverage for everything we do in order to be able to trust it. And that that's used to be quite extreme but now it's a must. And then of course rapid feedback loops are really really important. It doesn't help if you even if you code twice as quick and then you have to wait two hours for a build pipeline. It's an extremely long feedback loop. To me the big open questions in this space when it comes to safeguarding is around software architecture and design.

19:52Adam Tornhill

So that's where things also can deteriorate and that's a space where I'm not aware of any good automation or tools today but I think it's the the next frontier.

20:02Oren Toledano, Swimm CEO

So it's still a heavily human in the loop in terms of the design. That's what you think.

20:08Adam Tornhill

Yeah, I think so. And again a good design can serve as a guardrail in itself, right? Because it sets an example. It establishes a pattern that they can follow. But we we need to get that right. So what what I tend to do is when I do something novel, I always spend much much more time up front on it than I used to do

20:28Oren Toledano, Swimm CEO

Because only so can I accelerate later. So the load of the work changed a little bit from working on the output and it goes upstream you would say to the design and preparation before sending your agents do the work.

20:46Adam Tornhill

Yeah, I would say so. And I also tend to be much more strict in how I work, right? So I I found that some things I used to do might no longer be meaningful. For example, I no longer do test re development in the sense I used to, but I do insist on having end to end tests written before any code whatsoever is produced because that's how I validate that I have managed to specify the right thing to the agent.

21:13Oren Toledano, Swimm CEO

Okay. So that's the Spider-Man with great powers comes great responsibilities I guess.

21:19Adam Tornhill

Oh, a bit like that. Yes.

21:21Oren Toledano, Swimm CEO

Okay. so as I mentioned in my introduction you come at this from psychology also as much as engineering right. can you tell us a little bit how you see the world and what does psychology actually explain about code bases that you know pure technical approach misses. So I think what psychology brought me was on a better understanding for how we people think, reason and solve problems and it has influenced a lot of the stuff I've done. So the whole hotspot concept it's a behavioral analysis technique that one I actually picked out of forensic psychology. So it's based on a method for tracking serial offenders. Right.

22:06Adam Tornhill

Yeah. In our case, we're tracking code that is being modified frequently. But it has also in influenced how I view code at a deeper level. So the human brain as fascinating as it is is also quite limited. We have so many cognitive bottlenecks, things like working memory. How many things can we hold in our head at once? And that has definitely influenced how I approach software design. So I think a good design is a design that works with the human brain rather than constantly trying to fight it.

22:38Oren Toledano, Swimm CEO

And if we have followed those recommendation then that pays off now because coding agents they I mean they are trained on humanly written code right even synthetic training data looks very much like humanly written code.

22:52Adam Tornhill

So that means that they are sensitive to exactly the same problems as we humans are.

22:57Oren Toledano, Swimm CEO

So I think psychology is more important than ever. And then of course now with coding agents I reflect a lot about how how does it actually impact us as human developers this massive massive shift that happened over the past year.

23:11Adam Tornhill

Can can a coding agent can also be your psychologist or how does it work?

23:18Oren Toledano, Swimm CEO

Oh [laughter]

23:19Adam Tornhill

I'm not sure about that. It's it's interesting with agents. I I have to say that I never had as fun writing code as I have now with agents because I can take on larger problems and I can get there faster. But I also find find that it's quite draining. So I used to be able to code for a whole day. There's no way I can do that in aentic work. You know after two hours I'm completely fatigued mentally.

23:43Oren Toledano, Swimm CEO

Yeah.

23:44Adam Tornhill

Because uh

23:45Oren Toledano, Swimm CEO

What an agent does is basically it compresses the amount of decisions you need to do into a very very short time span. And that is that is draining,

23:55Adam Tornhill

Right? It's it's rubber duck debugging on steroids on steroids, right?

24:00Oren Toledano, Swimm CEO

Yeah. Yeah. It's a bit like that. Then you have to pay attention constantly and then you also have these you know you have these interruptions that that I think is a big problem with theic workflows today that you you know you instruct the agent hey do this and you never know how long the agent is going to need but you assume that yeah it'll probably be a minute so you go to LinkedIn or Rex or whatever or Reddit and now you lost 10 minutes surfing the internet right so it's and you start to multitask So that I think is the problem right now that agents are like on open invitation to multitasking and lost focus.

24:40Adam Tornhill

You lo you lose the zone like experience that developers used to have right with the

24:45Oren Toledano, Swimm CEO

Headset. Exactly.

24:46Adam Tornhill

Because there's so much wait time in the loop today.

24:49Oren Toledano, Swimm CEO

Mhm. Understood. Interesting. okay, last question. you're one of the founders or the founder of CodeScene, right? And if you were to build it today from scratch with AI, what would you do differently?

25:05Adam Tornhill

What would I do differently? I mean there we always been good at coding focusing on engineering discipline. But what I would have done would that I would have turned that discipline from 10 to 11 on day one. M

25:20Oren Toledano, Swimm CEO

And when it comes to it comes to feature set, there are probably features that I wouldn't have considered because they might no longer be relevant in the AI era, but the fundamentals are more relevant than ever. So, I would have doubled down on those.

25:35Adam Tornhill

Yeah, that's interesting. I think a lot of companies either their business went you know down because it's not relevant anymore or some the the essence of the business is even truer now it's more important. We started with documentation and suddenly it becomes like a must rather than something that people you know put aside. Adam thank you so much for for joining the podcast.

I learned a lot and you know I think your unique approach and experience and also the way you're articulating your experiences and how you encounter these challenges I think it's it's humble and interesting and and helpful I think for many managers that will see that hear this podcast and you know think about how do I build the right discipline into my you know my team in my in in my workflows in my infrastructure and we've I think we've passed the the hype right and now there is room for a more structured serious look at how we are going to deliver the software of tomorrow.

So thank you for your input and for the learnings and hopefully we can see you soon again.

26:57Oren Toledano, Swimm CEO

Thank you so much and thanks for having me.

27:00Adam Tornhill

Pleasure.