Cameron Barre on why Relying on Specs Alone Won’t Prevent AI-Induced Code Failures.

0:00Hi Cameron. Hey Orin. How’s it going?
0:04Welcome to the swim podcast. Cameron Beret. Um, nice to meet you. Uh, and nice to have you here. Uh, we’re going
0:12to talk about all kind of aspects related to AI for development. Uh, but I want to know first your uh, background
0:21story. I will ask you through you know what brought us here and uh in our uh prep discussion uh you told me a lot
0:29about information systems you’ve been thinking about them for a while and why why the decay long before AI coding was
0:37a thing right and so my question is what kept you you know back to that problem what is the the origin story of Cameron?
0:47Yeah. Uh, thanks for having me. And as far as or origin story goes, you know, I’m just a working engineer. I’ve been
0:56working in the industry since around 2013. And, um, I have to work to survive, it turns out. And when you have
1:03to work, you got to work at different companies. And as you work at different companies, you find out that every company you go to has a different way of
1:11doing things. And depending on how they choose to go about that, um, it really has an influence on how successful their
1:20projects are or how likely they are to fail. And as someone who wants to stand out professionally, I’ve always been
1:27well, I developed this mindset of like, how can I mitigate execution risk on projects that I’m responsible for? And I think if you follow that thread long
1:36enough in software, you realize that the design of systems really does have an outsized influence on how much uh execution risk there is in any given
1:44initiative. And so that’s just the thread that I always come back to is how can I make sure that I’m successful or my team is successful or whoever is
1:52responsible for getting this thing done that also has to live for many many years, right? uh how can we be how can we make sure this is a success and we don’t have to rewrite it in five years
2:01and you know we can always understand how it works and things like that makes a lot of sense so uh back to the
2:08AI story right when AI coding tools started becoming a thing becoming real what did you see happening that most
2:17people you think we’re missing well so I’ve been uh I’ve kind of been messing around with uh the sort of
2:25modern iteration of AI since around like GPT2 like before chat GPT was even a thing you had to go into the playground and mess with it there. Um, and I was
2:34kind of building prototypes and stuff and just over the years, uh, it became very clear to me that the way that they work is has a lot to do with all of the
2:43code that they’ve seen in their training data and they’ve been trained on all of the code on the internet and all of that code is very reflective of how coding
2:51works in industry and there’s a lot of variance in that. And so I just noticed I think uh the classic thing that
3:00everyone’s noticing right now is like oh I can put I can get something stood up really fast. Um but after a couple weeks uh I can’t really progress any further
3:08with it cuz there’s so much variance dayto-day and how the coding agent chooses to actually build what you’re asking that eventually it all kind of
3:16falls apart and that’s where we arrive at this kind of slop designation. And I just realized that because of the thread
3:24I’ve been following in my career, the answer is like uh uniformity. So if we can get these things to behave in a uniform way, then there’s no reason we
3:31shouldn’t be able to keep uh keep on the paved road and keep moving fast. And I think that currently a lot of people are
3:39missing that just because they’re catching up on how to even think about the technology.
3:44Okay, we will get there to the uniformity aspect. I think that’s the most interesting part of the discussion.
3:50uh before you you just explained how you know models were trained on a lot of existing code and uh you told me in in
3:58the call we had previously that AI accelerates bad habits right if your the foundation is wrong um can you give me a
4:06concrete example of what that actually looks like you know when the model does something that is you know fundamentally
4:14wrong yeah so it doesn’t seem like this is going to change anytime soon. These things are
4:23really good at pattern matching and they pattern match on the environment that they’re in. And so if you build
4:32code bases that uh use lots of complicated object-oriented design patterns or you use code bases that have
4:40hundreds and hundreds of relational database tables and every time you make a change, you have to do dangerous uh schema migrations. Um, and maybe you
4:50have a system that’s been around for a long time and has certain hot spots in the code that have really bad patterns,
4:56but everyone’s moved to like new better ones over here, but even that changes over time. It really gives a pattern
5:04matching tool nothing to latch on to and everything to sort of like have to average in between and kind of uh it it
5:11waters down the effectiveness of how these tools work. um because there’s no uniform and coherent way to do things.
5:19And so what that looks like um is really just an if if you have a problem like you don’t like something about your
5:26legacy code base uh just imagine that problem times 10 as you’re trying to move uh fast and do the work of multiple
5:33people and do a lot of work in uh one day that you could you know weeks worth of work. But you’re piling up these pull requests that everyone has to look at
5:42that’s full of code that you have to inspect. not because you want to make sure that the product and whatnot works right, but you’re trying to figure out like is it applying like the right
5:49architecture here? Um I don’t like the way it’s doing that. Uh just so just concretely it just tends to magnify and
5:57replicate existing bad habits. Um and that that’s what I’m trying to avoid basically.
6:03Sure. Um so I think like over time and definitely in the past I would say six months there is a lot of discussion
6:11about structuring the work right and making it like not only a vibe coding experience but something you can bring
6:19to production and there there is a lot of discussion around spec you know driven development and most engineers
6:27now thinks the better specs the more detailed instructions better prompts will solve that. So, but I understand
6:34from you that you don’t fully buy that direction. Can you explain why?
6:39Yeah. Um, I think spec driven development is great. Better prompting is great. All these things are great
6:46because they’re one side of a two-sided coin.
6:50Unfortunately, they’re not the side of the co. It’s the side of the coin that you can live without. Um, and the reason is, uh, I think it’s it’s pretty easy to
6:58sum up in we’ve heard stories about people’s production databases getting
7:04dropped by some, uh, coding agent. And I have no doubt that they gave that coding agent instructions not to do anything
7:13destructive against the production database. But instructions aren’t enough to actually structurally limit um
7:21software like these agents that that make decisions based on probabilities and statistical outcomes. And so without
7:29something to really constrain them, they’ll do anything even if you’ve uh given them good instructions or good
7:36specs. And it gets worse over time because there’s variance over time. And so if you have a system that lives for 5
7:4310 years, the way even if your spec is good and coherent, the way that things actually play out in the code is going to keep changing. And that’s why specs
7:53and spectrum and development aren’t enough to solve this problem. They’re they’re part of the solution to a bigger problem, but they’re not enough to give
8:01uniformity to the underlying architecture of longtime horizon projects.
8:07Okay. So, we’re we’re going to get to what you built and and how it it you know, you think it solves the the part
8:15of the problem or most of the problem and and when we talked you talked about um that you need a bottom for decision
8:23making space and you know um can you tell more about that and what does a drift look like without it? you know
8:32we’ve talked about giving specs how the code evolves with the specs but eventually this drift happens uh so can
8:41you give more color why it’s happening and yeah uh so
8:48when I use the term bottom or floor it’s to imply something that uh you can’t go beneath the floor easily right it’s it’s
8:58difficult if you’re standing unless you have a shovel or a jackhammer or something or an excavator like you’re the floor is the floor. So what you have
9:05there is what you have. Um and how that relates to decision-m space is kind of like I’ve already mentioned how human
9:14teams of programmers at different companies have so much variance in how they choose what good looks like for software architecture. It’s the same
9:22kind of thing for AI because it’s been trained on all of that. And so it has no uh it has no way to keep itself
9:30on a consistent path because it has no taste of its own. It has no life experience of it its own. It does not solve problems in the way that we solve
9:38problems and learn from them. So you really have to give it just a constrained kind of grammar
9:46to work with. Um and when you do that that prevents all of that kind of architectural drift. it doesn’t prevent product and domain level drift. That’s
9:54actually where specs come back into play. Now, if you’ve prevented drift at your architecture level, um you have more leeway to work with specs to
10:02prevent drift at your product and domain level.
10:06So, in a way, the floor is ensuring the code quality when the spec is ensuring
10:14the product quality in a way. In a way it’s uh it’s a little higher level than that in the sense that uh the floor
10:22creates a space that limits the blast radius of code quality in the small like individual function bodies and things
10:29like that. It constrains the sort of connective tissue of how the entire application is put together and then
10:37that way some of the code quality issues are someone else’s problem for another day. they because they exist kind of in the small and they don’t build up big enough to cause real problems in a system like this.
10:49Okay. Now coming to something that you’ve built called grain uh to address this. So for our audience what is it in
10:57the simplest possible terms because I know you’re a very smart person and what we’re trying to address here is a complex problem and many people are
11:07trying to address it at different levels. Uh so please give us the the elevator pitch.
11:14Okay. Uh so grain is essentially a domain specific language for building
11:21event sourced information systems. It’s a DSL. It’s a DSL that you hand to any competent uh coding harness plus model
11:30combination and it can work with that DSL as a tool. And because the DSL’s grammar is constrained, the only valid
11:40approach for building information systems with it is the tool you’ve given it. And because it’s at the code level, it actually operates as a constrained bottom or floor to the decision-m space.
11:51So it allows you to produce correct information systems. And correct is subjective like I’ve said. So this is our ideological approach to how information systems can be structured.
12:02Grain is a system of ideas that if anyone was curious about they could go and look at and they could decide how do I how do I do this for myself to make
12:09sure my company can build software that we’re confident in.
12:14So if I understand correctly uh it’s a set of primitives and you know to to constrain the code right to like
12:23organize it uniform it uh how did you get to select these you know specific
12:30primitives and what did you have to cut um take us to this you know um you know
12:37process and what guided you to to make what decisions. Sure. Um, ultimately I’m a pragmatist.
12:46So, one of the cool things about grain is that I really haven’t invented possibly any of the ideas that are in it. I believe that all of the ideas that
12:55are required to make AI software production reliable have already existed for possibly more than 20 years. The
13:03industry has just been catching up for a while. And since about 2021 uh through uh co-founding some other startups that no longer exist and
13:11working at some other companies, we I found a very smart person by the name of Adam Demetri who came up with this
13:18product design methodology called event modeling. And event modeling has a constrained set of primitives that you
13:26can express any uh information system with. And it’s really a tool designed to bring engineers and nontechnical people to the table so we can all speak one
13:34language. Um it’s kind of an alternative to domain driven design where no one’s ever going to read like a thousandpage book on domain driven design but you can
13:42teach someone event modeling in an afternoon and then it serves as a blueprint for your information system that you can return to as you do change management and whatnot. So there’s a
13:51select set of primitives. Um and I just made grain the executable version of that. So, I’m kind of standing on the shoulders of event modeling because I’ve
13:58seen it turn around companies abilities to actually uh understand when projects were going to be done and velocity and all sorts of things. And I said, I’m
14:07just going to make an executable version of that. So, therefore, grains grammar is the same set of primitives as event modeling. It can express any information system. And uh that’s how it works.
14:19Basically, that’s how I arrived at the primitives. And so that gives me the confidence that I’m standing on the shoulders of someone very intelligent in their methodology. And then I see it
14:28work uh in how grain systems are built and architected. And then you also get the added benefit of any grain system can be visualized as an event model. Any
14:37event model can be turned into a grain system. It’s just really interesting.
14:41And I was already on this track before AI. It just happened to be like, oh, this makes sense. Like this actually makes sense. I can use this. I can work with this uh in my client work and stuff
14:49like that. Yeah. Okay. So, taking that from the theory that you just explained to the execution, right? You’re you’re actually delivering services and and
14:58work, you told me you’re working now um at rebuilding a $10 million company uh
15:06completely on grain, right? You’re using rain. So, what what does it mean? What does it look like daytoday, right, to work uh with grain versus how it was before?
15:17Yeah. Uh so how it was before is a classic tale of there was a development team that built this enterprise application for a company that has a
15:26pretty complex finance domain with a lot of like essential problem complexity that has to be managed.
15:31Those people are no longer around. uh they have a developer who’s newer in the ecosystem and it’s difficult like AI is
15:40useful to make changes in that application but it’s difficult to do a lot of work because it’s difficult to understand the implications of those
15:47changes and uh it’s built on a mutable database paradigm so it’s easy to you you could run into situations where you accidentally corrupt your production
15:55data and stuff when you’re working with AI our build process however is myself and this developer. We sit
16:04together for 3 hours a day cuz I’m also mentoring her on systems engineering principles and how to work with grain
16:11and we do the work of a team of probably 3 to five people every day and we do multiple days worth of work and weeks worth of work per day. This project I
16:20think would have taken a whole team of people at least a year in the past and it probably would have failed because like rebuilds are one of the riskiest and most expensive things that you can
16:29really engage in as a company especially a small company. And so what we’re doing is we’re building a system that is tolerant to change over time because
16:37it’s an immutable database paradigm. AI slots into it perfectly because the codebase as it matures becomes more and
16:45more self-escriptive and declarative. So it like really hones the AI harnesses in on the correct way to do things. And
16:53then it also leads to automated uh debugging and verification cycles and the ability to uh figure out any state
17:01that your user got into, how they got into that state. And that’s just a benefit of event sourcing. But it’s essentially preparing them with a system
17:09that is cohesive and coherent. And simple in the if anyone hasn’t seen simple made easy by Rich Hickey, simple means that you you go into a problem
17:17space and you take it apart and then you put all the pieces back together in a way that you choose so that you’re fighting the incidental complexity so
17:25that you can have more leverage to manage your problem complexity. And so the system has the quality of being simple which makes it easier for any
17:32competent harness and model to work with um at any stage of the process now and in the future after it goes live.
17:39Yeah. So that that’s something interesting I wanted to like uh double click on because a lot of engineering managers now are struggling on you know
17:47complex code bases you know sending AI to work with and come back uh a bit sad
17:55about the results. So when you p point AI at a codebased build this way with grain, what’s different about what comes back? So how would you describe that?
18:06Yeah. So grain as a tool is very self-descriptive and declarative.
18:15That makes applications built with grain very self-descriptive and declarative.
18:22And because all of the primitives each have one job and they do it well and it creates these isolated boundaries that
18:30cause the code to compose together rather than being like uh complexed together like braided together. And so when you point an AI even weaker models
18:39we’ve been experimenting with like we went from using Opus 4.8 daily to now we’re using Claude Code together with
18:46Sonnet 5 and it’s performing great. We pointed Haiku 5 the other day at a pretty complex part of the domain and
18:55asked it to tell us how it worked and it came back really great. And so what I’m noticing is that with because of how the
19:04properties of the grain approach, we’re actually able to start like figuring out where we can descend the model size ladder. And because harnesses and their
19:12capabilities have decoupled kind of from model capabilities, so the harnesses are getting really good and they can control smaller models a lot better too, we’re
19:20actually able to descend the ladder on the most firepower we actually need to work in the domain because we want very
19:27little novelty dayto-day in information systems work. The more novelty in the process, the more things can go wrong over in a system that lives for 5, 10
19:35years, whatever. So the fact that we can start applying less powerful models means that something must be working well about the harness’s ability with
19:43the model to understand how the system is built, understand the implications of making changes. And then also there’s
19:51the added benefit of event sourcing as a paradigm. It’s just more tolerant in the long run to unreliable sort of changing
20:00specifications and requirement underneath. Okay. you’re a very um dedicated person.
20:06So you didn’t mention the cost aspect but for you know when you’re uh lowering the the the novelty of the model you
20:13also pay less which is uh definitely something that more and more enterprises are becoming aware of you know the token
20:21price. Uh so I think uh it’s uh super interesting. You you you also are a very opinionated person and you said to me
20:30that you personally refuse to take accountability when a project hasn’t been built this way. What do you mean by that?
20:38Yeah. So you know it goes back to what I was saying in the beginning. I’m a professional and I’m trying to stand out
20:45in a professional environment. And so I find it to be I find it the idea of generating all this code and delivering
20:54it to someone whether it’s at a company or whether it’s for a customer and then when something goes wrong not being able
21:01to look under the hood and understand exactly how it’s constructed. I find that to violate all of my principles as
21:09a software engineer who takes it upon myself to mitigate execution risk. And so I’m only willing to take
21:16accountability for surface areas where the execution risk has been mitigated in some way. And to me, generating code
21:25without applying a uniform architecture underneath um is very unrisisk mitigated way to work. And I think it’s something
21:33that the industry will start catching on to. It’s just taking people time to catch up.
21:40Interesting. So um the idea behind the podcast is to have a community of people
21:47who are you know dealing with uh software engineering with AI at scale and most engineering managers aren’t
21:54going to build their own grain right uh so what should they actually do with this idea
22:02well I would suggest they should think again um the way that we make software as a society is changing and everyone
22:10who is involved in that process needs to understand what old habits we need to throw out and what new habits we need to
22:18adopt. And so I would say, you know, our work grain, it’s open source. Uh we release a lot of information about it.
22:26Point your AI at it and do some research because these ideas aren’t secret.
22:31They’ve existed for a while. And I think if any engineering leaders were to really sit down, simmer with these, and think about them, it would give them a
22:38real foothold and foundation for actually coming up with a provable methodology to support their engineering
22:45teams going forward. And I’ll be honest, like we’ve chosen to operate in the green field space and in the sort of conversion from legacy application
22:54space. And so I’m not telling you that I have all the solutions to legacy company problems. We’re kind of living in the AI
23:01native world. And so I think anyone who wants to get on that train would really really benefit from going and looking into these ideas.
23:09Yeah. Okay. So we will uh we will put together with the description of this episode the link to your open source
23:16project and I think uh in this era there is a lot you know to learn one from the other and evolve with this. So uh I hope
23:24you build a strong community around that uh that is opinionated the same way you are. Um we’re getting to the end of this
23:32interview. Um, and if there is one thing you want an engineering leader who is watching to walk away uh thinking after this conversation, what is it?
23:44I would say I want them to walk away thinking about how they and their team and their company can be accountable for
23:51their work and get the speed increases that come with AI technology,
23:58right? I guess that’s the whole the holy grail of uh engineering uh organizations
24:04these days. Uh Cameron Bar, thank you so much for uh this insightful discussion.
24:10Uh you’re definitely on an interesting, you know, path uh to find the right balance between
24:19speed, right? uh disruption of how software development is being built and building the foundation and the the
24:27guard rails and you know the floor in the space to make the right decision while building code and building
24:34applications. So thank you so much for this discussion and thank you for taking the time.
24:39Thanks Orin and hope to come back one day as we continue proving out these ideas and talk about what we’re learning. All right, see you soon then.
24:47Yep.